ARC Challenge Live Coding

Evaluate.


ARC Challenge Live Coding

Evaluate on riddles.


ARC Challenge Live Coding

Good.


ARC Challenge Live Coding

And here, I want evaluate on data set and report.


ARC Challenge Live Coding

So instead of riddles, we'll take sub directories.


ARC Challenge Live Coding

No, actually, that's fine.


ARC Challenge Live Coding

That's just good enough.


ARC Challenge Live Coding

Very good.


ARC Challenge Live Coding

By metrics has everything it needs.


ARC Challenge Live Coding

Nice.


ARC Challenge Live Coding

All right, so format it.


ARC Challenge Live Coding

Now, we can, I still don't know how we should get the agent now in there, but we can make an example.


ARC Challenge Live Coding

We can rewrite the eval test right here.


ARC Challenge Live Coding

To this.


ARC Challenge Live Coding

So let's have.


ARC Challenge Live Coding

Uh huh.


ARC Challenge Live Coding

Yep.


ARC Challenge Live Coding

This.


ARC Challenge Live Coding

That.


ARC Challenge Live Coding

Actually, yeah, let's just say.


ARC Challenge Live Coding

Evaluate and report.


ARC Challenge Live Coding

Let's rename it.


ARC Challenge Live Coding

Good idea.


ARC Challenge Live Coding

Very good idea.


ARC Challenge Live Coding

What plugins are using?


ARC Challenge Live Coding

I don't know. I've lost overview myself.


ARC Challenge Live Coding

So, you know.


ARC Challenge Live Coding

And that should probably optionally take metrics, right?


ARC Challenge Live Coding

Because.


ARC Challenge Live Coding

Yeah.


ARC Challenge Live Coding

No, let's do it right. If metrics is.


ARC Challenge Live Coding

Cool.


ARC Challenge Live Coding

We don't need to apply metrics.


ARC Challenge Live Coding

This will do it all.


ARC Challenge Live Coding

So.


ARC Challenge Live Coding

Yep.


ARC Challenge Live Coding

OK, now the one thing I think we also need is we'll need some default metrics, right?


ARC Challenge Live Coding

Get default metrics.


ARC Challenge Live Coding

And that.


ARC Challenge Live Coding

Should give us a list of metrics.


ARC Challenge Live Coding

And that would return.


ARC Challenge Live Coding

Yeah, why not these two?


ARC Challenge Live Coding

So here we can actually instead of this.


ARC Challenge Live Coding

We can say.


ARC Challenge Live Coding

Default metrics because most people are just going to use the.


ARC Challenge Live Coding

Default metrics.


ARC Challenge Live Coding

Test it again.


ARC Challenge Live Coding

And good. So I have about more five, five to ten more minutes of time.


ARC Challenge Live Coding

And I'm going to use that to probably make.


ARC Challenge Live Coding

An example.


ARC Challenge Live Coding

And the example is going to be pretty much just the test.


ARC Challenge Live Coding

So here is the example.


ARC Challenge Live Coding

Let's.


ARC Challenge Live Coding

Copilot go go. Thank you.


ARC Challenge Live Coding

Arc agents.


ARC Challenge Live Coding

Cheating import cheating agent.


ARC Challenge Live Coding

Will.


ARC Challenge Live Coding

From arc dot metrics import default metrics and.


ARC Challenge Live Coding

Eval.


ARC Challenge Live Coding

So, oh yeah, the riddles. Yes, now what we'll actually need is also the data set from utils.


ARC Challenge Live Coding

Maybe we should evaluate and report. Maybe we could.


ARC Challenge Live Coding

Export that somehow in arc.


ARC Challenge Live Coding

Because it might be very common, right? So.


ARC Challenge Live Coding

Like this.


ARC Challenge Live Coding

E. Good.


ARC Challenge Live Coding

OK.


ARC Challenge Live Coding

The task data.


ARC Challenge Live Coding

How do we know if it's not a bot and copilot?


ARC Challenge Live Coding

Yeah, we'll need.


ARC Challenge Live Coding

Get riddles from.


ARC Challenge Live Coding

The training subdirectory, that's fine.


ARC Challenge Live Coding

And the task data, let's say.


ARC Challenge Live Coding

Mm hmm.


ARC Challenge Live Coding

Top K is.


ARC Challenge Live Coding

Three.


ARC Challenge Live Coding

Correct fraction.


ARC Challenge Live Coding

Now this is going to be one. Hopefully, let's try it out.


ARC Challenge Live Coding

Let's go to examples. Let's.


ARC Challenge Live Coding

Try it and we'll see.


ARC Challenge Live Coding

We should probably have some sort of.


ARC Challenge Live Coding

Indicator of the progress and stuff.


ARC Challenge Live Coding

I think that would be.


ARC Challenge Live Coding

That'd be something, no?


ARC Challenge Live Coding

Or correct fraction one.


ARC Challenge Live Coding

Yeah, so do we have a logger already?


ARC Challenge Live Coding

Three months ago he looked more jacked. That is true. That is actually true. I have not had time to work out too much.


ARC Challenge Live Coding

Was busy getting cancelled, so.


ARC Challenge Live Coding

Info.


ARC Challenge Live Coding

Evaluating agent.


ARC Challenge Live Coding

On some riddles, let's in here.


ARC Challenge Live Coding

Evaluate agent on riddles.


ARC Challenge Live Coding

This goes up here probably. That's better.


ARC Challenge Live Coding

Let's put in some Tuckadum here.


ARC Challenge Live Coding

Do we have it in the requirements? Do we have it in the setup?


ARC Challenge Live Coding

We have it good so.


ARC Challenge Live Coding

Where was I just now?


ARC Challenge Live Coding

There we go.


ARC Challenge Live Coding

Alright, let's see how that works.


ARC Challenge Live Coding

Nice.


ARC Challenge Live Coding

Is wearing shades make you better code?


ARC Challenge Live Coding

No, Vim had auto complete functions. It's plugins. It's all its language servers.


ARC Challenge Live Coding

Downloading.


ARC Challenge Live Coding

Get riddles.


ARC Challenge Live Coding

So here.


ARC Challenge Live Coding

Alright, that should do. Then the last thing, applying metrics.


ARC Challenge Live Coding

How about this?


ARC Challenge Live Coding

Let's try.


ARC Challenge Live Coding

We're loading the data set. Very nice.


ARC Challenge Live Coding

Evaluating very nice loading bar. Excellent.


ARC Challenge Live Coding

Now the apply metrics correct.


ARC Challenge Live Coding

Apply metric board size correct fraction one.


ARC Challenge Live Coding

We should probably.


ARC Challenge Live Coding

Good.


ARC Challenge Live Coding

And that is a really nice example.


ARC Challenge Live Coding

It's pretty simple if you ask me.


ARC Challenge Live Coding

And then you can store that to some file if you want.


ARC Challenge Live Coding

And you can probably also implement your own agent, but that's a task for another day.


ARC Challenge Live Coding

So I.


ARC Challenge Live Coding

Added example.


ARC Challenge Live Coding

Isn't it hard to see? Yes, especially if they're polarized. So if I do this, I can't see anything.


ARC Challenge Live Coding

But, you know.


ARC Challenge Live Coding

So in case anyone is interested in solving the arc challenge, this is currently our community project on our discord, which I hope you'll find if not, it's like why culture.com slash discord.


ARC Challenge Live Coding

In case you're interested, I will be pushing all of this code soon. And I'll also make a bit of a.


ARC Challenge Live Coding

I'll make more examples on how you can exactly interact with this, but ultimately you can. You'll also be able to use the CLI for it.


ARC Challenge Live Coding

And yeah, and I'll make the reports do prettier outputs.


ARC Challenge Live Coding

And that will be that I think that will go right here. You'll be able to load some sort of data set.


ARC Challenge Live Coding

Hopefully.


ARC Challenge Live Coding

Probably.


ARC Challenge Live Coding

Yes, you'll be able to load some sort of data set, generate a report.


ARC Challenge Live Coding

Yes, this is a library that you can use to evaluate your agents. What I don't know yet is how do apps like flask and so on, or UV corn.


ARC Challenge Live Coding

You give him you give him like a file name, right? And they dynamically seem to load whatever is in the file name.


ARC Challenge Live Coding

So, crap.


ARC Challenge Live Coding

How can we look this up? Python dynamically load flask from file by file name or something.


ARC Challenge Live Coding

Import from dynamic file name. Yeah, see.


ARC Challenge Live Coding

Python script config file. Yes, exactly. So that is my problem.


ARC Challenge Live Coding

The import module to dynamic import modules. This is what we need.


ARC Challenge Live Coding

So this I'll make it a bit bigger.


ARC Challenge Live Coding

Ah.


ARC Challenge Live Coding

Cool. So.


ARC Challenge Live Coding

Yeah.


ARC Challenge Live Coding

Import module and then get adder.


ARC Challenge Live Coding

Hey, this isn't too hard. We could do that. We could totally do that.


ARC Challenge Live Coding

OK, let's try. Let's try. We got five minutes.


ARC Challenge Live Coding

So.


ARC Challenge Live Coding

We'll make a command. This is eval. Let's just call that.


ARC Challenge Live Coding

Yes.


ARC Challenge Live Coding

And that's going to take.


ARC Challenge Live Coding

How should we how should we do this? That's going to take like agent.


ARC Challenge Live Coding

I then agent path or something.


ARC Challenge Live Coding

No.


ARC Challenge Live Coding

I want something like flask has. So let's just do this a string.


ARC Challenge Live Coding

Yeah, it's not it's not particularly good.


ARC Challenge Live Coding

It's not a particularly good interface yet.


ARC Challenge Live Coding

But.


ARC Challenge Live Coding

Hmm.


ARC Challenge Live Coding

I'm not I'm not a fan of it yet.


ARC Challenge Live Coding

Sub-dir.


ARC Challenge Live Coding

I'm not a fan of it yet.


ARC Challenge Live Coding

Hmm. How about we just do this for now?


ARC Challenge Live Coding

So what I want to do is I want to agent file path agent class name.


ARC Challenge Live Coding

Is.


ARC Challenge Live Coding

As we split it by this.


ARC Challenge Live Coding

So you would be able to do hopefully you would be able to do.


ARC Challenge Live Coding

Arc eval and then you'd say something like arc or agents.


ARC Challenge Live Coding

Cheating agent.


ARC Challenge Live Coding

Like this right.


ARC Challenge Live Coding

And.


ARC Challenge Live Coding

Use import lib.


ARC Challenge Live Coding

Yeah, exactly. That's pretty good. We'll do it.


ARC Challenge Live Coding

So.


ARC Challenge Live Coding

Import import module.


ARC Challenge Live Coding

But it needs to be in path so we can't actually do this right here.


ARC Challenge Live Coding

We'll have to do. We'll have to do. Well, this is super duper slow.


ARC Challenge Live Coding

Like this.


ARC Challenge Live Coding

Yes, yes, yes. Good.


ARC Challenge Live Coding

Agent module name. OK.


ARC Challenge Live Coding

Agent class. Very good.


ARC Challenge Live Coding

I copilates doing all of this for me. Really, there's nothing nothing I need to do.


ARC Challenge Live Coding

And now I need to to evaluate it right so.


ARC Challenge Live Coding

Import evaluate and report. So I'll get my.


ARC Challenge Live Coding

Dataset get riddles and we'll get like.


ARC Challenge Live Coding

We'll make later parameters but.


ARC Challenge Live Coding

Report equals evaluate and report.


ARC Challenge Live Coding

No, that's just a function.


ARC Challenge Live Coding

Evaluate and report takes agent riddles task data and metrics.


ARC Challenge Live Coding

Riddles so.


ARC Challenge Live Coding

ARC import task data and.


ARC Challenge Live Coding

Get default metric. That's probably also something we should put into ARC itself. No.


ARC Challenge Live Coding

I think so.


ARC Challenge Live Coding

Yes.


ARC Challenge Live Coding

Yes, good.


ARC Challenge Live Coding

OK.


ARC Challenge Live Coding

Again.


ARC Challenge Live Coding

What do we need here? Agent, riddles.


ARC Challenge Live Coding

Task data.


ARC Challenge Live Coding

And metrics.


ARC Challenge Live Coding

OK, and then we.


ARC Challenge Live Coding

Print from the report. We probably want to print something like.


ARC Challenge Live Coding

The metric results.


ARC Challenge Live Coding

Let's try.


ARC Challenge Live Coding

If this works.


ARC Challenge Live Coding

Yes, it actually appears to work wow.


ARC Challenge Live Coding

OK, now it just needs a bunch of parameters and then it's cool.


ARC Challenge Live Coding

Very nice, very nice. OK, yeah, it gets everything correct. Very cool.


ARC Challenge Live Coding

So yeah, we'll just be able to use this CLI.


ARC Challenge Live Coding

Added eval to CLI. This is going to change a little bit the eval, but I hope it's going to be useful.


ARC Challenge Live Coding

And that's that from me. I'll just push this as I said, the repository is open. Discord community is open.


ARC Challenge Live Coding

We're currently just in the beginning stages of solving this. I hope we get.


ARC Challenge Live Coding

Farther than the people so far.


ARC Challenge Live Coding

And get actually something to work, which would be nice.


ARC Challenge Live Coding

All right. Thank you everyone for being here. That was it.


ARC Challenge Live Coding

Just some chilling Saturday afternoon coding.


ARC Challenge Live Coding


Pickling and Scaling - Practical Machine Learning Tutorial with Python p.6

What is going on everybody and welcome to the sixth machine learning tutorial in this tutorial?


We're going to be talking about pickling a little bit about scaling


And then we're going to move on into diving into the inner workings of linear regression and of course the other algorithms, so


Pickling really doesn't really have anything to do with regression. Just simply is a good


Good thing to have it at your disposal, so you can save yourself a lot of time


So first of all what is pickle pickle is just serialization of any python object


So this could be a dictionary or in our case it could be a classifier or a whole host of other things


So first of all what we're going to do is we're going to import not in caps. We're going to import pickle


Okay, also I noticed


Yeah here for some reason I was like double defining. Why is the label thing my mortality is showing anyway?


Um yeah, just get rid of that if you did that hopefully hopefully nobody followed me into that mistake, but anyway


Yeah, so there's that now


What we're going to do is the way the pickle works


It's like a file right you open it you save it right and then when you want to use it you open it


You read it, so


We're going to come down basically


You know at what point would you want to save the classifier right so like thinking logically when should we save the classifier?


Well should we just should we save it here?


Probably not right. I mean you could you could save that classifier, but that's an untrained classifier


So you probably want to save it here right so like what is the purpose of saving a classifier?


It's to avoid doing the training step


Because that's a very tedious step right that that's what's going to take the most time in our case


It doesn't really take that long because we don't have that much data, and we were threading it


But you can imagine if you had gigabytes and gigabytes or even terabytes of data that you were doing this on


Every time you wanted to make a prediction you would not want to


Have to retrain an algorithm so instead you can save it now of course if you save it


You know you might want to retrain it like once a month or something like this


But you don't have to retrain it every time you want to use it so


What we're going to say here is we're going to say with open


and we're going to say linear regression dot pickle as


WB


As f so we're going to open this file with the intention to write and we're going to just use temporary variable as f


And then we're going to say pickle dot dump. What are we dumping? We're going to dump that trained classifier so CLF


Where are we dumping it?


f so that dumps the classifier and then to use the classifier all you would actually do is you would say


like something like pickle in equals open and


Then we would open this file


Open with RB and


Then we would say


CLF equals pickle


Pickle dot load, and we're just going to load pickle in so whoops hello


What I do here you go copy this paste okay, so now let's go ahead and just run this and see how we do


Okay, so everything works at this point and whether or not you recognize the fact


We have actually renamed classifier or redefined classifier here


So now what would happen if we did this?


Right now there is no definition of classifier. We're not saving it as a pickle


We're simply reading a pickle so if I refresh this to reload this what we get is


There we go, so you get this information here, and the pickle is saved in the directory


We're working in so that would be if I could find it here open it up right


There's your pickle data so in our case the pickle data or the classifier is actually a really really small classifier


That's not a big model


But anyway, we save ourselves the time of actually training that


classifier


Also, I said it in the introduction video, but I'll say it again


Remember that we live in a time where you can spin up a server


For a very short amount of time and you could do that so if you have like a slower computer


Maybe your only computer is a relatively slow laptop. You've got a I don't know one of these little netbooks or something


If you are one of those people you can spin up a gPU cluster. You can just spin up a regular server


You can spend a very powerful computer and basically rent it for a few dollars an hour on lino digital ocean or Amazon web services and


If you do that what you can do is generally what I do if I'm going to use a big server


Well pretty much all these hosts work the same way you take your data


You put it on their server and like you'll set everything up, so you'll you'll transfer all your data there


You'll set up all the code that you want to run and all that and while you're doing that you're using like the smallest


Version to have so you're paying like a half a penny an hour to rent that version of a server and once you're ready


scale that server up and


Then you're paying maybe a few dollars an hour. Maybe even ten dollars an hour if you're crazy


Scale it up run your operation take your classifier save it to a pickle take your classifier


Scale back down the server destroy the server whatever and you're done, so


Just a couple of quick you know pointers because we're not really going to be talking much about that


For scale for a while, but these algorithms actually you can scale linear regression very well, so


Just keep that in mind so anyways


That's really all I have to say about pickling and scaling and now what we're going to actually be doing is


Writing our own linear regression algorithm


To learn how that actually works and all that so very exciting stuff if you have questions comments concerns


Whatever leave them below otherwise as always. Thanks for watching. Thanks for all the support subscriptions and until next time


Linformer: Self-Attention with Linear Complexity (Paper Explained)

Hi there, today we're going to look at Linformer self attention with linear complexity by Sinan


Wang, Belinda Li, Madian Kapsa, Han Feng and Hao Ma of Facebook AI. So on a high level,


this paper observes that often the way we build transformers, the self attention matrix is low


rank and can be approximated by first projecting the signal to a lower dimensional space, and then


performing these inner products that are responsible for attention in there. And thereby, you save a lot


of the complexity of multiplying full sequence length, full sequence length by full sequence


length matrices, but instead do these operations in the lower dimensional space. And they achieve


a linear scaling of the transformer attention. And we'll figure out how that is. As always,


if you like content like this, consider subscribing, sharing, liking, and commenting,


if you feel like it. Okay, let's dive in. They say large transformer models have shown extraordinary


success in achieving state of the art results in many natural language processing applications.


Okay, so these, if you don't know what a transformer model is, you can watch my video on the paper


attention is all you need. That was sort of the beginning of these transformers. And it introduces


the attention mechanism that we're going to look at today. If you don't know what an attention


mechanism is, you're not going to have a fun time in this paper. They say, however, training and


deploying these models can be prohibitively costly for long sequences, as the standard self attention


mechanism of the transformer uses n squared time and space with respect to the sequence length. Now,


why is that? So really shortly to recap, recap this, the attention mechanism, the this attention,


these transformers, they transform for basics, let's say they transform one sequence into another.


So here we have five tokens, and the next layer will output five tokens, okay, for five tokens in


five tokens out. And the question is, how do you route information between these five tokens from


the first layer to produce the next layer? In a feed forward network, you would simply connect


everything to everything and sort of learn the weights of these of these connections. That's not


what we do here. In a convolutional network, you would simply connect each node to its immediate


neighbors like this. But this is also not what we do here. What we do here is we route the


information according to the information itself. So according to the incoming information right


here, we route the information that goes out. And we do that by expressing key queries and keys. So


these incoming information is transformed, first of all, into what are called keys. Now keys are


simply vectors. So each node is going to expose a vector right here. And each node in the higher


layer. Now, these are produced by the same from the same information down here, but I'm going to


draw it conceptually on the higher layer. So each node here is going to expose a query, which is


sort of like calling the query is calling for what kind of information do you want from the lower


layer. And the key is sort of exposing what type of information this node contains right now. Now,


the information is simply routed by looking at the inner products of these of the keys and the


queries. So this information right here would probably be routed to this node right here,


whereas this one would probably be routed here, this one would be routed here. In fact, this is a


soft assignment. So it's not like a hard routing, it's a soft routing, everything is routed to


everything with different weights, but the majority goes to the place where the inner product is high.


And this one is again routed here. So you can see this is the attention mechanism. In order to do


this, we need to compute the inner product of every single one of these queries with every single one


of these keys. Okay, and this if our sequence length here is of length n is going to require n


squared operations. Okay. Now, here is another parameter we need to pay attention. These vectors


here, they have a certain dimension and the certain dimension we're going to call d, the inner, the


embedding dimension of the vectors. Now, in modern transformers, you can think of n as something like


maybe 512 tokens go into a transformer like this. And the hidden dimension here also is in the same


order of magnitude. So you can also imagine this to be something like 512. Now if you think of these


matrices, if you multiply the keys by the queries, however you want to do it like this, then you have


the keys are n by D and the queries are D by n. Okay. Now since n and D, in this case are the same


dimension, this matrix is of rank of rank 512 doesn't have to be but it's a pretty good bet


that it's of rank 512. Maybe it's approximately lower rank but now this isn't actually the modern


way of transformers as such because usually what we have is multi head attention, which means that


we're going to split this inner dimension right here, we're going to split these vectors into many,


many lower dimensional vectors and then have attention mechanism on these lower dimensional


vectors. And that's such that you don't only have one attention mechanism, you have multiple


attention mechanisms. So you can route different kinds of information with these multiple attention


heads. Now, sometimes you would split this, you could split this in a modern transformer up to


like 16 different heads. But here we're going to let's say we're going to split this into four


sub vectors each of 128 dimensions. Okay, so we're going to split this up. And now if this in this


product here is only computed on these lower dimensional vectors. So all of a sudden, you no


longer have n by D, but you have like n by D over four. And now this is 512 still, but this now is


128. So the rank of this matrix is going to be 128. Mind it's still the thing that comes out is


still a 512 by 512 matrix, but it is of rank 128. And that means even though this matrix contains


vectors that are of size 512, they could be they could be represented accurately by a matrix that's


just 128 dimensions. Okay, so these these 512 dimensions actually only contain information


that is 128 dimensional in nature. It's just distributed over 512 dimensions, but most of


these are redundant. So in fact, in these modern transformers, these thing here, this matrix here


is low rank. And therefore, that's what this paper sort of exploits, we could we could approximate


this by 128 dimensions. Okay, this is our starting point. They go on and they say, in this paper,


we demonstrate that the self attention mechanism can be approximated by a low rank matrix. We


further exploit this finding to propose a new self attention mechanism, which reduces the overall


self attention complexity from n squared to n in both time and space. The resulting linear


transformer, the linformer performs on par with standard transformer models while being much more


memory and time efficient. Alright, so let's dive into their thing. This is how they formulate the


attention mechanism. So right here, the attention has queries and keys, as you can see here. Now


these w matrices, you can largely ignore the W simply maps the queries to so this is these are


simply D by D matrices that are a linear transformation of the queries, you can sort of overlook them for


the arguments in this paper. So these are the keys and the queries we talked about the values here,


this is the actual information that's being routed. So what we want to do is we want to


compute this product between queries and keys right here and scale it appropriately. But ultimately,


this is this product, then run this through a softmax operation. That means we, we normalize it


such that it sums to one the distribution sums to one. And then we want to route this information


according to that distribution. Okay. So that's how they formulate an attention mechanism. Now


notice something. This thing in here is what they call the matrix A. And this is what I've demonstrated


to be low rank. Now, the actual thing that you would need to be low rank for their paper to hold


is the matrix P, which is different because this is after the softmax, right. So if the matrix P


is low rank, then you have a legitimate claim of approximating this routing via a low rank matrix.


However, if P is not low rank, you don't. Okay. All right. Now, the first thing they're going to


show is that this is in fact a low rank. So self attention is low rank. And for that they make an


empirical investigation into Roberta. So Roberta is a model that's based on BERT. And I have made


videos of both BERT and Roberta, I believe, if sorry, if you want to go look those up.


But it is one of these transformer models. And they take two data sets, wiki 103 and IMDB,


and they run them through this model. And they look at this P matrix. So they look at


how this this information routing matrix is built. And then they calculate the eigenvalues of that.


So you calculate the eigenvalues. And by looking at the eigenvalues, you can look at the rank of a


matrix, broadly speaking. So if you list the eigenvalues in order of their size, then a matrix


that is sort of high dimensional has a high rank would have sort of a slope like this. And that


means as you go, as you go to the next and next and next eigenvalue, they drop. Like if you order


a set of uniformly distributed numbers, if you order them, then it would look like this, right?


So there is no particular dimension that's better than any or has much more information than any


other. However, if the matrix is approximately low rank, you would look something like this.


And that would mean that most of the information is concentrated in very few dimensions. And those


are the ones with very high eigenvalues. And most of the dimensions have no information.


The thing you see here is simply the cumulative sum of these things. So if you calculate the


cumulative sum of this, you'll get that over here. So if this is very high rank, you would expect


a curve that goes like this sort of slanted, but not very, if this is very low rank, you would


expect a curve that goes very much into the corner right here. And they show that the general shape


here is such that there is this kind of a kink to it, as you can see here. Now also notice that the


axis here starts at 0.4. So actually, this comes from down here somewhere and goes up and then goes


like this. So they have a, I feel they have a legitimate claim here that these matrices are


approximately low rank. And here they look at, I don't actually know at which layer this is,


or if this is in all of the layers overall, or something like this. But they look at how this


develops inside the layers. So they look at the always the 128th eigenvalue. And they


discover that as they go deeper and deeper into the network, this cumulative eigenvalue is higher


and higher. That means that network puts more and more information into fewer and fewer dimension


in this routing as you go up the layers. So it gets more and more skewed. As you go up the layers,


it gets more and more into this corner right here. So their claim appears to be more and more true.


Now, I have sort of thought about this a little, and I've tried it out a bit myself. And I invite


you to just follow me here shortly. So right here, I have a matrix that is just a random


Gaussian matrix of size 512 by 512. If we look at the eigen spectrum of that, so I have this function


SVD, it simply gives me the eigen spectrum of that, then you can see that it sort of falls off


uniformly. And that will result in a in this cumulative sum of of pretty much flat curve or


slowly ascending curve like this. Okay. Now, if we actually have a low rank matrix, this would look


different, this would have this sort of typical kink in it. And we can demonstrate that by making


a lower dimensional matrix. So let's just take let's just go 512 by 128 of this lower dimensional


m. And let's look at the mt. Now this only goes to 128 because we only get back 128 singular values.


So let's make a lower dimensional matrix that's actually 512 by 512. So if we do this,


this is sort of what they're doing in the in this. This will construct a 512 by 512 matrix,


but that is only of rank 128. Right. And you can see that at the 128th singular or eigen value,


this snaps right at the at the one. So it's sort of like what they what they have. Okay,


so we've seen the difference between a, let's say higher rank matrix and the low rank matrix in this


cumulative sum plot. Now I want to go back to the original matrix right here. Of course,


they're the matrices they look at these routing matrices, they're not Gaussian, they're not sort


of distributed with mean zero and the nice variance, they are the result of a softmax


operation. And in particular, that means they're all positive. And that means that their mean is


not zero. So if you look at a data set, and it's mean, it's not zero, and you calculate like the


eigenvalues, or in this case, the principal component, you will find that the first one


will be very strong, because that must account for the fact that the mean is not at the center,


the first few will be like this. So it is sort of, maybe we can replicate this right here. So let's


say, we'll put m through, let's first go with the absolute value of m. Okay, not much of a change,


okay, not much of a change, but you'll already see that this axis doesn't start at zero. So let's go,


let's actually, how do we do this? Xlim, right? Xlim zero, none. So, okay. So the first one,


you simply have to imagine, or I can do even something, something more, we can just put a zero


in front here. And that should do the trick. No, yes. Oh, that's X, I meant Y, god, I'm a dumb.


Nevermind, this will work as well. So you already get this sort of kink. And let's put it into the


softmax. So we put a softmax. And that gives you also this kink. Now you might think that, wait,


this is that this kink looks a lot smaller than the other kink. So, but if we simply modify,


let's modify the standard deviation of this random matrix. And you can see that the spectrum


immediately changes, right? Because of the interaction now between the softmax and the standard deviation.


If I only were to change the standard deviation on the normal M matrix, and we can actually try this


right here, that wouldn't do much, that would still look pretty much the same, it's just


differently scaled. But in the interaction with the softmax now, this changes the spectrum


dramatically. And here, as you know, these these transformers have always sort of like layer


normalization and so on. So probably the standard deviation, if we if, if these are sort of Gaussian,


the standard deviation before the softmax would be a lot smaller. So let's go something like this.


So smaller than one. And can we run this please. And you can see that this kink immediately appears.


Now it's not it's it's it's not the same thing as this other as this here, because this is a lot


smoother, as you can see right here. But still, I feel that this might not actually be a result


of the, you know, the fact that this is an attention mechanism, but it simply might be the result of


that you apply a softmax. Now, still, that doesn't change the fact that it is approximately


a low rank matrix. So everything they say, holds but yeah, maybe, maybe one should also look into


why exactly that happens. But in fact, it is low rank, okay, it is approximately low rank, they've


demonstrated this. And now they go to their first first theory below, we provide a theoretical


answer, a theoretical analysis of the above spectrum results. Okay, so the theoretical


analysis theorem one is self attention is low rank. And we're going to go to the second


one is low rank. And we're going to go through this. Just glance at it for now. They say for any


of these query key values and these matrices, which of course you can ignore for now, for any


column vector w of matrix v w and w here, that's the information that needs to be routed. There


is a low rank matrix p tilde. So this p tilde here is going to be their low rank approximation


of the P matrix, you can see it's still n by n, but it's going to be low ranking. In fact,


it's going to be of the order of the logarithm of the rank of the full matrix. Or, well, the full


matrix of the rank that the full matrix could have, as we have already seen, the full matrix


doesn't have full rank, but yeah, okay. So if you use and this is the type of guarantee you get,


so what do we see here, it basically means that this distance here is smaller than this. And this


here, this is just the norm of one of these vectors projected times this error coefficient epsilon. So


all it says is that the distance on the left is smaller than something. And that's occurs with


high probability. Okay, so the entire guarantee here, the entire formula just basically means that


this thing is small, this norm is small. What's this norm, this norm is the distance between these


two things. Now, what are these two things, this is the information that we want to route. And this


is the routing matrix. And that simply means that if I route my information, using the P tilde,


this approximation, then I won't be too far away as if I had routed my information using the original


P matrix. Okay, that's, that's it. That's what the theorem says. The theorem says, if I route my


information using this approximation, then I am not too far away as it had I routed my information


using the original routing matrix, that I don't say how they're going to construct, they simply


say there exists a low rank matrix like this. And the proof of this, and it's sort of worth


looking at the proof of it, it uses the Johnson Lindenstrauss lemma, this thing here, or the JL


for short, and they're going to get this out of the JL. Now the Johnson Lindenstrauss lemma,


in a classic sense says something like this, if I have data in a high dimensional space here in a


three dimensional space, okay, I have data distributed, and I use a certain kind of


projection matrix. And there are a number so the JL gives conditions on what these projections can


be. But for example, a randomly sampled matrix with zero mean Gaussian entries and one over k,


standard deviation where k is the dimension you project into can do the trick. So if I project


my data in a certain way, into a lower dimension here dimension two, then the projected data is


related to the original data by the fact that the distances between the points in the original space


will not be distorted too much. So the distances between these points are approximately preserved


through this projection. Okay, so that's, that's the that's the Johnson Lindenstrauss lemma. Now


you'll notice here, there is no reference to the fact that this data is or isn't low rank,


it's simply high dimensional data projected to lower dimension, and the distances are approximately


preserved. And this theory here, and I've looked at it for a while now, they simply define,


okay, they define this P matrix as this attention mechanism. And here you can see the A matrix we've


discussed before, which is actually low rank, but we don't know yet if the softmax is, they write it


as this form right here of the exponential of each entry of A divided by this diagonal right here.


So in the softmax, of course, you have the exponential of each entry divided by the sum of


the entries, and they write this simply as two matrices. But ultimately, this is a matrix right


here. And all they do is they take this P matrix, and they apply the Johnson Lindenstrauss lemma


by having this projection matrix R, and R is entries from this Gaussian, as I said,


so this is the special type of projection that the JL addresses. And then it simply says, if you pull


if you, this here is going to be your P tilde. So if you project R in this manner, and obtain P tilde,


and then you use P tilde instead of P, then this, this is going to be very close. In fact,


you can reformulate the JL into different variants, such that it gives you things like this,


things like saying that the distance between this projected version and this unprojected version is


going to be a constant smaller than a constant time the norms of the unprojected version,


that is equivalent to saying that the distances are preserved. Now you can see right here,


nowhere in this theorem is the fact that this is self attention. And nowhere in the theorem


appears the fact that this inner matrix A is low rank, or even that this matrix A exists,


it's you can do this with any matrix P, right, the JL doesn't concern itself with the nature of this


matrix P, it says any matrix, any sort of high dimensional data, you can project to low dimensional


data. And this holds if you choose the projection correctly, which they do right here. So to claim


that this theorem proves that self attention is low rank, to me is a bit, it's a bit of a


statement that is not warranted. Like, this here should read something like the Johnson Linden


Strauss lemma exists, or something like this. It, I'm not I'm not sure like convince me otherwise,


but yeah. So they go with this. So they say given the low rank property of the context mapping matrix


P, now, again, I disagree that this has been shown, except empirically. One straightforward


idea is to use singular value decomposition to approximate P with a low rank matrix P low as


follows. So what you could do is you could simply learn these low rank matrices and approximate


P through it, or you can decompose P as such, and then have these easier inner products in dimension


K. But they say, however, this approach requires performing an SVD decomposition in each self


attention matrix, which adds additional complexity. Therefore, we propose another


approach for low rank approximation that avoids this added complexity. Okay, so they now come up


with their model, and their model goes as follows. So here on the left, you see a classic attention


mechanism with their projections built in, what they're proposing is they say, let's project the


matrix K using one of these random projections. And then this attention routing, if you route,


if you now multiply, so you multiply K and Q right here, K times Q, and then you put it into the


softmax, and then you use it to route this W. So they say, if we build in this projection matrix,


that will project K to a lower dimension, and then we won't have as expensive of inner products.


Now, the important part to see here is that if you think of this lower projection, the first thing


you think is that you project this inner this hidden dimension D, right, to allow a dimension.


And that's not the case here, you actually project the end. So in, in a conceptual framework,


so you can see right here, forget about this, this is this W matrix. In a conceptual framework,


you see here is this n by D matrix, which are the keys. So n is the sequence length,


and D is the dimensions. And what you want to do is you want to project that by this matrix,


which is K by n. So you want to reduce the sequence length. You can see in this matrix right here,


why that might work, because n is much larger than D. And that means this matrix can be at most rank


D, right? So you should not lose too much, you should sort of be able to preserve the information.


If you project this n to a K, where the K if the K is still larger than the D, or approximately in


the same order of magnitude, you should be able to preserve that information if you do it in a smart


way. So conceptually, if we have our five token sequence, like here, and the next layer produces


five tokens, again, what we first do is we say, we know, we know that the information we want


is not five dimensional, it's actually two dimensional. Because, okay, let's say the


this inner dimension D is, is two as well. So we have two dimensional vectors, each thing exposes


two dimensional vectors. So we first project the sequence of length five to a sequence of length


two. And we simply do that in a random manner. So we have a random Gaussian matrix that assigns


weights to mix these five into these two. And again, because the JL works for any sort of data,


but in my argumentation, if you, you know, think that this here is low rank, it's of rank two,


then you shouldn't lose too much information by projecting it to a sequence length two. And now we


do this attention mechanism. So now we expose the keys. And now we expose the queries up here.


And now you can see instead of routing five things with five things, you only have to route


five things with two things. And so instead of having o n squared, you now have o n k, if k, k


is the number right here. Okay, so this is the idea, you project the sequence length.


And it comes from the fact that the sequence length is much larger than the dimensionality.


And therefore, you can sort of preserve the information if you project in a smart way.


They build this in this fashion right here. So the attention mechanism now,


before we saw it was between the queries and the keys, right here, they built now this projection


matrix here, that projects the keys into a lower dimensional sequence. And the now such that


this will result in an n by k attention matrix we saw over here, you don't need to route n by n


things, you need to route n by k. So this, this routing table in here is now n by k.


Now the next layer, as you can see here, it actually needs to produce a sequence of length


five again, right? So we always transform sequence of length five into sequence of length five.


But now we have, we have this n corresponds to the sort of corresponds to the next layer,


and this k corresponds to the down projected sequence of the last layer. And in order for


that to fit, we of course, also need to down project the information that we're routing. So


if we down project the routing table, we also need to down project the information that we're


routing. That's we do this by a similar matrix F that is also sampled in this way in this special


way. And that gives us a k by d. So we have projected the sequence to size k. And if we


multiply these two things, again, of course, we'll get out an n by d matrix, which is the


signal for the next layer. Okay, so an n by d signal comes in down here, it's projected down to


k sequence length, it's and it's routed up again to n sequence length. And you have again an n by


d matrix here. Cool. So that's how they do it. And they build this into the transformer. Now,


as I understand it, these projection matrices, again, they're not learned, they are do they are


built up in this JL conscribed way, they are not learned, they are fixed once. And then that's


that's that, at least that's how I understand it. So there are no more learnable parameters. Okay,


so here they have a demonstration where they up the sequence length. And you can see the batch size


decreases, but that's just to sort of keep the total amount of flops to be done the same, you up


the sequence length and down the batch size. As the sequence length increases, the standard


transformers requirement in inference time goes up. And this here, as you can see, this is not a


linear scale, it's a log scale, log two. So this goes up with the sequence length, and it should


go up quadratically, right. And you can also see that the linformer keeps fairly constant for the


same k. Now, of course, as you increase the k of the linformer, the inference time will go up,


because now it's dependent on n times k and not on n times n. Okay. So let's look a bit further of


how you have to choose that k up here in the first theorem, we there was a already a hint to it. In


the first theorem, you had to choose k by five log n. And this is a problem. So here you have log n,


that means it's not so O of n k is equal to O of n log n. Now that's not linear. That's actually


that's the same as the reformer. But they want to get to a linear place. And theorem two explains


goes now to a linear shows how you can make self attention linear. Okay, they show again, blah, blah,


blah, blah. Now you have to choose k at the minimum of these two things. And you can see right here


that one of them is independent of n. So that means as n grows, of course, the minimum is no


longer going to be this here, the minimum is actually going to be the thing on the left. And


that is dependent on just D. Okay, so you have D log D in here. And that makes sense. Because in the


very beginning, we said, hey, D is actually much smaller than n. And that means the information


that is contained in these matrices is at most rank D. So if we down project to k, we should


adjust k to what D is right, if we adjust k to about the same thing as D, we're guaranteed to


not lose too much information. So now we choose k according to D instead of according to n and


therefore, the computation is linear in n. And n times k is like n times D to log D. So it's linear


in k and linear in D. How do we get there? So the first thing they do is they make these sort of


Johnson-Lindenstrauss statements again. But now instead of the general statement, they plug in


their actual modified attention mechanism. So here they have a bound on the distance between if I


route my this is the information to be routed, right? If I route my information using the


original softmax, and this in here is the matrix A, if the original attention mechanism, I won't


be too far away, as if I were to route my information using this modified attention


mechanism. Now, the tricky part here, mathematically, I believe is that is exactly the


softmax what what I alluded to, right? So this softmax is the tricky part, because if this


weren't a softmax, so if the softmax weren't here, this would simply be a projection down and a


projection up and the dilemma would almost apply as it is written, right? You wouldn't have to


actually do anything. But the question is, if this inside the softmax is low rank, can you make a


claim that the entire softmax then is also low rank? And it's not entirely clear, because,


because, oh, yes, we've done this. So you can see right here that the softmax we have actually done


the softmax of a low rank matrix. So we have already seen the low rank matrix itself, and how


it immediately snaps to the to the upper axis after 128. Now, if we do the same thing for the


softmax of that, and we probably have to take away some of these dimensions, the first few,


let's go with, let's go to dimension 100 and look from there. Okay, same thing. Okay, that's pretty


good. I did not expect that. Hi, there. So this is Yannick from the future. I've realized I've been


an idiot and how I constructed these low rank matrices right here by multiplying MT by itself


of course, what's a better way to do it is to construct two independent 128 dimensional matrices


like these two sub slices of M right here, and then multiplying those together and looking at the


SVD. And you as you can see right here. So the softmax of this is now not of this super low rank


anymore. It's still low rank, but it's not not very, it's not like hard low rank. So if I just


look at the matrix without the softmax, then you can see it has a very peak that by at 128, which


gives us the indication it's actually 128 rank, which we already knew. But if we now introduce the


softmax, then you can see that this vanishes and it's no longer 128 dimensional. And it's only


approximately low rank as you can see. Alright, back to Yannick in the past, who is wholly


surprised that the two that if you multiply MT by itself, that that will give you back the


the exact same thing. Alright, so did we try this before? Maybe we did. Okay, but the mathematical


difficulty still remains and their main thing here is so they have a first first version,


where they pretty much plug it into the JL again, and they they get out this k is the k needs to be


by log n. But they say this result does not utilize the low rank property of matrix A.


And the resultant k has a dependency on sequence like n. And then in the appendix, they finally


go through the math to show that now if they choose E and F like this,


they can actually pull out this and show that the k is where we where you have it,


the k is independent of n like this. And I think the main the main step in this proof is the step


B here, where they say uses the fact that the exponential function is Lipschitz continuous


in a compact region, then we can choose a small enough delta such that the as you can see here,


this now directly relates to this projection matrix within the exponential function to the


projection matrix out of the exponential function. So you can basically say that if I project first


and then use the exponential function, that's not too different than if I first use the exponential


function and then project. Okay, so that's the that's the sort of of of catch here. Now they


only do this for the exponential function, not the actual softmax. As you can see here throughout,


they do it to the exponential function and also here in their statements. The softmax isn't the


exponential function, the softmax is the exponential function divided by the exponential function.


The exponential function divided by the sum of the exponential functions, but I believe that this


generalizes straightforwardly. Alright, so for given choices of delta and k, they have shown that


the linformer in fact, can do in a linear fashion what a transformer can do in a quadratic fashion.


And they are not too far off. Okay, that's that's their point right here. Their results on


these benchmarks, oh, sorry, let's first go to the perplexities in language modeling. So they show


right here that they pretty much can keep up with the standard transformer as you can see here.


So with the standard transformer, they can keep up here. Now think that this the the computation is


n times k, okay, so something like this linformer with k equals 256 will only so instead of n by n,


it's n times k, it won't save you too much in that case. But it's it's not too surprising that


in fact, you have the same performance because probably the standard transformer is distributed


over more heads than two. So the information necessarily has a lower dimensionality 10 to


56. One thing I want to draw attention to though here is that you can see that here, it's not


really done learning yet. And as you can see, the standard transformer sort of surpasses all of these


models towards the end. I wonder, I wonder what happens, I wouldn't be surprised if they end up


sort of at the same place. But I wonder if these diverge even more right here after that.


They also compare with a higher sequence length. And the standard transformer outperforms the


linformer. But of course, the point here is that the linformer is much, much, much faster,


and can keep up. Now, also, the scale here of the perplexity. You see, these are percentage points


in perplexity, but I can't actually tell if that matters or not. I think, I think the original


transformer paper, the perplexities hovered between like three point something and five point


something. So this might actually be sort of significant differences. And I'm not sure. They


investigate different methods of sharing these weights of these of these projections. And they


seems like they don't find real differences, but I don't want to go into that because this video is


already really long. And then they look at what happens if they up the sequence length that they


put into the linformer. And you can see that the linformer can deal with higher sequence lengths


and arrive at the same perplexities. Though again, I don't know how much how much different that is,


and the scale here is larger than before. But yeah. So how does this fare on these benchmarks


where you first train a transformer with pre training with language modeling, and then you use


it to do certain NLP tasks. And here you can see that the linformer is on par in some of these


tasks with the original transformer. But also you can see like a pattern where you can see pretty


wild results in that, you know, sometimes the the linformer here will be better than this,


but then also variants of the linformer will be worse. And they'll even be worse than this. And


sometimes they'll be better. Sometimes this linformer is good. And sometimes the original


model is the best. So this sort of points to you can make the general claim that the linformer


doesn't destroy your gains, but also it's not like a a better model, it's simply a faster model


that in some tasks can keep up with the original model. And they show that, of course, this is the


real deal here, that as you go up in length, the performance gains, and also sorry, this this way,


the performance gains, and the memory gains that you get by the linformer are dramatic. Of course,


the longer and you go into the lower dimension you project, the more these gains are, but of course,


the more performance you're going to lose potentially. Hello, again, Janek from the future,


just wanted to draw your attention on this beautiful, broader impact statement in this paper,


saying our work focuses on making transformers more efficient, everything cool, potential positive


inspect impacts of efficient transformers. That's pretty cool. It also has potential impact on


training transformers on images, since we can support very long sequences, very cool. Furthermore,


there are positive environmental benefits. Very cool. I mean, these are all very cool things.


They say, as such, we see no immediate negative ethical or societal impacts of our work beyond


what applies to the core building blocks of deep learning. Do better. Now this this,


I honestly I agree with them, right? I completely agree with them that this is sort of a good thing.


You might trade off, you know, some accuracy, you might make some approximations, but you'll get a


much faster model. And this model, as any model can be used, you know, for things, and that they


now have to pull out of their out of their butt some way in in over five steps of intermediate


layers. This could be used for bad, it just seems ridiculous. So good on them for defying the


please also think about negative impacts right here. All right, back to back back to past Janek.


All right, this was the linformer paper, I hope this somewhat makes sense made sense to you.


I had to read it multiple times for it to make sense to me. But ultimately, it's all about the


fact that you have these multiple heads. And therefore, your information is probably lower


dimensional and you can abuse that and to just calculate in this lower dimension.


BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding&Generation

Hey all, this is a comprehensive paper review of the paper on Blip. This is a model and


a technique for bootstrapping one's own data set in vision and language pre training, which


is pretty cool. So the video is a comprehensive review. We'll dive into the paper, we'll see


what the paper is about. I'll explain you what's in it. And by the end of the video,


you should have a good understanding of what's in the paper in the next video, which I'm


going to release tomorrow, there's going to be an interview with the authors of the paper.


So also be sure to check that out, because that answers a few very, very interesting


questions that I had while reading the paper itself. So I wish you a lot of fun. Let me


know what you think in the comments and I'll see you around. Bye bye. Hey there, this video


is sponsored by Zeta Alpha, which is a new neural discovery and recommendation engine


for papers. Yes, for scientific papers for trends in research and code in AI. Their goal


is to become your research assistant and streamline how you organize, share and stay up to date


on the latest R&D. This is really cool because the flood of papers in machine learning is


sheer overwhelming in recent months. Zeta Alpha uses neural embedding based search and


can give you the best recommendation of research that matches your interest and that you don't


want to miss. And what better way than to just try it out. So first I start off searching


for today's paper, which is the blip paper. And this is really cool because not only do


I get the paper, I also get the GitHub code implementation. And I can directly see the


impact on social media that this paper has. This is much better than something like Google


Scholar, which would just give me a few links to the paper itself. I can now save this paper


under a tagging category that I'm just going to invent right now. And I can use Zeta Alpha


to find similar research. Here I'm going to limit my search to the last three months.


So I make sure that I don't miss anything that has recently been going on that I should


know about when reviewing this paper. Now I also like a bunch of those other papers.


So I'm going to save them as well to the same category. Once I have a bunch of papers in


my category, I can use again as Zeta Alpha's recommendation engine to give me more suggested


papers to add to the same category based on what I have already in there. And I can also


share this entire category with my teammates because everything Zeta Alpha does is not


only for individuals, but also for teams. This is really powerful and can dramatically


accelerate your discovery of new and relevant research. Now this doesn't only work for categories


that you define. Once you interact with the search engine, Zeta Alpha is going to be able


to give you a list a feed of recommendations from archive from conferences from blogs from


GitHub and much more. This saves you a ton of time and lets you stay up to date with


whatever is happening. If you're at all into ML research, this is hyper relevant for you.


And I definitely invite you to check it out. Now they do have a free tier, but I got you


a great deal. If you go over there right now and use code Yannick, you'll get 20% off a


personal assistant subscription. Again, go to Zeta dash alpha.com use code Yannick for


20% off right now. Thanks again so much to Zeta Alpha for sponsoring today's video. And


now let's get into it. See ya.


Hello there today we'll look at blip bootstrapping language image pre training for unified vision


language understanding and generation by Junan Li, Dongshu Li, Taiming Xiong, Stephen Hoy,


yeah, that's it of Salesforce research. So this paper proposes two things. One is a new


architecture and I want to say a new conglomeration of existing things. So an arrangement of modules


for multitask pre training. This model will take in an image text pair and perform multiple


tasks on it, it has multiple losses, and therefore ends up being able to do multiple things.


Now that being said, this is a pre training method. So the idea is that for any of these


modules, you'll take them, you recompose them downstream, and you fine tune them on a task,


although they do have some zero shot results. So this is one thing and this could be really


cool if this alone turns out to be successful, because it leads the path to a future where


we have much more dynamic compositions of models, and where we would pre train these


models with a lot of different tasks in in one thing, rather than pre training them on


on just a single task like language modeling. The other thing is a bootstrapping method


for the data and these two things are not necessarily disconnected, although I do lament


the fact that it's two things in one paper a little bit. But there's a bootstrapping


method for these image text data set that includes training captioners and filters,


which means that this there's a part that learns to synthetically generate data. And


then there is a part that learns to distinguish good from bad data. And that allows them to


collect lots and lots of data from the internet and filter out bad, badly poorly labeled images,


which there exists a lot on the internet, and also allows them to augment the data set


by labeling images themselves. So this is also really interesting. And it feeds really


well back into their model because their model is uniquely capable of doing this being the


multitask model that it is. So we're going to go through the architecture through the


data set bootstrapping method. And keep in mind that I think, you know, if this catches


on this could be there could be recipes in here for future research that lead us to a


much more dynamic world, where we compose these modules, much like we compose different


modules, low level modules and deep learning, we could compose these higher level modules


and losses and do lots more multitask pre training, maybe even dynamically configured.


But let's dive in. So vision language pre training, they say, as recently recently been,


you know, the hit, for example, if you think of something like clip, and that's not even


pre training, but there are lots of architectures that do vision language pre training, meaning


they take pairs of images and text. So you'll have like some sort of an image, and you'll


have like some sort of text that goes with it. And you'll try to come up with a system


that connects the two in any way, they say the major the existing methods have two major


limitations. So first of all, the what they call the model perspective, they say they


are either the existing methods are either encoder based, or an encoder decoder architecture.


So in an encoder based setup, what you would do is you would take in both of these things,


and you would try to come up with probably a number that represents how well they fit


together. So are they good together or not? This is the clip architecture, essentially.


So in encoder based models, they criticize that encoder base are less straightforward


to directly transfer to text generation tasks. So it's not it's not simple to take clip and


actually make it produce something. Remember, if we have to, if you have to produce an actual


image with clip, we need to do this diffusion clip guided diffusion or clip guided GANs,


VQ GANs. So it's really cumbersome to make clip generate an image. And it's probably


even more cumbersome to make it generate text because it's not trained on that. So they


criticize on these methods, it's not easy to make them do generation tasks. Whereas


encoder decoder models have not been successfully adopted for image text retrieval tasks. So


an encoder decoder model is where you would take the image probably, and then make the


produce the text. So you train it as a language model to autoregressively produce the caption.


And that's really neat for producing captions. But you cannot necessarily do this task up


here very easily with such a model. You will you will be able to do some things, but they're


not necessarily successful, because the task is really a different task. So both both approaches


for doing this currently are not ideal. The other thing is the data perspective. They


criticize that these models are pre trained on image text pairs that are essentially scraped


from the internet. So collect it from the internet. And they say noisy web text is suboptimal


for vision language learning. We've known for a long time that there is a trade off


between scale of data and quality of data. And ideally, you'd have both if however, if


you scrape from the internet. So let's say you scrape websites, and there is like some


text and there is an image somewhere, and the image will have alt text. And that's what's


usually used as the label in these systems. So if you don't know in the HTML, if you have


an image tag, that's how that's how the browser knows it's an image, you have the image tag,


you have the source attribute, which leads, it's a URL, usually that leads to the image,


but then you also have an alt attribute. And it's really recommended that you put an alt


property to the point where frameworks and linters and so on, they will yell at you if


you don't have it. So what does this do? This specifically is for visually impaired people


for screen readers, but also for bots to know what is in the image. So you put a description


there. However, a lot of people don't do that. And I think it makes it actually worse that


linters and so on almost require you to do it. Because if you don't want to do it, you're


just going to put like some some dumb stuff there like image, or people do lots of search


engine optimizations in there. So since you know, the search engines don't usually look


at the image itself, but at the alt text, they try to come up with buzzwordy things


so that it's ranked high in search results. So not necessarily the best quality data.


And their bootstrapping, their bootstrapping method right here is helping in that of getting


higher quality data out of the internet. So how do they do this? The first thing they


propose is this model, the multimodal mixture of encoder decoder. They say it can operate


either as a unimodal encoder, or an image grounded text encoder, or an image grounded


text decoder. So yeah, we're going to look at these things. But I think here they say


it can operate either as one or this or that. It's not like this. It's not like that exact


same model can do this. It's just that they put all of these models into one big model.


And then they just use the part of the model that does the particular thing. So it's not


necessarily super duper unified is what I wanted to say. Yeah, they train the three,


the three sub parts of their models with three objectives, which we're also going to look


at. The second part is this captioning and filtering. This is what this is what boosts


the data set quality. They say they learn from noisy image text pairs by cleaning them


by producing more and cleaning them. They train a captioner, which whose goal is to


produce synthetic captions given web images, and a filter to remove noisy captions from


both the original web text and synthetic text. So the captioner will get images, produce


labels for these images or produce alt text. And then the filter goes over both the generated


ones and the collected ones, and just filters out everything that it deems to be qualitatively


low standard. Of course, this needs to be trained on a high quality data set. But these


sort of bootstrapping methods we've seen a number of times in the recent past that they


actually work. In fact, this model, this paper here seems to be a good accumulation of zirconic


of recognitions and good practices over the last few years. And we're going to point those


out as we go through their their contributions. Here they say, we show that the captioner


and the filter work together to achieve substantial performance improvement, which, okay, I don't


know what substantial means in these kinds of tasks, but it's I, it's an improvement.


They are they achieve state of the art performance in a wide range of vision language tasks.


And interestingly, also, this is a property of maybe synthetic data generation, they show


more diverse captions yield larger gains. This might also be a good lesson for people


who want to go and apply these methods. Lastly, they say, next to having state of the art


in downstream fine tuned tasks, they also achieve zero shot performance when directly


transferring our models to two video language tasks. So never, they were never trained on


video language tasks, never pre trained, never fine tuned, yet still, they have a good zero


shot performance, which is okay, like if you understand images, then there are going to


be some video tasks that are your that you're particularly good at, right. So let's dive


into the model. And I've already shown you a diagram of the model, they quickly go through


this here, they have three parts, they have actually, well, I want to say four parts to


their model. One part one is a visual transformer, a VIT as the image encoder. So again, they


take an image and they take a piece of text, and now they do stuff with it. And the first


part is they encode the image using a visual transformer. That's all they do with the image,


they encode it using a VIT. With the text, they do three different things. The first


thing is they also just encode the text unimodally. So put the text through an encoder. And that


with those two things already, they have essentially reproduced clip. Except they say it's the


same as BERT. Yeah, so they've reproduced clip with those two things, because now they


can set it up this visual transformer, and the unimodal encoder, they can set it up as


a similarity metric. So the unimodal encoder will give you some vector in an embedding


space, the visual transformer will give you some vector in an embedding space, you can


set up a contrastive loss to check whether these two things go together, and whether


they are apart from, let's say, any other encoded image or text. You can do this via


contrastive learning, you can do it via regularized methods. But essentially, this is what we've


come to known as encoder only models. The second thing they have is this image grounded


text encoder. So the image grounded text encoder does almost the same thing as the unimodal


text encoder. However, it doesn't encode the text separately, it encodes the text while


incorporating attention into the visual transformer. We're going to see how that goes in a second.


But essentially, it produces a vector, let's say this one. And while producing that on


the path as it produces that it incorporates information from the visual transformer. So


it will, this here is the output of the visual transformer, it will incorporate that at multiple


layers here via cross attention into the process. So this here is really a joint kind of encoding


of the text, given the image. That's why it's called image grounded text encoder. What this


can do is you can build a classifier on top of this, like a binary classifier, because


it is a representation of the text that has but that has already the information of the


image inside of it. So it's kind of a joint representation of the image and the text.


So you can build a classifier, for example, whether or not the two things go together


again, but you don't have to use a contrastive loss, you can in fact use a supervised loss


and classify and build a classifier. The third thing is this image grounded text decoder.


Now again, being image grounded, that is a that is a long what is going on. Some things


up here. There's an image grounded text decoder. The image grounded text decoder is much like


the image grounded text encoder in that it incorporates cell across attention. However,


it's a text decoder. So what it will do is it will actually produce text. So it will


auto aggressively produce the text while incorporating again, information via cross attention from


the visual representation. You can see that they have a different section on the pre training


objectives. These just map to these three parts. So there's the image text contrastive


loss, which is the loss for the first part, there is the image, the image text matching


loss, which is the loss for the second part. And again, this is just a binary classification


task where the model uses a linear layer head, what they call it an ITM, an image text, text


matching head, but it's a linear layer to predict whether an image text pair is positive,


which means matched or negative unmatched given their multi model feature. The special


thing here is they do have a hard negative mining strategy. So they go to the top part


here, they go to the joint, no sorry, to this joint encoding to this part, and they look


which ones are the hard negatives, which means that negatives that have a high contrastive


similarity, and they use those specifically to train this loss here. The last loss is


a language modeling loss, which is obviously relevant for the third part. This is a cross


entropy loss that maximizes the likelihood of the text in an autoregressive manner.


If we put all of this together, we get this model right here. Again, if we go through


it, the input data are two things, the input data are the image down here, and the piece


of text here. Again, we know these go together because we've scraped them from the web. So


these two, we know they go together. This is not an unsupervised training. This is essentially


supervised learning for two things that we know go together. The first thing is we're


going to encode the image through the image encoder. That's the image encoder. This is


the image representation. This is just a bit, this is a visual transformer. I don't know,


I don't think they freeze it, but they may start from a checkpoint. All of this is jointly


trained. So all of these losses, as I understand them, are jointly trained. So then we have


the vision representation. What we can do is we can put the text, first of all, through


the text encoder. You can see we can append different tokens right here to let the encoder


know what we're currently doing, because we also have some parameter sharing going on.


So the text encoder gets the input text. It will also compute an encoding. And then we


have this contrastive loss between the two encodings. They need to be close for pairs


that we know go together, and they need to be far apart for other pairs. You can do something


like in batch negatives, or you can, as we said, mine hard negatives from the contrastive,


from this part. Well, that makes no sense. You can mine contrastive, so you can mine


hard negatives for that part over here, given this part over here. Which makes me believe,


okay, maybe I haven't read closely enough. Maybe they also just train one of the losses


maybe for each batch, because they have to sample differently for the things. It doesn't


make too much of a difference whether they train it really all jointly, jointly, or always


activate one of the three text pathways. This would be interesting to figure out. Yeah.


So the last thing, the second thing they do is they give it to this image grounded text


encoder. Again, this gets the text and a little token to show what's going on. It will encode,


and now you can see that it has this cross attention module. And the cross attention


module, as it encodes, it incorporates information that comes from all the way over here, that


comes all the way over here from the image. So the image representation is part of the


encoding here, which means this thing has information about both the text and the image.


Now yeah, of course, it's still a, it's still, it's not symmetric, right? We don't, the joint


encoding is asymmetric in the sense that it is the text that is encoded based on the image,


and it allows them to only compute the image representation once. So they only need to


do this pathway on the left here once, and then they can reuse that representation for


all of the, for all of the different paths in the text here. Yeah, you can see that on


the left, this is the difference on the left here, this is skipped, the cross attention


is skipped. We don't have cross attention, it's just an encoding of the text itself.


And here, it's really a joint encoding, which means that this thing here contains information


on both the image and the text. And we can perform any sort of task that we want with


this joint encoding. In our case, we simply train it on a very similar objective as the


contrastive loss in that it's a binary classification, it needs to figure out whether or not the


two things actually go together or not. The third thing, again, almost the same, is this


decoder, the text decoder, same input, except there's a little decode token. There is a


difference in that this is bi-directional, the other two modules have bi-directional


self attention, because they are encoders. So they get to use bi-directionality. Here,


we use causal self attention, which essentially means that in the text, you only get to attend


things. So if you produce a particular token right here, you only get to attend to tokens


that are behind yourself. This is a bit of a hack, because otherwise, we couldn't train


these things with batches or in parallel. It is definitely possible to use bi-directional


self attention as long as you cap, as long as you mask whatever comes next. So you want


to mask sort of the future, but within the past, you could totally use bi-directional


self attention. Again, this is just a hack to make training easier. But it's become,


it's come to be a popular hack. So everyone's doing it. Again, you can see there's cross


attention coming from the image. And here you can really see that it's necessary, right?


If I want to actually produce text, I need some sort of information of what I want to


produce. And so this language modeling loss here really needs the cross attention, really


needs the input from the image. So again, this comes from here, from the image representation.


So there you have it. It's an unholy concoction of many different things in one. And this


is all trained jointly, right? And yeah, I'm excited about this, because I think not necessarily


this particular arrangement, like I have lots of stuff to criticize or, or like lots of


choices here that are kind of arbitrary. Like, why is this asymmetry in, you know, I have


the image encoded once and I have cross attention into all the text encoders. Why not the other


way around? Why don't we do image generation tasks? Why don't we do any sort of masked


modeling, like masked language modeling? This could even be in the image. There's lots of


stuff, let's say to criticize. But I think what this thing shows is that a good recipe


for the future could be to combine lots of these different methods together, combine


lots of them into one big thing, reusing parts intelligently, and then train them jointly.


We could even think of frameworks that do this automatically, or that allow you to really


easily set this up with a few lines of code, and it will figure out by itself, like the


framework would figure out itself, how, what it can compose and how, how it could reuse.


What you can also see right here is I've overshadowed it a little bit with my thing right here,


but there's color, and the color indicates shared parameters, which is also really interesting.


So you can see that essentially the text encoders aren't three separate encoders, but they largely


share parameters. For example, the feed forward parameters are shared, the cross attention


parameters, they're all shared, except of course, they're not active in this encoder.


The bi-directional self attention parameters are shared. The causal self attention, those


ones are separate over here. But if we had some sort of other autoregressive, autoregressive


module, they would be shared too. So you'd share whatever you could in these architectures,


and that reduces the overhead, but also in their evaluations really helps, which I guess


makes sense. Well, I don't know, if the tasks are too distant, you might get this catastrophic


forgetting, but in their case, it does help. Yes, which I can, I can, I could guess, right,


by for example, the bi-directional self attention right here, since these two modules are almost


doing the same task, it's, it's reasonable that they would share parameters. So we've


gone through a whole lot of things that they say down here. They do reason through their


choices a little bit, even though I think, I think these choices, they are either arbitrary


or they're guided by experiments, you know, just seeing what works better. They do bring


up some hypotheses of what they think, you know, why, why do things work and why do things


don't work? They say the text encoder and decoder share all parameters except for the


self attention layer. The reason is that the differences between the encoding and decoding


tasks are best captured by the self attention layers. So they're essentially saying that


whether you want to encode or decode, that is mostly going to be different in the attention


layers, not from the architectural perspective, but from sort of the how the task is done


perspective. And that I don't, I don't think necessarily you can say this, right? Like,


you can't necessarily say the feed forward layers have a similar job in or have similar


features and perform similar functions, whether you're encoding or decoding, I don't just


don't think that's out of the box, really evident that we need to be supported by evidence.


So yeah, but it seems to work well in empirical evaluations. And so I'm going to, I'm going


to, I'm with them sharing the parameters, but the reasoning are more hypotheses. So


the second part they go into is this cap field. Again, this is a bit disconnected, although


it plays well into their model. Here they criticize how these data sets are usually


collected. They say alt text often do not accurately describe the visual content of


the images that are scraped from the web. And that's why they have a bootstrapping method.


So what they do is they collect a data set from the internet. And yeah, well, I find


this diagram here to be a little bit complicated. So we're just going to make our own. So they


they have the internet, I'm going to this is a globe with, you know, the lines and so


on. So we're going to collect a big chunk of data of pairs of images and text, images


and alt text from the web, really noisy. And what we're going to do with this stuff is


we're going to train a first blip architecture or a first, not how they call it, MED architecture,


multi something, something, whatever their model is on top. We're just going to train


that with this noisy data. And that's going to be our first iteration model. Now this


is really noisy so far and so on. But what we're going to do then is we're going to fine


tune this. We're going to fine tune a filter and a captioner. So we're going to fine tune


a filter and a captioner on supervised data. There exists some supervised data sets. And


one of them I believe is the Cocoa data set. Yes, the Cocoa data set. So this step here,


we need supervised data and supervised data of image text pairs. So human made captions


for existing images, which it's a sort of a proxy for quality. So of these things, we


can be sure that the quality is relatively high. If we could find some sort of an automated


way to get really high quality image text pair data, it doesn't necessarily need to


be human labeled. It just needs to be high in quality. So they use that to train a filter


and a captioner. Now what is the filter and the captioning model? Now these are going


to be fine tuned versions of their MED models. For example, the captioner takes in an image


and gives you a caption, a synthetic caption. Now this is something our model can do. If


we, you know, we just take two parts. So we take this part and we take this part right


here. This is now a captioning model. So the idea here, the general idea of blip of this


MED model is that we pre-train all of these things together and we sub-select or we rearrange


even the different sub components and then fine tune them on a downstream task. And one


easy way is to take two components, simply deactivate all others and let them run in


inference mode. So now we have a captioning model. The captioning, the filtering model


on the other hand, very similar, but it takes an image and a piece of text both inside and


it will output a score of whether the two things go together or not. Now this, of course,


we can achieve in multiple ways, but we can achieve this in the probably the most high


quality way by taking the image encoder and taking this part right here that is specifically


trained to jointly encode. You might ask, why don't we use, why don't we use this module


right here and then use this contrastive estimation? We could also do that definitely, but usually


there are always multiple ways of determining similarity. You can have sort of the two stack


encoder. So here is the image and here is the text. You can have separate encoders for


them and then at the end determine whether they go together. And that's usually good


if you want to do something like a search index, because you can pre-compute a lot of


these things. You can pre-compute all the embeddings for the images and then at inference


time, if you have a query using text, you want to search an image via text, you only


need to encode the text. Whereas with a joint encoding, it's really different. You need


to input both into the encoder and that will give you a score at the end. And if you want


to build a search engine like this, then for every single time you issue a query, what


you need to do is you need to go through the whole dataset and encode the query here together


with all of the images, get the score for each one and then evaluate that. So you can


see there is a trade-off. The left side is way friendlier computation wise if you have


an existing dataset. The right side is qualitatively higher because during computation through


these layers, the two things can already attend to one another. Whereas really the only interaction


here is the end over here. So this is qualitatively better estimate of whether the two things


match or don't match. And that's why we're going to have the filter here. Since we're


working, since we're filtering the dataset, we can jointly encode the two things anyway.


So we're going to fine tune that part to become our filter. So now we have a fine tuned part,


one captioner, one filter. What can we do now? Well, we can take our dataset, this thing


right here, and we can use the captioner to produce another dataset by just taking the


images. So we just take the images here. We put them through the captioner and we get


another dataset. So we get another dataset. It's going to have the same images, right?


But it's going to have different texts. So this is a synthetic dataset. We can then join


the two datasets together. So join the two datasets, and then we can put them both through


the filter. So I'm going to put them both through the filter and the filter will simply


filter out any image text pair that is not adequate, which means that it will filter


out any image text pair which doesn't match well together given the fine tuning of the


filter on the supervised or high quality dataset. So then we end up with a dataset of, and we


can restrict it like to only have one caption for each image or something like this. And


we end up with a dataset of image text pairs, which is large because we've augmented it


with synthetic data, but also is of high quality because we have done the filtering. Now that


all of this being said, again, this highly relies on the quality of the dataset that


we fine tune on and of the diversity of that dataset as well, because you can also imagine


if that dataset isn't containing much of the domain that you're looking at, then your filterer


will learn to essentially down rank everything because it says, well, my dataset says these


two things don't go well together because I actually have just no data in that region.


So there's a bit of danger in doing this. You really need to pay attention at what dataset


you're fine tuning, but this is how you bootstrap a good dataset. So you can see, go from here


to here and you can think of multiple things. Again, I think this paper is less about the


particular method they choose. And I think more about, you know, what could be recipes


for the future. And I think in the recent times we've seen a lot of synthetic data generation,


first of all, being really helpful. We've seen this in a number of reinforcement learning


applications, a number of even NLP applications. So synthetic data is really, really picking


up, I want to say with advances in SIM to real and so on. And then also this approach


of filtering. This has come up more and more in recent years where generative models are


paired with discriminative models that either re-rank their outputs or filter their outputs


for quality. This seems to be a very good recipe for achieving generative tasks in general.


Not only train a generator, but train a ranker or filter on top of that. It's pretty computationally


efficient. It's easy to implement. And yeah, I think it's a good recipe for the future.


And one can think of various ways here to improve this, like to do this bootstrapping


multiple times. Yeah, to collect the supervised data set in a different manner and so on.


I think there's a lot of possibilities here that are not yet explored, which I find to


be pretty, pretty cool. So that's essentially all. Yeah. Okay, no, I was actually wrong


here. You can see the filter is actually fine tuned on both of the objectives to learn whether


a text matches the image. So this, it's both the contrastive and the single classifier


loss. Although I do think, I do think the filter, like what they actually pay attention


to at the end is going to be this thing right here is going to be the classification head.


But I guess it doesn't hurt to use both losses as you fine tune it. And since all parameters


are shared, essentially, you really don't have, you really don't have, you can, like


it's, it's easy to try and it's not too much of an overhead. So that's the methods. Again,


they have this concoction of modules that they all pre-train jointly with their respective


losses. And then on the other hand, they have this bootstrapping method where they can directly


use their model, right? That's the way these integrate these two. Since they have a model


that can do all of these different things, they can fine tune that model to become a


filter or to become a captioner. And the same thing holds for the results downstream. Here


they have some examples, by the way, of generated. And so the bottom text is always a generated


one. The top text is one from the data set. Anything that's red is filtered out by the


filter. Anything that's green is accepted by the filter. Yeah, so they, they also discuss


a little bit of the dangers of doing this, of training the filtering and the captioning


from the same pre-training state on the same data set, which is that, like there is some


going to be some confirmation bias in that the filter will uprank things that the captioner


produces because they're essentially learn from the same data. That's why they don't


share, they fine tune them separately to combat this a little bit. But I still think that


you're going to have some of that in there definitely. But, you know, it's, this is,


you know, this is a real data from bridge near my house, which might be true, right,


but it's not very descriptive and the filter realizes it. Yet a flock of birds flying over


a lake at sunset, that's pretty descriptive. Another interesting thing is that they use


nucleus sampling here, which is a common strategy, but they do find that using nucleus sampling


leads to better performance and that because it generates more diverse and surprising captions,


which contain more new information that the model could benefit from. They compare this


to beam search and beam search essentially goes for the highest likelihood sample. It


tends to generate safe captions that are common in the data set, hence offering less extra


knowledge. I think that's also really cool recognition right here that if we sample things


from generative models, we might have different goals and therefore it might not always be


good to, like it might be good to have an objective or a sampling method that encourages


diversity. We've already seen this in alpha code and my question there was already a little


bit, do we even have the correct training procedures for this because we train maximum


likelihood or do we have the correct sampling procedures for this? All of these are interesting


questions and I think this kind of research validates that it's not all the same, like


depending on what we want to do, our training and sampling procedures need to adjust. I


don't want to dive too deep into the results. They are outperforming other things by some


margin. I don't necessarily agree that they outperform things so heavily as they advertise,


but that's research currently. Again, they allude to the fact that they share parameters


here and why that is. They say sharing all the layers except for the self-attention leads


to better performance compared to not sharing. That's the part I believe, right? Totally


you share, numbers go up, good. But then they say if the shared attention layers are shared,


the model's performance would degrade to the conflict between the encoding and the decoding


tasks and this, I think, yeah, this stuff needs evidence because, I mean, yeah, I'm


fine with just going with the numbers. Here you can see the various ways they combine


the things, for example, for visual question answering, they first encode the image, then


they feed that to the text encoder, then they feed that to the decoder. So you can see,


you can not only sub-select modules, but you can rearrange them, right? Because you fine-tune,


you can adjust the parameters. So this connection already exists in the previous model, but


this connection doesn't. So you can sort of rearrange and recombine these modules to do


various things. You can see here we have two image or a double image encoder, or I guess


the image encoder just gets two samples. And then we also have two, one, a duplication


of these cross attention modules, and then we output that into a newly trained merge


layer. So this is the exciting part right here. And I feel really, I don't want to necessarily


go into this because we might go into this in the interview, but I feel a future where


we have frameworks, coding frameworks, where this kind of stuff could be supported in an


automatic fashion where I don't have to, you know, go and really hand define exactly how


I want these things combined, but I could have a more high level descriptive language


that allows me to do this whole pre-training arrangements and this recombination for downstream


fine-tuning. That's really exciting. All right, I'm going to leave it at that. I hope you


had a good overview. If you want to dive into the results, you know, feel free. There's


lots of tables in here. It's a really thorough evaluation, which is really cool because it


lends a lot of credence to their methods. And with that, let me know what you think


[ML News] Uber: Deep Learning for ETA | MuZero Video Compression  | Block-NeRF | EfficientNet-X

Uber now uses deep learning to predict arrival times, mu zero is used to compress YouTube


videos and nerf scales to entire city blocks.


Amazing.


Welcome to ML News.


Hey ho there, this video is sponsored by Weights and Biases.


Today I want to tell you about a new feature in Weights and Biases, which is their integration


with the OpenAI API.