And exploiting that data to me, when I saw this, it made perfect sense to say, you know,


CM3: A Causal Masked Multimodal Model of the Internet (Paper Explained w/ Author Interview)

we should just keep the HTML and just learn the language model over the HTML.


Right?


So what can you do if you have such a language model?


Well, if I have trained such a language model, I can maybe, you know, start a paragraph,


start a paragraph, I put like a piece of text right here.


Right.


And then I just start an image tag.


And I say source equals, and then I let the model generate whatever is here.


Now, there is a trick right here, I can't obviously put a URL, I actually have to put


the image itself there.


But if the model is good enough, it will look at this, it will generate an appropriate image.


Or you know, I could do the same thing by simply having an image tag.


And first generating the alt, first putting the alt text, I put something here that I


want, and then source, and I say equals, and then I let the model continue.


It will generate me an image, I can reverse that, I can put the image first, and then


say please generate me the alt text, I can put an entity and say please generate me the


link to the entity, and so on.


So you can see how powerful this is.


We can do many, many different tasks if we have a model like this.


This is one thing that this paper does.


And I said it's inspired by previous work.


However, it pushes it a bit further.


So first, we have to discuss this, and then we have to discuss the novel objective, which


makes it even more powerful.


The only thing to discuss right here, actually, is how do they treat images?


Because language modeling is fine, I can just have an appropriate tokenizer for HTML, which


needs to be, I guess, a little bit of a different tokenizer than for regular text, because you


have to handle these tags correctly.


But essentially, I have to have a tokenizer.


And transformers are pretty good at learning to open sort of appropriate tags, and then


close appropriate tags again, and so on.


The only part really are the images.


So we don't want to have URLs of images in there.


Instead, what they do whenever they encounter an image tag, so whenever they encounter image


with a source that equals some URL, www dot something, what they do is they would go,


they would fetch that image, they would put it through a, I think, a VQ GAN model, some


vector quantized GAN model that is pre trained.


They would extract the latent embedding from that, and they would put that embedding here.


So these models, these vector quantized models, they would take some image and have like a


neural network, and they would encode that into a series of tokens, which are going to


be something like, I believe it results in 256 tokens, latent tokens.


So these are essentially because it's vector quantized, every one of these is part of a


vocabulary.


And so these are essentially tokens, like language model tokens, like letters that I


can build images from, I can simply unroll, oops, I can simply unroll the tokens in these


images that the VQ GAN gives me, right, I can have some scheme of how I go through here.


And I can replace the source property here, just with the, with these tokens, or, I mean,


appropriately, the embeddings of these tokens, all right, this, and this goes here, and so


on.


So once I have these tokens, right, I can train the language model, and then the language


model will generate these tokens again, again, they're not continuous values, because it's


a vector quantized model, they come from a fixed vocabulary, and that's what I ingest,


and that's what I predict.


And therefore, I can treat it exactly the same as the language model.


There is a bit of a difference with how these things are distributed.


They do talk about this in the paper, as language tokens are Zipfian distributed, and image


tokens are by design uniformly distributed.


But I mean, essentially, from a conceptual standpoint, it's the same.


The second thing they do is they have a different objective than language modeling.


Language modeling goes usually goes left to right.


So that means the language model whenever it generates a token, it looks at what is


generated so far.


And then from that will generate the next token.


And what it cannot do is it cannot look at the like right, like the the head, it cannot


look ahead, you can't tell it, you know, here is a piece of text, and here is a piece of


text, please fill in this piece of text, that would be a masked language model like BERT.


But some a model like BERT isn't really good at auto regressively generating text for that


the left to right causally masked language models are much, much better.


And you know, higher, higher performing.


So is there a way we can get the best of both worlds, or at least some kind of a trade off?


Turns out, yes, there is with the following objective.


So as I said, we have an example right here in a standard language model, we have the


following the following thing, which is a way we can do entity linking, right?


So imagine, imagine we'd have to predict this piece right here.


So as you can see, this is the link, it's an anchor tag, this is the link to the page,


the Wikipedia page for American, sorry, Armenian, Armenian nationalism.


So Armenian nationalism, we want to predict that link, which is essentially solving entity


linking for this sentence.


If we only have a causally masked language model, all we can do is input this piece of


text to the left.


So this would be our entire context.


Now, this example is constructed such that this thing right here, right, this word right


here is really important to classifying to seeing what is there.


Therefore, if we only had a causally masked language model, if we only ever trained left


to right, we couldn't make use of the word that was behind right here.


If we had something like a masked language model, we could absolutely do that.


So that is this example right here.


If we had a masked language model, then we could absolutely, we could do that.


We could input this, and we could input this, and we could say, you know, here is a mask


token, please generate what's in the mask token.


However, we already discussed the weaknesses of that approach.


Instead, they have a new objective, which they call a causally masked language model.


But I called this before causally masked language model, because there's also this sort of causal


causal mask inside of it.


I'm sorry, the causally masked language model is the thing they are going to propose inside


of these language model, usually there is something like causal masking.


So it's a it's a bit confusing.


If I look at this right now, what they do is during training.


So during training, what the masked language model would do is it would just mask out these


parts, and then it would try to fill them in.


This limits training, because you can only mask out so much, you can't train in parallel


and so on.


Whereas with the autoregressive language models, you can train a lot of stuff in parallel,


there is no none of these noise and so on.


Everything's everything is decomposed nicely.


Here what we will do is we would take the things during training, we would simply have


a span that we mask, but we don't just leave it away, we actually put it at the end.


So and there is an identifier token right here to show you can see that this token right


here and this token right here are the same.


So we tell the language model we tell it look here is a sentence, okay, there is a mask


right here, there's something missing, it could be one or many tokens.


And then here, we want you to generate that thing again.


And the model simply has to generate the thing back here.


There can be one mask tokens, there can be many of these mask tokens, in which case we


just, you know, if we mask something else like this right here, we just put the corresponding


token right here and ask the model to generate it on the model will learn if there are two


mask tokens, the model will learn to after it finished the first thing that it's supposed


to produce to automatically put a the next mask token there.


So that is that is the objective, it still benefits from this left to right thing.


As you can see, we can train this left to right.


Once we reorder the sentence, we can just input the whole thing here into training,


we can train it like a decoder only language model.


And we get all the performance off of that.


Yet we can still do kind of like masking.


So we get by directionality by design, because now if we want to predict this mask right


here, we have seen all of this context.


So essentially, we have seen the whole the whole data point, we do sacrifice, like a


little bit of performance, because, well, inherently, this part here is still left to


right.


So that there's that, like in itself, it's still left to right.


Also, we do take stuff out of order.


So there is the question of, you know, how long can I memorize stuff and so on, with


transformers, maybe a bit less, but we do take stuff out of order, which introduces


some noise and so on.


So it is definitely a trade off wherein pure language modeling is still going to be more


powerful.


But this now enables us, this enables bi directional context, essentially into the things that


we generate.


And that has a lot of advantages for many, many different tasks.


There is a whole scheme, it's it seems to be really important how exactly, oh, yeah,


256 tokens for each image, see, sorry, it seems to be quite important how you generate


these masks during training, how long they are, they try to make them quite long in order


for the model to learn important structure and so on.


We'll go through all of this in the interview.


The scaling laws are pretty, pretty astonishing in that their large model right here and these


are large models, right?


These are like the scale of this.


It was trained, it was trained on 384 a 100 GPUs.


No, I think that's even that's the base.


Is that the baseline?


That is even the baseline?


Where is the where is their model?


Yeah, I don't I don't currently I don't currently find it.


But you can just see sort of the scale here of what they're going for.


So this is these are not small models.


But if you make them sufficiently large, you can see that largest models, they're not done


training yet.


Even after they put sufficient or put enormous amounts of resources through them, you can


see they're not even not even the same ahead, like the same advanced inside of the training.


So yeah, this is it is very promising.


I think this is a very promising direction to make use of that to make use of the HTML


structure.


You can see a little bit here.


So essentially, if you just put this as a prompt, you can have the model generate the


alt text and the image at the same time, right?


It interestingly chooses to put the alt text in front like it chooses to generate a little


description before it generates the images, which is interesting.


You can also force it to first generate the image by just putting the source tag directly.


So then it needs to generate the image.


And it's interesting because the quality of the images when you force it to generate image


before alt text, it is a lot lower, as you can see here, than if it just if you just


let it generate the image, in which case it chooses to generate the alt text first, you


can do many things.


You can do image inpainting by masking out a portion of the tokens of the image, you


have to mask out entire tokens.


But still, you can do like crude image infilling, you can do conditional infilling by providing


alt text first, and then do infilling, you can you can do conditional generation by providing


alt text.


So the the like the the possibilities are very, very great right here.


You can see this is infilling conditional infilling, and so on.


The possibilities are great.


And remember, this is a very particular data sets and very particular cleaning methods


of HTML, I believe if we extend this to even more structure, and so on, maybe even take


cascading style sheets into account, take all of the structural elements of websites


into account, title tags, headers, footers, and so on.


This could be could be really powerful beyond the applications that we see right here.


They can also do text, pure text modality data sets, as we said, entity disambiguation


by predicting hyperlinks.


They also do get new state of the art in summarization and zero shot summarization by simply generating


like the they simply generate like the title or the the meta tag, the description tag of


the website to give it a fake website with the text they want to summarize and they generate


these tags.


They do say for completeless below is an example of a prompt that can do basic summarization.


I did not find that prompt anywhere.


So yeah, maybe I didn't I didn't look enough or maybe law text screwed up where some kind


of a figure is.


In any case, I don't want to go too much into the results right here.


But I think the direction of using that structured content is pretty cool.


The new the new objective is also pretty cool.


I do criticize a little bit that these two things are kind of decoupled from each other,


like they could all be their own paper.


And that's also something that we talked about in the interview.


So in the interview, we're going to go briefly over the model again, over the research process


over what it means, what it could enable, and what difficulties there were and also


over the results, which are extremely, extremely interesting.


I enjoyed the interview a lot.


I hope you do to tell me what you think of it.


And now I'll leave it up for the interview.


Thank you very much and have fun.


Welcome everyone.


Today I have with me Armin Aghajanyan and I've practiced that name 10 seconds ago and


I think I got it down.


Armin is the first author of the CM3 paper.


Welcome Armin to the channel.


Thank you for having me.


So I saw this paper and of course you have like some big names here.


There's lots of authors, there's Facebook AI research.


But still, like given all of that, it was still impressive.


Like I was impressed by the what it could do and sort of the results it gave.


Like it seems to be, well, there's zero shot, there's image generation, there is like a


new objective, there's HTML in there.


So there seems to be a lot in one pot.


If you gave the pitch, right, I will have made an introduction, but if you gave the


pitch to the paper, what is it mainly about?


I mean, the goal here was kind of to have a single multimodal model that can do everything.


Image generation, image captioning, image infilling, to even pure text tests, like summarization,


but mostly focusing on this zero shot setting, specifically this popping setting.


And how did you like, were you, this is a very popular thing, I think in the last few


years, this came up, maybe starting with something like GPT-3, where people could really say,


okay, stuff is possible zero shot, if we train on large enough data, then came things like


Dali and so on, where, you know, we saw for the first time, okay, maybe stuff is even


possible in other modalities than text.


This goes even further, this is multimodal.


There have been a lot of other approaches to multimodal.


There is like this, this Rudolph even model, I don't know if you've seen that, that goes


like image to text to image and so on.


And they all work, let's say with very cleaned up data.


It's very, you know, I want text, I want images that go with the text, which makes sense,


right?


How do you get, how did you get the idea to use, let's say, relatively unstructured HTML


for this?


How did your thought process go until you came to this idea?


So usually, there are pros and cons to having super strong alignment, right?


So like Dali, for example, they have like a very specific alignment of like, you know,


text on the left side, and then you have like 1024 image tokens on the right side, right?


Super strong alignment.


And in general, it's easy for the models to kind of learn this type of single alignment,


but then you're incredibly limited on the prompting side.


And I think it's prompting is, I think it's incredibly creative.


It's kind of, if you have a general model, it takes a little bit of creativity to extract


out the prompt.


So the key here is we don't want to have any strict alignment in terms of the modalities.


So the goal was like, what is the weakest alignment that we can go for that would still


give us the ability to prompt in non-trivial ways?


So actually, this is kind of a follow up to an older paper that we published, it was just


accepted in ICLR actually, which was this HTLM paper.


And the core idea of this paper is that we argued that document structure is really,


really important.


So what we did there is we took BART, BART large, and then we pretty much trained it


on just web data, like minimized HTML, right?


So minimal HTML is we pretty much do multiple passes over the DOM and take out anything


that we don't think is semantically important.


So in that paper, we showed really strong results.


So for example, for zero-shot summarization, in a structured language like HTML, this is


pretty much just generating the title, right?


Or generating the meta tag where the attribute is the headline, right?


In some sense, we could exactly replicate how CNN and Daily Mail was collected, which


was they looked for headlines, right?


So in the prompt, you can actually describe the way that the data was collected.


So we saw that there was some rich structure available to be used in HTML.


So after Dolly came out, we thought, okay, there are some fundamental restrictions with


Dolly.


So the first one being the causal approach.


So they train a decoder only left to right model.


So in some sense, you can't do things like generate the text given the image, right?


Just because of the positioning of the image.


It's on the right side of the input, right?


You can't really do image infilling either, which means conditioning on both the prefix


and postfix of the image.


It's not possible.


So you'd have to like train specifically one particular type of infilling, right?


You could rearrange stuff such that you could infill one part, but you can't like dynamically


infill something.


Exactly.


Yeah.


So those were kind of the first weaknesses that we saw there.


The approach was very clever, though, right?


So pretty much taking continuous data, discretizing it, and just doing sequence modeling, it seems


to work very, very well.


So the idea went that we kind of combined the two from the HTLM paper, which was that,


you know, document structure through HTML is really important, but let's also encode


images there and see if we can recover something like Dolly.


So here you're kind of looking at the data that we collected.


So the data set size is actually quite good.


I mean, we're around like the 200 billion tokens, which is a relatively good size if


you're training large models.


But one kind of downside that we have here is because we don't have the strict alignment,


we can't artificially increase the amount of images that we have available in the documents.


Actually, look, I think we have 25 million unique images.


I don't know about Dolly.


Dolly was trained on 400 million.


I don't know how many of them are unique, but regardless, they still have an order of


magnitude more images than we do, right?


But then we have the other benefits, right, which is we're also training on a ton of text.


So we can do a lot of text only tasks.


And I think the rest of the paper will show that we can do not only text only tasks, but


we're actually competitive to T5, which is actually really hard to do.


And I can explain why we think this is the case in a little bit.


So the very first thing was, okay, so now we kind of have this data, but HTML is also


very localized, right?


Like the title always comes first.


It's in the head, right?


Or like the meta tags always pop up first, right?


So if you want to generate meta tags or generate title, right, condition on the rest of the


text, it's kind of non-trivial how you would do this in decoder only setting.


And so we kind of started thinking, you know, there are multiple ways around this, right?


So the first thing is using encoder decoder architecture, right?


And then with some masking, you can kind of recover this type of bi-directionality.


This is true, but there are pros and cons to this.


So encoder decoder only architectures, they're really good for fine tuning, but they're not


so good for prompting is at least what we noticed.


And also training them is a little bit more non-trivial.


So decoder only models are quite nice because you get per token generation.


So you pretty much generate every token for the source.


Whereas for encoder decoder, most of the time you're generating, I think like 15% is what


BERT and like BART or like Roberta do.


It's all around that 15%.


So most of the times you have to go through the data multiple times.


For some reason they don't prompt super well.


And the kind of the other big thing is if you want to do score-based prompting, it's


kind of hard to do with encoder decoder only architecture, right?


Like if you want to ask, what's the log probability of this sequence?


With the mask language model, it's kind of tough to do, right?


So we knew that we wanted to go kind of this decoder only route.


So we introduced this new objective that we called causal masking.


And so the idea behind causal masking, if you want to scroll down, I think there's a


figure there.


This one.


Yeah.


So the idea there is relatively straightforward, right?


So pretty much think of mask language modeling where you place in the mask, but take the


mask and put what the mask represents simply at the very end of the sequence.


So if you do this, you kind of get, it's very, very simple, right?


But you get a lot of the benefits, which is you still get per token generation.


You optionally allow for bi-directionality, which is actually a really, really big thing


to have, right?


And the other thing that we noticed is that depending on the setting, prompting versus


fine tuning, the size of the mask is really important.


So for fine tuning, localized information is really important.


You want to have a lot of small masks.


For prompting, we saw kind of the opposite, which is you want to have very, very few masks,


but they can be very long.


So the strategy that we use here is for every document, we sample from a Poisson distribution


centered around one.


So the majority of times, right, and we clip it to one.


So if you get zero, it becomes one, right?


So majority of times, you're only going to get a single mask, right?


Over 50% of the time, you're only going to get a single mask.


And then you pick, you uniformly sample a subset of the document of any size.


And then you kind of place that in the end.


So you get these very, very long kind of infilling naturally.


And so this objective turned out to be quite strong.


So it's competitive to language modeling in the sense that when you get per token generation,


our perplexities were not that much higher than just a language modeling objective.


You get optional bidirectionality whenever you want it, right?


You can score probabilities of sequences super, super easily.


So we're kind of going all in on this objective.


And so we have some follow up work looking at causal masked scaling loss for text.


So this is some ongoing work that we have now.


So we're pushing heavily on this.


So the general argument that we're trying to build is that, you know, if you're doing


language modeling, decoding only language modeling, you should be doing causal masked


language modeling.


So that's kind of my...


Yeah, I mean, I was, I was, it is intuitively a good trade off.


So I think here you make the case, if I interpret this correctly, that this word nationalist


right here is really important to fill in this mask.


And if it were just sort of left to right, it would be very difficult to fill this in


yet, since you move it to the end, right.


And the model has to extra learn kind of to keep these tokens in context to sort of realize,


you know, what's there.


So it has to waste kind of some extra memory to remember the context of each of the mask


tokens and so on.


But yeah, I think it is very intuitive.


It is also a good trade off between, I want to say, left to right has, at least for, you


know, most there are right to left languages, but for left to right languages, left to right


objective, actually makes sense, right?


That is how we generate language, you know, when we write it down.


So there is something to left to right that I was never happy.


There are other other approaches like XLNet or so they were saying, well, we just train


on all possible paths, right of decoding, like all possible sequence of masking out


tokens and it was, it was never really satisfying because I always thought, but there is something


to left to right.


However, sometimes, as you say, is really important to know what's after.


And I think this is like a really good trade off.


Yeah, like specifically in this example, right, like in the zero shell prompting case, right,


like, let's say we want to tag nationalist with some entity link, right?


If it appears beforehand in the sequence, there's no way to prompt the language model


to generate like an entity link before the entity appears, right?


Yeah.


So that was kind of another reason that we had, because like I said, like HTML data is


very localized, right?


Like in Wikipedia, this, this a tag, which represents the entity link always appears


before the entity.


Either we have the option of, you know, training two models, right?


One left to right, one right to left or you can kind of do this kind of clever rotation


of the document.


You said, yeah, the AxelNet approach is definitely interesting, which is, you know, having different


permutations of the source document.


But like you said, I think there's a lot of inductive biased for a left to right, which


is why I think left to right models are kind of de facto now.


Is there just, just for my understanding, is there a reason behind these arrows?


Like why do the arrows like are like double arrows and there's a line and there's like


a double arrow again, like it does have a specific meaning and here the arrows are only


here.


So arrows pretty much was the tokens that you actually generate.


So the language model, you're generating every token and the mass.


So you go like this.


Okay.


I see.


Cause I was, I was like, okay, is there some meaning?


But yes, there is.


And this shows that in the mass language model objective, you only actually generate very


small number of tokens and you wouldn't even get like a loss for the other tokens.


Yeah, exactly.


You said before that you had a certain number of tokens, right?


And you said, well, that's actually good or bad for, you know, that's actually in a good


order for language modeling.


Yet a special thing about your model is that images are also tokens.


You push images through a VQGAN encoder, right?


Which is pre-trained and these images, these just become tokens in whatever sequence.


Do you, and this results obviously in larger data because some of it is images.


So you say you have a terabyte of data in this dataset, which is obviously way larger


than for example, a text only dataset.


Do you find there is a difference?


Like do you find the number of tokens is really what matters in the size of the data?


Or is there a qualitative difference between image data and text data, even though both


are tokens?


Yeah, so there's a couple of ways to approach this.


So the very first thing is that modeling, and I think we mentioned this quickly in the


paper, but modeling image tokens versus text tokens, it's quite different actually.


So for like text usually follows like textual tokens follow like a zipfian distribution,


right?


Whereas I think in Appendix, we have a figure, it's pretty much uniform for images.


So there's different, like in terms of the distributions that you have to predict, they're


actually quite different.


So we saw a little bit of challenges and we saw some kind of weird behavior during training.


We didn't mention this in the paper, but the one weird behavior that we saw was that there


were regimes during the training, like parts of the training that only optimize for text.


So on our image evaluations, like it pretty much would be flat.


And then there were times that it was quite the opposite where images would be being optimized


for but text kind of stayed flat.


So we don't really have explanations for why this is happening.


I think there needs to be future like scaling laws looking at multimodal sequence modeling.


And when I say multimodal, I'm not just talking about like images and like natural language


text.


I meant like you can even include code as a different modality, right?


So the scaling laws there I think are a little bit different than what we're used to with


The reason for using tokens is purely because of a compute thing, right?


So we're given some amount of GPUs, right?


For some amount of times.


So what we do is we take the number of tokens that we have, we take the amount of compute


that we have and try to find the largest size model that we can train.


It's kind of optimization problem to find the largest architecture.


So that's kind of why we used a number of tokens as the as the guiding principle.


I mean, it seems to also align with what others yet, for example, this Rudolph paper, so that


it seems to be a common approach to lift images into like the space of textual tokens, which


is, I guess, a bit surprising because a couple of years ago, no one would have gone that


route.


Even if you even if you were to inject images into a sequence model, you'd probably inject


like a single vector, right?


So I find that to be, well, a bit surprising, but also, yeah, it seems appropriate that


an image could be expressed in something like a sequence of tokens.


It's just a bit I'm not too big of a fan of how this is currently done because the tokens


they also they already they seem to be a bit localized in the image and so on.


Like I don't I think there's like a better and I think there's a better way if you're


a human you're not that's not really what you do with an image you see more like the


different layers maybe or what's there.


In any case, I was surprised by the scaling plots like these are these are brutal like


this is like we scale it up and it just like the the loss goes down for the largest model


it seems they were nowhere near done.


Right?


Like this just so you said you had some different experiences during training.


Yet also I think in the paper somewhere you hinted at well, we didn't really see any


pathologies.


So what's like what's what was the process like you had the data you train the thing


did it immediately work?


It took a little bit of hand holding to work, especially the 13 billion parameter model


took a little bit of hand holding to work.


So a lot of times the pathologies we see as are things like gradient underflow or overflow.


Gradient explosions happen although they're more they usually happen in much bigger models


like the 100 billion scale.


But the surprising thing was that we almost used exactly the same hyper parameters as


this paper that came out from Vesto and those group.


So the surprising thing is kind of just worked out of the box apart from having to tune.


I think we tune like learning rate.


We had to tune weight decay and batch size apart from tuning those things.


It just worked almost straight out of the box.


And what you said is actually correct, which is if you look at the large model, it's actually


not done training.


So the good news is once CM3 is released, we're going to release the checkpoint that


we use for this model.


I think the model that we have now is continuing training.


So we'll release that one too.


So people will be able to play around with both.


Excellent.


But one thing I'd like to point out is that the multimodal scaling laws are a little bit


different than text scaling laws.


One thing seems to be that scale plays a slightly larger role in multimodal than it does in


text.


So I think the quantitative thing that we saw is that if you looked at the data efficiency


jumps between like, I'm forgetting the exact numbers, but like let's make them up, like


the 1.3 billion model and the 13 billion model from Vesto's paper.


And the data efficiency there, let's say it was like the larger model was five times more


efficient in terms of data.


So in order to reach the same perplexity, you would need five times less data.


Using the same exact models, we saw that in the multimodal case, it was 10x.


So it was almost a two times difference for some reason.


And that's why I think it's really important to kind of chase these multimodal scaling


laws and fundamentally understand what's going on here.


There's a lot of unknowns here.


Can you say we had to do a little bit of hand holding?


What does that even mean in these large models?


Like can you afford to restart training?


Or is it more like, you know, you have checkpoint, checkpoint, and then something goes wrong


and you go back to the last checkpoint and you do something there?


Like what does the process of training these very large models look like?


It's just really, really tedious.


So one of the main things is, you know, whenever you have a ton of nodes that you're running,


there's infrastructure issues that pop up, right?


So like if one GPU goes down, right, then all of training is paused, right?


So infrastructure issues are kind of a big thing.


And we have some automated systems in place to take care of that.


Other things are like, for example, like we didn't set a high enough warmup period in


the beginning.


So we saw that we actually had to pause training, increase the warmup, load up the last checkpoint


and go there.


And so we also kind of tune learning rate a little bit as training goes on.


Although with the large models, I think it might have been just a handful of times.


So failures, do you always have like multiple models running ahead and then you choose the


one that looks best?


Or is it really like you change and you train one model and you see how it develops?


Yeah, because of the computer is one model.


So it really comes down to intuition.


So both Mike Lewis and Naman Goyal who are on the paper have trained these really, really


big models before.


So they had a ton of great intuition about how to get things to work in terms of these


very large models.


Yeah.


Cool.


I mean, yeah, I'm excited.


And it is very cool that you actually are going to release these things.


I think people will love to play around with them.


For in order to do now the tasks, you tackled some tasks.


How did you decide which there are some natural tasks, let's say there are some that are more,


you know, you have to come up with something.


Did you have some targets of tasks that you want to tackle?


Or was it more like the model came first and then you sat down and saw what can I actually


do with it and whatnot?


And what worked?


And were there also tasks that you tried that maybe didn't work at all?


Yeah, that's a great question.


So I think at the beginning of the project, the push was really to have a single model


that can do any image task in the zero shot case.


And so kind of the story that we built around it is, can we describe all the tasks that


we're interested in through some prompt, through some HTML prompt, even before we trained the


models we talked about this.


So we came up with a ton, right?


And some prompts were very complicated, like style transfer for one, right?


So you can have an image that has a picture of the mountains in the summer.


And then you have another image tag that says the same picture, but in the winter, and then


you ask the model to predict the image tokens, right?


So you can get this kind of zero shot style transfer.


So you have some kind of complex prompts.


So some of them didn't work.


Some of them only work that scale.


And we can kind of go through this.


Specifically like one thing is that like the captioning only worked at scale.


So the 13 billion model was the only model that could caption well.


And the captioning you go mainly with the alt text of the image.


Alter the title, you know, one.


Yeah.


But like the figure that you're on now, I think is kind of interesting.


So we can kind of get unconditional image generation by just asking the model to generate


a sequence of tokens after the image tag.


So we saw one interesting behavior is that the model for some reason almost always wanted


to first generate the alt text before generating the image.


For it was actually easier to condition on the text before generating an image than doing


this type of freeform generation.


When you say wanted to, that's just what it did.


Like when you when you sampled, did you like, I mean, this, when you say wanted to, it could


also be that in the internet humans most of the time, right?


Alt first, and then the source.


Yeah, so we actually looked into this.


So a lot of text does have alt.


But it's around like, I want to say like 70 to 80% mark, if I recall correctly.


So it wouldn't explain why the model almost always wants to generate alt text.


Now the theory that we kind of have is that without alt text, you have much higher perplexities


for images.


So the model, you know, because we're doing like sampling, right?


So it's going to pick out high probability, low perplexity tokens, which most of the case


means picking out the alt.


Just because it appears so often.


So that could be it.


But overall, I think if you look at these images, they're rather like they're semi coherent,


especially the ones conditioned on the text.


And the same thing I think you see with you can kind of force the model not to generate


the alt text by giving a prompt to generate the image tokens immediately.


And do you think so that the VQGAN tokens naturally they are predicted as one, right?


There is there's some encoder they're not as far as I understand, they're not in the


image encoder that makes the tokens, they're not predicted autoregressively.


So there is no inherent sequence nature to these tokens, could that be like some sort


of a reason why there's also a difference because text naturally is sequential, whereas


these tokens, the only thing they have is they're kind of localized, but there's no


inherent sequential nature.


Yeah, that's true.


There isn't for VQV again, there isn't something explicit.


But I think the way that the layers are constructed, you do still get some implicit dependencies


across the tokens.


And so I think this is what the transformers kind of pulling apart here.


And to be honest, I think there's still a lot of work to be done on the discretizing


images front.


So one thing about VQV again is that it blurs a lot of fine detail, so like human faces.


In our case, this is kind of good because it's privacy preserving, you're not going


to generate like a person's face unless it's a really, really popular and like close up


face.


So in our case, it kind of worked out.


But in the future, I think we need to get much, much higher fidelity image tokens if


we think that the way of doing things is to treat everything as a token.


Of course, I think there are a ton of new approaches that are not token based.


I think Glide was fantastic from OpenAI.


The diffusion models are doing great generative work.


But if you want to maintain the same benefits of generative models, so being able to generate


trivially, being able to compute log probabilities, I think tokens are probably the easiest way


to go.


And one thing is you can naturally increase the resolution of tokens images just by increasing


how many tokens you use per image.


So in some sense, if you have enough compute, you can scale up to arbitrary resolutions,


right?


Yeah, I mean, yeah, down to probably, probably you could at some point get more tokens than


pixels.


I wouldn't know what that would mean.


But I guess the resolution isn't even limited by the resolution of the image itself.


So there's this interesting thing you can do, as you said, infilling by letting the


model generate sort of middle tokens.


Now, you I mean, you could probably do arbitrary infilling, but you have to have like multiple


mask tokens.


So I guess the the natural thing to do is just to infill since the tokens kind of go


left to right, top to bottom is to infill one of these stripes, which you've demonstrated


right here.


Yeah, sorry, did you did you try infilling like arbitrary things?


Or was this sort of the natural thing to do?


Yeah, so actually, because of our objective, because we sample the number of masks, right?


Yeah.


You can actually mask out like 567 masks.


And it's still work.


I don't think there was any specific reason that we stuck to masking out a single thing.


I'm sure it would work with multiple as well.


I mean, if you like if you were to if you were to infill, let's say, you know, if I


infill a square like this, and it covers sort of multiple token lines, this would already


result in like if it covers three token lines, it would already result in like three mask


tokens, right?


So yeah, I mean, that there is there is some with just with the sequential nature.


But I think that can be can be worked around.


So what here we see, so left is source image, then you mask out something in the middle.


Then you also give the ground truth, which is here on the right.


And then there's one model that does infilling unconditional.


So just looking at the image, and then there is one model that does it conditionally.


And the conditional is conditioned with this thing right here as the the alt text.


So the understand, okay, so understand it correctly.


I was Yeah, I mean, I was surprised, for example, by this one right here, this, the park bench,


because obviously, if you see the, the model that does infilling conditionally, it can


do it quite well.


However, the unconditional one, it kind of warps the bench or something like this, like,


it's, it's a bit I'm not I'm not sure the unconditionality has something much to do


with it, because there is no this doesn't look like natural, you know, you know what


I mean a little bit?


Like yes, this shouldn't be like, just because it's not conditioned on it.


If it's not conditioned on text, I would expect it to be maybe a red bench, right?


Or, or something, you know, something that is conceivable in nature, but is not according


to the text, like there is an ambiguity of what's behind the mask.


However, here, it really seems to degrade in performance when you don't give it the


text.


Yeah.


So, so one theory that we kind of had here is that the the model needs to understand


the continued continuation of the horizontal lines, right?


That requires some semantic understanding that this is, for example, a bench, right?


And actually, if you look at the the massed out input, the horizontal lines are not completely


horizontal.


So the bottom of the bench is at a different angle than the top of the bench.


So I think the model has a tough time understanding the high level semantic content of the image,


which is fixed by feeding in text.


Now, I think, of course, if you have, I think if you have a larger model that's trained


for longer with a higher resolution, this probably should not be an issue.


But VQVAE again, it blurs out a lot of things, number one.


Number two, it's just if you change the tokens even a little bit, the blurring aspect happens


very, very quickly with VQVAE again, compared to, for example, the VQVAE from DALI, which


requires more tokens, so 1024 tokens versus the 256 we use here.


But it's more direct in some sense.


So, yeah, I think the main thing here is just that you need to get some like high level


semantic information about what's going on in the image.


And it's hard to do if you're only looking at like the VQVAE GAM tokens.


Yeah.


Okay.


I mean, that makes sense.


You go on and you have some examples of conditional image generation.


So on the left side here is a prompt and then you sample images from that with the same


technique, right?


You give the all text and then you sample the image.


So the avocado chair is like forever going to be to stick in history, right?


I think that's just that's just a given.


Was there one, was there something that surprised you with conditional image generation?


So the models are quite good at actually generating something that's somewhat coherent.


So for example, like the red car, you can see it generates two red cars, that one looks


like a truck or a tractor.


Sometimes the model tries to cheat and generate something that's easy, for example, in the


case that it doesn't generate a car at all, it just generates mountains, right?


Because landscapes are easier to generate.


The other thing that we saw kind of tough compared to Dali is, you know, the data that


we used only came from Wikipedia or Common Crawl News.


So none of it was fictional in some sense, right?


We don't have any like art.


So like our images always try to be as non-fictional as possible, which is it adds weird if you


try to give it like really fantasy based prompts.


So that's kind of one downside.


And actually, this is one criticism I have of the evaluation that we did for the FID


matrix, which is a way to measure, you know, the quality of images, which is we actually


took the table from Glide for the FID numbers on the conditional generation.


One thing was is that MS Coco is almost all nonfiction, like non-fantasy images.


So this is really like it's under-representing Dali.


So I think if you casted a wider net here and had something that included a wider array,


a bigger distribution of images, I think Dali's results here would be much, much stronger.


Which is why I think we're kind of comparable, our largest model is comparable to Dali on


MS Coco.


But in terms of image generation, it's not as good on the like the fantasy front at all.


You did discuss a little bit, you also said you sub-sampled web data and you cited some


concerns as well.


But there is also a quality issue with sort of the wider you cast the net, the sort of


more the quality goes down, I guess the alt tags quality go down, whether or not the images


even have alt tags, whether or not they're ads or something like this.


What were like, why did you limit to this subset of the data and not bigger or smaller?


I think at the beginning, we had some ethical concerns of like, like I said, we have very


weak alignment, so you can prompt with anything, right?


We had some ethical concerns about images that you can generate if you were just trained


on all of Common Crawl.


So we tried to think about what are like large scale data sets that we can get that are somewhat


filtered.


Wikipedia is definitely one of them.


But even then, actually, Wikipedia itself has a gender bias.


And I think this is a new, I think other papers have showed this before.


And Common Crawl News, which probably is not going to have the terrible content that we


don't want to pick up.


So we kind of picked those two and it was okay at the scale that we wanted to.


So we stuck with those two.


But yeah, I think it's hard.


I don't know what the solution is.


Like the LayOn 400 million data set that was released.


I don't know if you've heard of it.


But this data set, I think there was a critique paper written in like a month about it, right?


That showed that it was like a highly, highly problematic data set.


So in terms of the ethical approach, I'm not really sure what the right answer is for collecting


at scale.


There are tricks you can do, right?


So like if you look at the CC100 data set that Facebook collected, they use this trick


that they train a language model on Wikipedia and then use it to score Common Crawl and


then take only like medium perplexity.


So you could probably do something like this here.


I questioned the efficacy just because very large models, they only need to see a data


point a couple of times in order to pick it up.


So I think there's like some very fundamental engineering work that's being done for scaling


up these data sets to like trillions of tokens essentially.


Yeah, I mean, I guess it casts much wider questions such as, you know, I as a human,


I'm perfectly capable of going to 4chan and seeing kind of the worst of humanity.


And it doesn't instantly make me like, you know, I don't know, a terrible, terrible,


like it doesn't make me want to repeat everything or something like this.


And there's various considerations like shouldn't we be able to build model that also ingests


stuff but kind of may also a bit distinguish between things like if the models are able


to distinguish, it might help them to ingest more of this critical data.


On the other hand, I can absolutely understand that, especially if you're the maker of a


model, you don't want your model to output, you know, that I think that's why for example,


OpenAI keeps such a tight grip on on GPT-3.


If you want to build anything with it, right, you have to go through approval processes


and whatnot.


And it's, it's, yeah, it's, I think it's tricky topic.


I also don't know what exactly to do.


I'm happy that there are models that are filtered, like say on filtered data.


I'm happy that there also exist models that aren't.


Yeah, I think the maybe the sort of the, let's say diversity makes is, is probably the best.


So you can always choose which one you want to you want to use.


I don't know.


Sorry, this is just a rant by now.


You do have some, sorry, go ahead.


I was gonna say, with respect to what you're saying, there's the solution doesn't necessarily


have to lie on the language model side.


So one thing is you can think of language modeling is just pure density estimation over


tokens, right?


So if you're doing that, like, of course, you're going to model like 4chan, for example,


right, but it's up to your generative sampling strategy to remove that part of the density


and only sample from, you know, parts of the density estimation that you know are safe,


for example.


And so we're actually seeing, I think, a lot of movement from, you know, having a singular


model that does generative work into having like multiple models.


So a great example is like Dolly, right?


So they do density estimation over, you know, text and image tokens, right?


But the way they generate images is they sample like 128 candidates and or whatever number


of candidates, and then they use CLIP, a secondary model to kind of select in some sense, the


mode of the slice of the density, right?


And so something probably similarly can be done here.


Like a great example is like take codex, for example, right?


I think in the codex paper, what they do is they generate a ton of samples, and then they


re-rank the samples in terms of perplexity, so average probability, and then they take


the mode, so essentially the exact mode of that density estimation, right?


So one thing to argue is that, you know, you could train language models that do pure density


estimation over all the text that we have, and then have smarter generation algorithms


that are able to select subsets of that density that are safe.


So like you said, in terms of research, I think there's pros and cons to having unfiltered


and filtered models.


But that's kind of the way I've been thinking about it recently.


Yeah, and it's probably a good approach because the sort of the handle we have on, let's say,


discriminative models like CLIP is a lot larger than the handles we have really on generative


models.


Like, yeah, the only handle really we have there is kind of data.


You also do some experiments on text, pure, I don't want to say pure text data because


it's more than that, right?


It's entity disambiguation, entity linking and so on.


Now, is that purely a result of the fact like of you use Wikipedia as a data source and


Wikipedia is essentially, it's not really only text, it's kind of a huge entity link


and database.


Is that kind of, is it fair to say that it works really well because you use Wikipedia


as data or is there something more to it?


Yeah, no, that's exactly it.


So actually, there's this work that we said in this paper a couple of times, the genre


paper.


So in the genre paper, I think the paper is called auto aggressive entity linking or entity


disambiguation.


So the idea there was exactly that, which is, if you take all of Wikipedia and then


you train a language model that tries to predict entity link post entity, you get a model that


does really, really good entity linking, right?


So in some sense, the genre objective was a subset of our much more general objective,


right?


And it's not too surprising we beat out genre just because our models are bigger in our


fine tune case.


But the really, really cool thing I think was that we can do this, the zero shot, which


is exactly what I showed in the first figure.


If you mask out the entity, if you know that you want this entity, you want to disambiguate


this entity, you can place a mask there with this a tag, right?


And then our model will fill in what it thinks the disambiguation is.


So that's kind of cool.


I couldn't find any, like zero shot baselines like this.


So I think this is kind of the first paper to do this type of zero shot entity linking


And so I mean, you also have other tasks like summarization.


We also didn't look at the like the alt text generation and so on.


Is there one result that we didn't talk about that you want to highlight in particular,


like what maybe one surprised you the most or so?


Yeah, so the captioning one was interesting.


I think we can look at that.


So the captioning is, this is pretty much the dual of Dolly, right?


So what we're doing is saying, okay, you know, now that you have an image, generate the alt


text for me given the image, right?


So in some sense, we can exactly describe the captioning task in HTML, which is again,


kind of solidifies the argument that you want some level of document structure for prompting.


So the results are quite good actually, at least from a semantic level.


So one problem is that we don't actually generate in the style of, I think, MS Coco here.


So we didn't report like blue four numbers or like the standard numbers.


But if you look at the semantic similarity using BERT score, the CM3 captioning with


clip as a re-ranker is actually a very, very strong baseline.


And so you can kind of see the style here is weird.


It tries to explicitly state what type of airplane it is.


But that's kind of an interesting behavior.


So I think definitely at scale, you know, you could get a single model that I think


could be competitive with MS Coco with caption only models.


If you do things like increase the resolution of the tokenized images, I think scale is


really important here.


So if you just scale up so that you have a similar amount of samples that are trained


using MS Coco.


You said this a couple of times now, this sort of, you know, with scale, we could beat


this or that or and I guess you see this work a little bit as a maybe a signpost.


You know, to like later work that actually achieves this scale.


Do you think the scale you're talking about the scale at which you know that this is competitive


with on MS Coco, where the image generation is competitive with Dali?


Do you think that scale is currently achievable?


Or is it so large that it's kind of well, you know, we need entirely new hardware?


Yeah, I think it is achievable.


So let me tell you about the result that we just got a couple of days back.


That's not in the paper here.


So one one reason that we also changed chase this kind of multimodal setup is because we're


interested or at least I'm very personally interested in the grounding aspect of language.


So we kind of defined grounding as can you improve document level perplexity on text


by extra conditioning on images?


So that's one kind of way to measure grounding.


The other way to measure grounding is we call it symmetrical grounding.


So what you do is given a pretty much given a piece of text, generate an image from that


piece of text and then condition on that image, generate back that piece of text.


And I look at the perplexity differences between the two texts and that will give you the informational


content of that image that is generated.


So you can measure grounding that way.


The unfortunate thing is that even the 13 billion parameter model that we have here, that doesn't


ground.


But if you look at the scaling laws from, you know, or I think our 100 million parameter


model to our 13 billion parameter model, around the 60 billion mark is where we'll see grounding


in this setup.


Okay.


In your plot.


So our expectation is that if you scale this up to 60 billion, that you should be able


to achieve, I think, language image grounding, which is kind of a cool result that I think


a lot of people have been chasing here.


And that's insane that you can make these predictions, right?


This is like, this is something I think in machine learning is something new.


Because right now, no one could tell the most people could tell was like, GPT-3 is going


to be like, somewhat better than GPT-2.


But now you're you're able and you know, I am confident that this is a, you know, maybe


it might be whatever 50 or 80 billion parameters, but you can actually make these predictions,


which is, which is, you know, it's, it's cool.


Like, I'm amazed by this.


Yeah, I definitely don't think we're going to be like order of magnitude off, right?


Oh, so I think with the 100 billion parameter, 100 billion or 175 billion, like GPT-3 size,


we can get very, very non trivial behavior to the point of being competitive across all


tasks.


And I think the future in general is having a single multimodal model that can prompt


in an instructable way, kind of like instruct GPT, but with all modalities.


So I think that's kind of the North Star that everyone is chasing right now.


But I think we have good, I think we have a solid base for this work.


But yeah, I think the captioning surprised me.


And one thing that I want to call out here is that it only worked at a 13 billion scale.


I might've mentioned this earlier.


So there are fundamental stepwise changes in behavior from scaling up the model.


It's not something smooth, right?


So something that a 13 billion model can do is something that, you know, like a 2.7 billion


model will not be able to do at all.


So you won't, it's just going to generate random stuff.


So it's interesting to see what the next, you know, stepwise changes in behavior will


be if you scale this up.


With respect to the HTML, right, that you use, which is, I thought it was it was pretty


cool because it is data that is, you know, so available.


And your argument is a little bit that if you clean the HTML too much, right, these


other these other data sets, they just pull out the text content, maybe the image, they


try to align it and so on.


You know, if you clean that up, there's so much structure missing, right?


You're missing on all of this valuable information.


Yet, you also do cleaning, right?


You do quite a lot of HTML cleaning, you say somewhere up here in the data section, we


strip this, we strip that any, any sort of non, non whatever elements, we strip out all


headers, all footers, copyrights, forms, dialog boxes, we merge consecutive div elements


and so on.


Couldn't the same argument be made against you saying, well, you're losing so much of


the structure, there's so much information there, like, why are you doing this?


Do you think there is a valid direction to go in actually taking in even more context


of these HTML documents?


Yeah, so there are different constraints here, right?


So one thing that I mentioned is that we can only model X amount of tokens, right?


300 billion tokens, for example, right?


So if the majority of those tokens, right, like, I think the average document is like


95% of the document we removed.


So yeah, in some still, right?


Even though you're the ones that remove way less than the other ones.


Yeah.


So, so in some sense, do we want to model every single token?


So in the case that you have infinite compute, sure, right?


But here, there's kind of a min max problem that you have to solve, right?


Which is you want to kind of, you want to maximize the amount of semantic information


that is available while minimizing the amount of tokens that you have, right?


And this is kind of complex to do.


So I think we found a good enough balance of the two.


Like in most cases, like you don't want to repeat the same copyright, like 400 million


times, right?


I mean, there's probably a lot of information in the fact that jQuery is imported in this


website, right?


Right.


So things like that.


But we also do things that might break document structure, like the merging of elements, right?


There's probably something there as to why the person has multiple div elements, right?


Regardless, we remove it.


The other thing that we remove is attributes.


So we remove all the attributes except those that are structured.


So like Open Graph Schema, I think Twitter has a like a structured graph as well.


And the reason there was that the attributes were just, first of all, they were way too


long most of the time and they were not informationally rich enough.


So you kind of have to balance compute here with how much structural information you want


to maintain.


Yeah, I see.


And so there's no fundamental reason to use HTML, right?


It's just something that's there, right?


There's, I mean, for example, you can use Markdown as well, right?


And you can kind of recover a lot of the same things, right?


Like generating the title you can do in Markdown, right?


High-level links you can do in Markdown, right?


So maybe the future direction is explicitly codifying this min-max problem, right?


And coming up with the document structure that the document structure is described in


the minimal set of tokens.


So maybe that's a pure engineering project as well.


But yeah.


When you think of HTML and the DOM, it is a tree, right?


Which is different from a linear sequence.


Do you think there is, do you think there's value in treating the tree as a tree?


Do you think it's mainly a limitation of the models we have?


They go, let's say, like, token by token or left to right or something like this?


Do you think, you know, maybe it's still good to treat it as a sequence because there's


text in there and text is left to right?


Like what keeps us from building tree-based models, which would be much more appropriate


for something like this?


Yeah.


So one thing about transformers is it seems that they can learn the inductive bias of


the data fairly well, and it's not necessarily encoded.


So my argument to this is that usually for these large scale runs, the best thing is


just to keep it as simple as possible.


Mostly just because they're risky, right?


You get one chance.


But the other reason is that transformers are actually highly capable of picking up


this type of structure.


So this isn't in the paper, but we looked at the attention scores and then you can see


very clearly that the model knows what are like boundaries between HTML elements, for


example.


But again, there's also a ton of work to be done as well.


So like some exciting work is, I think you also interviewed like Ofer for the alibi work,


right?


Like that work is really clever, right?


Because it introduces an explicit inductive bias that the further away a token is, probably


less likely that you are to look at it and it gets rid of the need for positional representations.


So you can imagine like an extension of alibi here that would directly encode a tree-like


structure, right?


So there's a ton of work to be done here.


And the other thing is we didn't do too much for the images, right?


Like in terms of attending, like the positional representations for images are different than


of text, right?


So future work should consider like specifically embedding images in such a way that you maintain


locality of positions, right?


So this is all stuff that needs to be done in the future as well.


But that being said, I think if you have enough compute, these models can learn anything,


mostly becomes an efficiency angle.


So about this paper, so what I have a bit of a trouble with is, you know, too many things


in one paper, which in this case is, it's this idea of using HTML and so on, although


there was a previous paper of that, but then there's also the new loss and so on.


Have you like tested the new loss on pure text generation?


Something like this?


Would this be like, can you parse out sort of what the different things contribute to


the success of these models?


Yeah, and that's a great criticism of the paper actually.


So fundamentally, I think if we wanted to do this like the proper science way, this


would be like four or five papers, just teasing things apart.


But at the same time, when you're training these large language models, ablation studies


are pretty much impossible, right?


No one has much compute to do these ablation studies.


But the answer is yes.


So we're looking at causal mass scaling loss for text only.


This is a project that we're working on.


We've trained a code model using the causal mass objective that's, you know, outperforming


I think both Google and Codex of similar sizes while being able to have a bidirectional option.


There are a couple of teams within Facebook that are trying out this objective with some


success.


So there will be future work about this.


Excellent.


And apart from what you just mentioned and scale, what's sort of next in this direction?


Are you like, what are you excited about?


Maybe it's not even you working on it, but what kind of is exciting stuff that's happening?


So one thing is figuring out a way to have higher fidelity.


So the question to ask here is how do you represent continuous data in a discrete domain?


And I don't think we're there yet.


So that's some fundamental work that needs to move forward.


The other thing that I'm kind of interested in looking is can we start joining more modalities,


right?


So like Hubert that also came from Facebook had speech tokens, right?


Very simple.


I think they use k-means.


I might be wrong though.


Just to find discrete tokens for speech.


So imagine that you have a single model that has video images, you know, text, speech,


everything kind of put into one, right?


Like what level of grounding and what level of zero shot prompting can you get here?


And I think a lot of people are kind of chasing this at the bigger companies.


I'm kind of excited about that.


On the analysis front, I think there's still a lot of unknowns about transformers.


Like fundamentally we're still using the four-year-old implementation, right?


The only difference is just pre-layer norm, right?


From the original transformer.


So I think better fundamentally understanding transformers.


And I have some qualms with scaling laws.


Like I don't think perplexity is necessarily the measure that we should be using.


So internally we've been discussing like what does like memory-based scaling laws look like?


So if you use memory as the fundamental unit of transformers, what do those scaling laws


look like?


So there's some more fundamental work to be done there.


And the other thing is bridging, fine-tuning and prompting performance.


So far it's kind of orthogonal, which is, you know, if you want to get a better fine-tuning


model, you have to do something that will hurt prompting and vice versa.


So figuring out like, is it just because we don't have like bi-directional like masks?


Is that why?


Is it because we only mask for like causal models and upper triangular matrix?


Is there something more fundamental there?


I think kind of peeling that apart and figuring out what's going on there is kind of important


too.


But I think we're very early on.


I think this year is going to be the year of multimodal.


I think all that kind of kicks stuff off.


So I'm kind of excited to see what other groups are working on.


It seems like it.


Yeah.


Is there anything else about the paper or the research direction you want to shout out?


You want people to know that we haven't mentioned so far?


I mean, we'll be releasing all this code really, really soon.


We're just waiting on some internal approvals so people will get to play around with it.


I think we'll release a 3 billion model, but the 13 billion model is the one that really


shines.


So if people get that running, I think it's really cool.


I spent hours just playing around with it.


Nice.


What does it take to just to forward propagate?


What's like the minimal configuration?


So with the recent deep speed stuff that was released for inference, I'm not really sure


because I think they said that you can use one GPU for like a 6.7 billion model.


So if you do model parallelism, I think you need two GPUs.


So like without that, just give us a ballpark, what would it be like forward propping through


this model?


Yeah.


So one thing is you could do it on a CPU if you have a strong enough CPU.


But for inference, I think what I used was 4v100s model parallel.


Less than a node.


Cool.


Excellent.


Well, Armin, thank you so much for being here.


This was really cool.


Really valued the like also the kind of behind the scenes insights we got here.


And I hope to see you again very soon with even like CM4.


Yeah, thank you for having me.


PonderNet: Learning to Ponder (Machine Learning Research Paper Explained)

Hello there, today we'll look at PonderNet learning to ponder by Andrea Bonino, Jan Balaguer


and Charles Blondel. This paper on a high level introduces a recurrent architecture,


or a principle of recurrent computation for deep networks, that essentially says the network


recurrently computes its output at each step. And at each step, it can decide to stop now,


because it is satisfied with the answer that it has. The idea is that at a complex task,


you can compute for many steps, because it requires many steps of thinking, and then give


the output. And for an easy task, the network can decide to output right away, because it already


has computed the solution. This decision can be done on a per sample basis. So for each sample,


the network can decide when it's time to give the final output. And this is, this is not


necessarily a paper that just, you know, make something bigger and then pushes state of the


art on some benchmark. And that's why it piqued my interest is that it tries to re rephrase a


little bit how we think about the connection of deep learning and algorithms like classic


algorithms by themselves. Essentially, this is a dynamic if condition in this algorithm that


decides when it's when it's time to stop. And I appreciate that, you know, it not everything has


to be state of the art pushing here, this is simply a cool method to do something that's


relatively new. Of course, things like this have been done before. And they are discussed at length


in this paper, how this paper is different from other papers that do similar things. And it does


push state of the art, just not on benchmarks that you might be super duper familiar with.


But yeah, it's it's a cool paper. It's a short paper, the idea is pretty simple,


and it appears to work. And yeah, that's exciting stuff. So we're going to dive into this paper,


have a look, have a look at what's new in this particular model, how it works. And as always,


if you have feedback, leave a comment, subscribe, I'd be happy for that. And yeah, thanks for being


here. Okay, so in the abstract here, they say that in a standard neural network, the amount of


computation used grows with the size of the inputs, but not with the complexity of the problem being


learned. So which is true, right? In a standard neural network, you have a forward pass, be that


in a fully connected neural network where you have, you know, you have your input, and then you


go layer, layer, layer, layer, layer, and then you have your output. This computation here is always


the same, no matter the input, even in a recurrent neural network, right, you have kind of an input,


right here at the beginning, you have a layer, then you have an input again, and then you have


this that goes into the same layer. And then you have the next input that goes into the same layer,


even a recurrent neural network usually usually just does the same forward pass, this is a little


bit different if you have something like a language model that can emit at some point a, you know,


a stop token, or an end of sentence token, at which point the computation essentially stops.


But it's a little bit of a different thing than we consider right here, right here, we consider a


neural network that has to find the answer to a particular problem. And we're going to see the


problems down. But one problem that they present is the parity problem. So the parity problem is


you get a string of zeros and ones, I think there is also negative ones in there. But I think they're


a bit for a distraction. And the answer you're looking for is, as a whole is the parity. So the


amount of ones in this string, odd or even, right. So this requires a, let's say, an integrated view


of computation, this is essentially a classic algorithm that you have to perform over this


string. And neural networks as good as they are in computer vision and speech recognition,


they are having trouble with simple algorithmic tasks like this. So the idea of this paper here


is that, well, it doesn't make sense to apply neural network that always does the same amount


of compute, right, I shove this sequence just like in here, it doesn't make sense, because,


you know, if there's just a single one in the string, and I see that right away, I can give


the answer right away. However, if there's if it's a long string, and it has a bunch of ones,


I might need to think about this problem for a while, and thus adapt the number of computation


steps I do in my head, I might, you know, first, if I look at this string, I might first connect


these two, you know, and then that's two, and then I might connect these two, that's two again,


and then I might connect these two, that's four, there's nothing here, there's nothing here, right,


okay, four. So that's kind of like 123 steps of computation. So that's the the rough idea,


whereas this if the string was shorter, and and more regular, I might need less computation.


So they say, to overcome this limitation, we introduce ponder net, a new algorithm that


learns to adapt the amount of computation based on the complexity of the problem at hand.


Ponder net learns end to end the number of computational steps to achieve an effective


compromise between training prediction, accuracy, computational cost, and generalization.


So we are going to see how they do this. Yeah, exactly. So they then they go into the the tasks


that are experimental tasks in this paper are sort of these constructed tasks where people know


you need this dynamic computation, they're not going to, they're not going to compete on like


image net or something like this. So the majority of the paper is in in contra posing their model


against this ACT model, the adaptive computation time, I believe. So there have been previous


attempts at doing dynamic computation time, yet, either they have. So it turns out, they're kind


of finicky. And this model here, this ponder net model has a bunch of advantages. They say they


present ponder net that builds on the previous ideas. It's fully differentiable, which allows for


low variance gradient estimates, unlike reinforce. So a couple of previous attempts have been with


reinforcement learning. So let's just learn the number of steps or when to stop using reinforcement


learning. And that, as you might know, is very, very noisy. It has unbiased gradient estimates,


which is also unlike other models in the past. And yeah, so they say this has consequences in


all three in all aspects of the model. In ponder net, the halting node predicts the probability


of halting conditional on not having halted before. This kind of seems obvious, but apparently that


no one has done this so far. So what do we need for an architecture for ponder net, they say this


down here. Essentially, that's the architecture, it's an inline formula, which, you know, but


that's the architecture. So what you need is you need an input, okay. You need an input, which is


x, your input, and x is transformed into a hidden state. This is, let's say the hidden state at step


one, those two, or you can also reformulate this as just the hidden state, the hidden state is going


into s, the so called step function. And that's the recurrent function right here. So into this


step function, you can put anything you want, you can put like a CNN inside, you can treat this as


an LSTM since we're going to apply it recursively, sorry, recurrently. And anything you want can be


the step function as long as it can be applied recurrently. So this step function is going to


give you the next hidden state, right? So you can see it's a recurrent neural network. However,


it is also going to give you the output at that particular point in time. So y one, I guess that


be here. And it's also going to give you this number, lambda n. Now, what are these? So from


here, you could apply the step function again, you'd get h3, you get the output two, and you'd


get lambda, sorry, that's, that's a one, that's a two. So it seems like it's just a recurrent


neural network. And if I were to put push this to the end, right, I got give my h h h. And then at


the end, I get my y n. And I treat that as the output of the computation, then it's just a


recurrent neural network. However, as we said, the network can in this case, decide to stop


anywhere in between, for example, if it decides to stop at this particular step, then that would be


the output of the computation. So every computation step, the network computes and potential output,


a suggestion for an output. And then it also thinks about whether or not it really wants to


answer with that output, or whether it wants to continue and to do another step, essentially take


another shot at answering the question because it doesn't yet have the correct answer. And that's


where this lambda thing comes in. So the lambda is a probability of stopping essentially. So here


you can see the output lambda is a number between zero and one. And that is the probability of


halting. This is the output considered that the network halts. So whenever this is one, the network


will halt conditioned on the fact that it hasn't previously halted. Yeah, it seemed as I said, it


seems obvious to formulate it like this, because you can, you can only halt if you haven't previously


halted. But apparently, previous models have simply output a number that is sort of the


probability of halting in general, which doesn't give you a bias, sorry, an unbiased gradient if


you try to back propagate through it. So if you consider the lambda to be like this, if you unroll


for an entire training run, then you get, we get the probability of halting at any particular step,


this one. So this is what this is what the previous networks would have estimated directly.


However, this network estimates these lambda's these ones here, you can see how you can compute


the probability that for example, the network halts after three steps by multiplying up the


probability that network has not halted, which is this one at step one has not halted at step two,


and then the probability that network halts at step three that it given that it hasn't halted at


the previous steps. So that is a valid probability distribution. It's a generalization of the


geometric distribution. And essentially, it encapsulates a decision tree, right? So


at you're at the beginning, you can halt, sorry, let's go a halt or not or continue.


If you continue, then again, you can halt or you can continue. If again, you can halt or continue


and so on. And all of this. So if you want the probability that the network halts after,


you know, this the third step, then you would consider this node, which means that you'd


multiply that you multiply up these paths right here. And that's the probability that it holds


after three steps. Okay, so the network can output this lambda at every step, if the lambda is high,


then the network halts, of course, at inference, this is done probabilistically. Now at training


time, this is done a little bit differently. So you I hope you can see at inference time,


you simply go forward, and you get a lambda, maybe the lambda in the first step is point one. And


then you flip the coin, a biased coin, right? If if it comes up heads, you stop with the probability


of point one, it comes up tails, which is a point nine probability you continue, then maybe at the


second step, it's it's point zero five. So maybe maybe you stop, but probably you won't stop. And


then at the third step, it like comes up point nine, the network thinks Yeah, I should probably


stop here. And you sample from that. And yes, you you might indeed in nine out of 10 cases,


you actually stop there. So that's inference. How about training? How about we train this


thing? During training, what we do is, again, we input x, our input into an encoder for the


hidden state. And as I said, you can also input x all the time into your step function, as you see


right here. But what you do is you unroll the network for a number of steps, right? independent


of these output nodes independent of the sorry, the halting probability, let's say we we unroll


it for for five steps right here. And at every point, we get a output and a value y three, y four,


this is lambda two, lambda three, lambda four. So at training, we simply unroll until a given step.


Now there are some technical difficulties with doing with unrolling for a finite amount of step,


like how do you normalize the probability distribution, because essentially,


this tree can go on until infinity, they find okay, we, we can simply unroll until kind of the


rest probability, the probability we haven't used yet is is really small, and then just load that


all into the last step. But these are technical difficulties that you really only care when you


then go and implement. However, so we unroll for a number of steps. And then our, we consider all


the outputs at the same time. Now, this is one big difference, I believe, to one of the previous


networks to this a CT. So what a CT does is it always unrolls, and then the the output of the


network. So for a CT, the output of the network would simply be a weighted output of the lambda


i y i. So the output of the network is always a weighting between the different steps, okay,


and the network can decide, okay, how do I want to weight the individual outputs, whereas here,


it's different here, the output is really either y one, or y two, or y three, or y four. And to in


order to pack this into a single loss function, what we can do, sorry, I should probably leave


this. In order to pack this into a single loss function, we simply take, okay, what's the loss,


what would be the loss if we answered y one, right? What would be the loss, and we weigh that


by the probability. And we say, okay, what would be the loss of y two, we weighed by the probability


that the network output, right? So now if we and so on, so plus, essentially, we compute the expected


loss given the probabilities that the network has output. So now if we back prop this,


we back prop through these losses, we have, of course, two paths of back propping. So we back


prop through the wise, which means it's at some, so there is a loss, right? And both these things,


and these things go into the loss, right? So the loss is, well, how bad is this times how


probable it was, so on. So the back propagation path would actually attack at two different paths,


you can see. So the back prop goes into y, because you want the network to compute a,


a better output. But the propagation also goes into the lambda, because you want the network to


get better at estimating when its output is good, and when not this, I see a little bit as a tricky


situation, because usually, this, this seems a little bit unstable, just from experience from


other papers, and so on. If you have a back prop through two different things, especially that are


appear to be multiplied together, and that, you know, the network can now trade off one versus the


other, which might you might think is desirable, right? It can either choose to make its output


better, if it wants to keep the probability high of outputting this thing, or it can just reduce


the probability that it's going to output whatever it wants to output. And, you know, then it doesn't


have to necessarily make the output itself correct, because the loss, the loss won't be as high for


that particular thing, because the probability of outputting it is low. So the network essentially


has a choice. As I said, this might be desirable, but usually that's kind of unstable. And I think


this is just my personal opinion, I think a lot of them, why this might work might rest on whether


or not or let's say the complexity itself of assessing of making why better versus adjusting


these probabilities, of course, yeah. So you see, if the output y is very complex, right, then this,


you know, the same gradient signal for that might mean much less than simply reducing the probability.


So if the output is very, very complex, right, not the problem, but just the output itself,


like how to arrive at an output, if the output is an entire pixel map or something like this,


and that has dependencies, and so on, the network might just choose to always reduce the probability,


because it's like, well, how am I gonna, how am I gonna make this better at all? I don't know,


I can just reduce the probability, I'm going to output this crap, right. And it will probably do


this then for every, you know, single step, which you know, if it's a complex problem makes sense,


but still, that's it, that would be a bit my my fear here. And the this is not really discussed


in the paper itself. So I think the fact that this works might rely on sort of a balance of the of


the complexity or information content that you get from the loss at the output node versus the loss


at the probability node. So, okay, enough about that. So in Yeah, during training, you simply


compute the expected loss weighted by the probabilities. And then you can back prop


through that. And I hope you can see the difference between these two. One is a, they both seem to sum


up somehow the outputs weighted by these these factors. However, one considers the actual output


of the network to be a weighted combination of outputs of the individual steps, where the other


one says, no, no, no, the network output is actually one of them, we don't know which one


ergo for the loss, we need to compute the expectation of the loss, that seems to be a bit


of a, let's just say, yeah, it seems to be a more reasonable formulation, though, in hindsight, you


can say many things are reasonable if they work better, right? Yeah, so they discuss things like


maximum number of pondering steps and so on, again, which I think is a technical detail. And


this is interesting. So there you have the training loss, as we just discussed. Now, we've discussed


this part right here, which they call the reconstruction loss, because you have some kind


of desired y, and you have a y that comes from this. And that was a little bit wrong here in my


formulation. Of course, the expectation you don't have, you don't want to take the lambda, you


actually want to take the probabilities that each thing happens, which means that you need to compute


this p number, you know, going along the this tree, as we did, because the p is the actual


probability that you reach that node, whereas the lambda is only the conditional probability that


you reach a node, given you were at the previous node. So yeah, consider, consider that if you if


you are crazy enough to implement things straight, as I speak in the videos, lucid rains, shout out.


The second part of the loss here, and you can see this is a hyper parameter. So you're going to


trade off two of two losses right here. Because right now, we saw okay, you can either continue


or not continue. And for the network, you know, it might actually be easier, as I said, if the loss


of the output comes reasonably complex right here, might be easier to simply say, Well,


in this case, I'm just always going to reduce my probabilities. You might counteract this with


having this number of steps, not like maximum number of steps. But essentially, this term here


is what counteracts that really, there is a regularization term on these probabilities,


as you can see right here. So we regularize with the KL divergence, which is sort of a distance


measure. Don't tell this to a mathematician. It's a it's a divergence, it's a sort of a distance


measure between the distribution that the network outputs for the steps. And this thing right here,


which is a geometric distribution with this parameter. And this parameter lambda p is another


hyper parameter. So what does that mean? Essentially, if you consider here the number


of steps that the network thinks, right, think things for what you regularize for this distribution


right here is a geometric distribution, you know, go something like maybe no, something like this.


So essentially, a geometric distribution is set exactly computes this tree that we computed,


right. So at each step, you can essentially stop. And the question is after you know, this


distribution gives you a indication after what's the probability that you stop after one step,


two steps, three steps, four steps, considering the fact that in order to stop after four steps,


you already have to have made three non stopping steps, except in the geometric distribution,


the probability of continuing is always the same. Whereas in our network, our network for each node


in the tree can output a different probability. Otherwise, you know, there'd be no point, we can


simply put in the fixed distribution. Now, what that probability is of stopping at each point,


that's exactly this lambda p hyper parameter right here. So you regularize for a KL for this,


which means that you tell the network, look here is a a reasonable, reasonable distribution


of when you should stop. So you should stop. So it should be, you know, somewhat probable that you


stop after one step, and somewhat probable if you've already done one step that you stop after


two steps, and so on. So you give it sort of a default probability of stopping after each step.


So if this is 0.1, for example, you tell the network essentially, look, at any given step,


there's like a default 10% chance that you should stop I as a designer of the algorithm thing,


that's a reasonable prior to have now the network can decide differently, the network can decide,


no, no, no, no, no, actually want to stop way earlier, right, like, like this, it puts much more


emphasis on the first steps, which of course, in turn, because you need to normalize,


put less emphasis on the latter steps. So the network can still decide to violate this prior,


if the if it may reduce the loss for enough. So this is, as I said, a trade off, there are two


hyper parameters, the geometric distribution, shape, and the amount that you regularize by this


KL divergence. And, yeah, so now we come into the experimental results. And these are pretty,


pretty neat, because, yep, they, I think these are straightforward experimental results,


they're not super big, large scale results or anything like this. But they show that look on


tasks where we sort of know that this dynamic computation has an advantage, our model will


outperform both previous attempts at dynamic computation, and especially networks that have no


dynamic computation built in whatsoever. So this is the parity task, which we're going to look at,


as you can see here, the orange is this a CT, which is the previous work that they compare


most with that is most similar to them. You can see in terms of accuracy, pondernet,


that beats this network by quite a bit also appreciate the error bars in this one, they


almost overlap, but they don't. So you can say that you're definitely better. And interestingly,


the number of compute steps, even though yeah, the error bars overlap as well here, but pondernet


itself needs less compute steps, which might be, you know, I don't I don't know why why exactly


that happens. But you can speculate that it is because pondernet sort of fixes on a single like


it outputs a single answer, whereas the a CT, it outputs this weighing of things. And therefore,


when it when it outputs, let's say the first step answer, it always needs to consider that this


needs to be compatible with potential future steps. So just formulating, just formulating


how a CT output stuff, it seems like it becomes a lot less dynamic, because the output is always


a weighting of different outputs. And therefore, the first steps they have to, they can't just


output what they think is the correct solution. But they sort of already have to incorporate the


future and estimate well, if I'm going to continue computing, then, you know, there's going to be


stuff added to my output right here, and they have to take this into account. So it can be,


ironically, less dynamic of a network. And that's why I think pondernet might need less steps here,


I might be totally wrong, though. So this is the parity task. And specifically, they train with


string lengths between, you know, so this is a string length of one, and then string length of


we've before we had like eight, right, something like this. So they train up from one until 49


lengths, one until 49. And this is a little bit important, I think, because their training set


contains all of them, which, you know, this is a little bit of an experimental trick, right.


So in order for your network, what you wanted to learn is kind of the general principle of parity


independent of string length. So you construct the training data set to be sort of a distribution of


lengths of string, rather than just strings of a fixed length, and then you assess their parity.


So yeah, that's maybe a bit of a lesson for if you do experiments, construct your tasks


themselves already, such that they help find the correct solution, right. So they train with strings


of length one up, up until 49, right. And then they try to extrapolate, which is this B right here. So


this is extrapolation, where then they test. So first here, they test, they train on small strings,


they test on small strings. Here in B, they train on the same small strings up till length 49.


But then, as I understand it, they give it length 50 to 99 or so in two or 96. It says it somewhere,


just longer strings that it has been trained with, right. And now that the setup is, you know,


clear, it's clear why they did the different length strings in the training set and not just


fixed length strings, because there's a reasonable chance the network does not learn to extrapolate


just from one particular or two particular lengths of string. Nevertheless, they test,


how does the network extrapolate to longer strings. And you can see right here that ACT,


even though it also has been trained on the dynamic length strings, it is that's 50%,


right? That's pure chance. So it's a parity test, right? It's the output is either odd or even. So


ACT just gets a pure random chance as a result, whereas the PonderNet, as you can see, has like


an accuracy of point nine, which I guess is pretty good, especially on strings that are so long,


you've never seen them. So what can we read from this? I'm not exactly sure. There's always the


possibility that, you know, they've just trained ACT wrong or something like this. But it's also,


it's also reasonable to say that just how the previous models were constructed. Either they


didn't learn the concept or their their output is just weird in the way ACT is, or since ACT has


biased gradients estimates and PonderNet doesn't, yada yada, we don't know. What we do know is


that in their experiments, this PonderNet was actually able to solve the extrapolation task


right here. The interesting thing is that if you look at the number of compute steps done,


you can see that PonderNet in contrast to what it was trained with during inference. Sorry,


that's an alarm. In, in contrast to what it was trained with during inference during inference,


it has like two point between 2.5 and three steps, let's say three steps, computes for about


three steps during inference time. That's what it decides on for the smaller strings. Yet the same


model right train on the same strings, this is the same model during inference time on the longer


strings, all of a sudden, it raises its compute to five steps. Whereas ACT, okay, ACT doesn't


work in the in this one, it just decides to stick around two or three steps as it does in training,


right. So the authors sort of claim that this is good evidence that PonderNet learns to solve the