longer range compositions it might not produce as correct of a reconstruction but it can


Jukebox: A Generative Model for Music (Paper Explained)

learn long range dependencies such as the structure of a song or the structure of a


verse or something like this so these are independent of each other and they make an


argument as to why so people have tried to kind of share these architectures but have


found that mainly the models will will basically ignore the top two and only go over the course


grained ones so that's why they completely they separate these at this stage of training


right so we have trained three different VA is at three different scales of music to always


reconstruct the input what does that give us that gives us a distribution right here


that gives us a way to take a piece of music and map it to this hidden space to this very


compressed representation in this quantized world right I've said before this is a very


compressed representation of your data why can why can you do that sorry what's that


useful for what you can do now is you can try to sample from in that hidden space so


instead of sampling music we have no clue of how to sample music unless we are given


some music what we can do is we can say maybe this thing here because it's compressed it


kind of so if we just sample a waveform it's very unlikely that it's music and but if we


sample these hidden things you know it's quite likely that if we feed it through the decoder


something will come out and even better maybe our data set in this hidden space follows


a kind of a simpler distribution one that we could learn right so we're trying we're


going to try to learn a prior distribution over the distribution of codebook vectors


and that is naturally going to be a joint distribution between the top middle and bottom


VQ VA is and we can decompose this into the following things simply by applying the standard


probabilistic transfer algebra transformation and we can then they say when train separate


models sorry about that we train separate models for the top level prior Z top the middle


and the bottom so what that means is basically these are now neural networks if if you read


something like this in a paper like this this is going to be a neural network that takes


the right side as an input and produces the left side right so you start out with this


one this is a neural network that simply takes as an input sorry I'm going to draw this neural


network takes as an input maybe something like a Gaussian super prior right you sample


from that and that will as an output give you this Z top then the next neural network


will take this as an input and will give you Z middle as an output and then the final neural


network will input the two of those and give you Z top and you can train these neural networks


simply by kind of training a prior to produce this thing right here you simply use your


data compress it to the hidden space and then train a neural network to produce that distribution


and you can do this in any number of ways you can use classic VA's you can use you can


sorry you can use here they say we use transformers with sparse attention as they are currently


state of the art in autoregressive modeling and they say we propose a simplified version


which we call the scalable transformer that is easier to implement and scale right but


they see you see they model this distribution with these scalable transformers all right


so now what do we have we have a way to sample these hidden vectors right so we don't need


we don't need this part anymore this part sorry about that this part here was just used


for training we can we now have our transformers they take nothing as an input or they they


take like a Gaussian as an input and they can directly output this hidden representation


so we could technically sample from that and then just push it through this decoder of


the VQVA but the question is which of the three do we take and wouldn't it be great


if we can combine them because if we simply sample these this higher scale one we just


get not very long range dependencies right because that's what the VQVA learned if we


just sample this one then we just get a course music and we can sample all three but they


will just give us three different tracks of music so we want to combine the three decoders


into one somehow and that's we do this through these up samplers so what we'll use what we'll


target actually is this bottom one we target this one because this one gives us the best


quality music right because it was trained with the shortest time scale we're going to


try to in to take the other signals and and influence it so we'll start with the top level


prior right that will produce us these transformers will give us a sequence a sequence of tokens


in the hidden space that is very course as you can see here and then we'll feed that


into a up sampler and these up samplers again are on the neural networks that can connect


the different scales with each other right so you can connect this to this it's basically


like conditioning the model that produces the sequence on this sequence right here and


again we use an up sampler to up sample this to the finest scale and that we feed in the


bottom scale and then we get our music now throughout all of this do you have conditioning


information here which is a bit of an addition to the model so the conditioning information


can be things like artist genre and timing and this is seen appears to be pretty important


because you kind of want first of all want some variety and then second of all you sort


of want to control what music is produced and you don't just want to train a this model


for us one single artist because you have much more data across all of music so this


conditioning information is just included here via another neural network and you can


find all the architectures for all of these models in the paper it's not particularly


important I believe how exactly you include them but the fact that you do the last thing


is what they do is they do this kind of windowed sampling so in order to produce music you're


going to have to produce these these slices of music right here but you sort of have a


maximum length here that your models can handle and this is usually not the length of a song


you may know transformers and so on they have usually have token limits of like 512 tokens


in terms of audio that's not that much so what you do is this windowed sampling where


you sample something and then you condition basically on the first part and then you just


sample the next thing and then you again condition on the first part and then you just sample


the next thing the next few ones and that guarantees that each of the sampling steps


is basically conditioned on what comes before as you see up here so you would always sort


of condition on a part produce the next part alright and they say you can also basically


condition on so you can feed in you don't have to sample the very first one you can


also feed in an existing song in order to prime the system so what you can do is if


you have a beginning of a song then you can let the system finish the song by simply taking


the song running it through the encoder that we produce during training right you get these


hidden representations so you don't actually have to sample them from your prior and then


you run this generation process as if this came out of your prior instead of what you


sampled okay so let's have a look at how that sounds or listen this this is an explorer


they release many many samples from this and the part here where we're going to listen


is called no lyrics conditioning so as you can hear this is already pretty good music


and the genre is American folk the singer is Peter Seeger sorry Pete Seeger this already


sounds very authentic but you can hear that the lyrics are just kind of mumbly right and


that's because the model is basically asked to come up with lyrics from as pure audio


waveforms and that results in some subpar lyrics basically it just produces phonemes


that sound like the singer it doesn't produce entire words and of course it also doesn't


produce sentences that make any sort of sense and that's why they're building in an additional


thing to do lyrics conditioning so with lyrics conditioning the idea is that in the conditioning


information you also add lyrics so here is plus text so you add text and then the model


is basically can also look at the text now you never you still you still so even before


we had music with lyrics and the decoder was always asked to reconstruct that and so none


of that changes that's why it has learned to produce phonemes right but now the decoder


can also and also the encoder did the system can look at the lyrics that you provide right


here in order to help with its decoding so technically it could learn to bypass the encoding


of the exact way the lyrics are uttered and it could just look at the text that you provide


now this of course requires that during training you provide the lyrics of the song that are


actually that that you feed in but also it means that during decoding if you sample you


can then provide your own lyrics and look what happens so they say they provide lyrics


they always have to provide lyrics for chunks of audio so our data set includes song level


lyrics but to make it easier we train on shorter 24 second chunks of audio and this is partly


to make it easier for the model but also partly because those appear to be the limitations


of these systems right if you have transformers in there and whatnot 24 seconds of raw audio


waveform is locked so they have this problem of they have this problem of they have a song


from here to here and they have the lyrics blah blah blah and they need to know which


lyrics belong to which part of the song and usually it's it's monotonic right and linear


because you get the lyrics from some lyrics website this blah blah blah but you don't


know particularly to which 24 second chunk they belong so they say first of all they


started with simply linearly aligning the lyrics but then they had some they they had


some problems with fast songs so they had some heuristic here but ultimately the the


decoder needs to learn to attend to these lyrics and these the graphics like this you


see here is the music token position and lyrics token position so here you see the the system


learns that for example if it has it this music token needs to attend to this token


in the lyric so you can by inspecting these attention heads that you have on the lyrics


text in the system you can see which lyrics the model is paying attention to and the fact


that it learns to pay linearly attention to these things is kind of a confirmation because


you you don't you give the whole text or at least the 24 second chunks of audio you give


that at once as a as a text right and the fact that it learns to linearly attend to


the tokens is a confirmation that it actually includes that information into decoding and


that is a pretty gives you pretty much better results so we can maybe go to classic pop


it's christmas tiling you know what that means oh as i like to treatib khasare


as I like to treat us here, you'll be


So this are unseen lyrics. So the model has never seen these lyrics, right?


It was just asked to produce classic pop in the style of Frank Sinatra with these lyrics.


And that's what it came up with. That is pretty, pretty, pretty cool, I think.


They also have re-renditions where they basically feed, I believe, feed in the original lyrics.


We conditioned on lyrics seen during training.


And they have fun songs.


And in the fun songs, I like the hip hop in the style of Kanye West,


where they provide the lyrics of Eminem's Lose Yourself.


I'm grooving. I don't know what you're thinking, but this is cool.


And they can also, as we said, do these completions where they start with part of a song.


And just, I have to do this. I have to do this.


Hi there. So the first version of this video was copystriked because what you would hear would be the original,


never gonna give you up, like 10 seconds of it, and then followed by what the model continues with.


So as a substitute, you're now going to have to listen to me.


I hope that suffices.


Almost as good. Almost as good as the original.


So as you can see, the results here are pretty, pretty cool.


And I want to show you one last thing, and that is this Christmas song in the style of Frank Sinatra.


I believe it's this one right here.


And the special thing here is it's again, classic pop in the style of Frank Sinatra.


And you see on the bottom here, you see on the bottom which of the lyrics it's attending to.


And you see, you know, this this graph right here that shows you that first it's attending linearly through the lyrics.


But then it kind of jumps around and attends to different things because it doesn't it doesn't it doesn't just continue.


So. It's Christmas time.


And you know what that is. It's pop sometimes. Some red leather makes me see where I am.


That smuggle man of old time and time.


This is great. So it kind of falls out of this linearly attending to the lyrics.


And probably because there was sort of a pause in the lyrics and maybe this is just more than one audio window.


So it doesn't have this autoregressive property anymore.


And then it doesn't find the proper place to attend anymore.


And so just again comes up with sort of babbles. But it sounds pretty, pretty cool.


Yeah. So this they have released many, many samples here, some cherry picked and just a lot of samples with unseen lyrics, re-renditions and so on.


This all is very cool. They have their training set up described.


I believe they also released their code. Many more results in the paper of how to make this thing work if you want to do that yourself.


And with that, I invite you to read the paper.


Autoregressive Diffusion Models (Machine Learning Research Paper Explained)

Hi there, today we'll look at autoregressive diffusion models by Emil


Haugeboom and others of Google Research. This paper on a high level proposes a


new type of autoregressive model, specifically one where variables can be


decoded in arbitrary orders. This is akin to the new types of diffusion models


that have been used for generative models and it essentially amounts to


something like BERT in sequence. The training objective is made such that we


can decode variables as we like. And I can show you the results. The results


are going to be that we can, for example, sample pictures pixel by pixel in order


to make a generative model. So rather than GANs, which produce pictures all at


once, or what we had so far autoregressive models, but with a fixed


order from for example, from left to right. Now we can do it in any order. In


addition to this, they introduce techniques where you don't have to go


pixel by pixel, but you can do multiple pixels at the same time and speed up by a


lot. So this is a paper which is also community informed. So this is a


community informed paper review, which means that in our Discord server, we have


regular paper discussions. This was one of them. I tried to pay attention. I don't


I can't say yet whether that has worked. But I'm trying to try to recount here a


little bit also. So my opinions are influenced a lot by what was said at the


paper discussion. If you want to influence my opinion, feel free to join


our paper discussions. Okay, so there we go. They say they introduce these


autoregressive diffusion models, which is a model class encompassing and


generalizing order agnostic autoregressive models and absorbing


discrete diffusion models, which they show are special cases, yada, yada, yada.


They say they're simple to implement and easy to train, unlike standard


autoregressive models, which you might know as LSTMs or standard autoregressive


models or GPT type transformers. These are all autoregressive models. They do


not require causal masking of model representations, and can be trained using


an effective objective similar to modern probabilistic diffusion models that


scales favorably to high dimensional data. At test time, the ARDM support


parallel generation, which can be adapted to fit any given generation


budget. So you can trade off how long you need to produce a given sample with


how with the quality. So you can say I want it faster, and you'll still get a


sample, you'll just get a like a lower quality sample. We find that they require


significantly fewer steps than discrete diffusion models to attain the same


performance, yada, yada, yada. They also do lossless compression with it. Okay, so


what's the deal with autoregressive models, right? If I want to, if I have a


bunch of variables, let's say I have a piece of text or something like this,


what I'd have to do is I'd, you know, what you'd usually do in GPT, you give a


prefix, and then you decode a token by token from left to right, right, a cat,


and then the model has to predict sat on the and so on. So you predict from left


to right, one by one. That's also how you train, right, you train from left to


right, you predict from left to right. And with text, that makes kind of sense,


because we also read from left to right, right. However, it would also make sense


to do this in a different order. So if you have a cat, and you first decode,


let's say, Matt, right here, then if you first do that, then it becomes pretty


clear what's in here. So in order to give the model sort of the biggest


freedom, you could let it decode in other places first, and then it could


decode the Matt here first, which would sort of determine the rest of the


sentence. Whereas on the top, the model already sort of has to have in mind what


it wants to say later, like the fact that there's Matt here, in order to produce


all of these things here. But in this way, the model could predict that first,


and then the rest is sort of determined so it could impute that a little bit. And


this all of this is just to show you that it's not the only way to decode


left to right and even more so in something like image GPT. So you have an


image. And in again, I produce the whole picture as one at once. But in something


like image GDP, what I do is I start at the top left, and I simply start


producing the pixels left to right, top to bottom, right, that's it. And there


is not really a reason why this is the best order to produce things out. It's


simply that we train in this way. And that means we have to predict in this


way. What the autoregressive diffusion models do is they say, we're going to


train a model that can produce a sample in any order. It doesn't matter which


one. So we could start off with like this pixel, then go to this and ask for


this, then ask for this. We can even ask the model something like, which one do


you feel best about? Like, which one are you most sure about? And the model can


tell us and then that's the one that we could, we could decode further. We can


also tell the model to decode like three pixels at a time, and then these three


pixels and so on. So that's the trade off I mentioned. So this is how it looks in


practice. What you're going to have is you're going to have a neural. So here,


the vector is your sample, right? And usually you would decode top to bottom.


That's sort of the analogous to left to right. That's what you usually would do.


However, in this model, you can see first it's empty, so nothing is decoded yet.


You have your neural network, you have your predictor, let's say, that predicts


a distribution. So for every single item in the sample, it predicts a distribution.


So these here are categorical variables. So it's going to be predicting a


distribution. And so all of these, for example, if the years are pixels, all of


them predict color. So prediction is made for the whole image and not just for the


thing you want to decode. And after that, you decide on one of them that you


actually want to decode, you sample that or you take the maximum class or whatever.


And then you continue right then the next step. So in the next step, you have


the same sample, except that one of the values is now already decoded, the other


ones are still empty. Again, you use a neural network to predict a distribution


for the entire image, you'll see that, you know, for technical reasons, even this


here is actually predicted, it doesn't need to be. But the important part is


that you're going to predict the entire image at once. And then you decide to


again, decode one of them, that's your choosing. So this one, and you can see


that, you know, this how this goes on, specifically, which ones you decode is


given by a by this thing right here, this sigma is a variable that stands for a


given permutation. So what you would do is if before, before you sample, you can


select a permutation, you can say here is the the order in which I want to decode,


and then you decode according to that. But in my mind, it doesn't matter even if


you decide on the fly, so you can decide on the fly, you know, here's here's my


desired order, I want to decode in that way. Now, if this is seems familiar to


you, if you have seen a model, something like this already before, then if you're


thinking of BERT, you would be sort of correct. So even the paper says that this


is kind of like, you take the BERT model, and you just kind of stack it, or you


just repeat it, notice the this here, these are always the same neural


networks. So the same neural network will predict every single step right here.


That's why it's an autoregressive model, right? Because you input the output into


the same neural network again. So what do you do in BERT, you have a bunch, you


have a sentence, right? A cat sat on. If you do masked language modeling, you put


that through the neural network, right, that's BERT. And out comes one sort of


output per token. Now, what you do when you train BERT, you mask some of the


tokens, right, for example, this one, and this one. And then BERT predicts these


BERT predicts these at once, this one and this one. And what you want to do, sorry,


BERT predicts these tokens at once. And that's a categorical distribution. That's


a classification into your vocabulary, right, which word was masked right here.


So what BERT needs to do is BERT needs to infer from the words that exist, what


other words could be here. Notice one interesting property about BERT. The


question is, of course, you know, why do we even have to do this in a particular


order? Can't we just, if we are already predicting all pixels at once, right,


do the network already for each pixel that's not yet there predicts a


categorical distribution? Why can't we just sample that? Right? And the answer


is because these things are not independent. So if I, if I simply, if I


have a bunch of variables, right here, let me use this one. If every single one


of these nodes gives me a distribution, or let's say, just the ones that are not


just the ones that are not filled out yet, right? Here, I have two pixels or two


elements that are not filled yet. Now I'm going to take my input vector. And I


want to use that to predict for every of one of these two pixels. What's the


distribution of values that could be there, right? So the distribution of


values could be well, the first number one is really popular, to not so much


number three a little bit. And here it could be, let's say, number one, also


popular. Number two, a little bit number three, not that much, right? Now, if, if


those two are independent, then we could totally fill these in at the same time.


But they might not be right, pixels typically aren't independent if they're


in the same image. For example, right, if the entire, if the pixel here is blue,


that makes it makes, it's not independent of the fact of whether the


pixel, you know, right next to it is blue. And that doesn't only count for


pixels next to one another, that counts for pixels farther away, of course, the


further they are, the less dependent they probably are. But still, I can't just


sample both independently, I need to, in order to sample one, I need to know what


the other is. So I need to sample this one first and not just have the


distribution, I need to commit to one of the outcomes before I even try to sample


the other one. And by committing to one, that will actually change the


distribution of the other one, because this here assumes that the other pixel


will be according to this distribution. However, once it's sampled, it's no


longer this distribution, it's actually one of these things for sure, like it's


maybe this one for sure, if that has been sampled. And that will change in


turn, the distribution. So what I want to do is I want to put the whole thing


through the neural network again, in order to really get the true


distribution of this node right here. So maybe it's, maybe it was really


likely that number, class number one was it, but now that it sees well, this other


node really has chosen number one, so I'm probably not number one. So I am


class number two, maybe. I hope this is a bit clear that even though we can train


in BERT style, so we can predict all the things that are missing at once, what we


cannot do is we cannot decode all the things at once, because what some of the


elements or all of the elements are dependent on all of the other elements.


And being dependent means that we they need to know what the other elements are


before they themselves commit to one of the classes of their distribution.


And that's the whole point of it. The point is these models, they train like


BERT, but they decode like autoregressive models, except that the


order isn't fixed, the order can be any order you want. And they do actually


apply this to text. So just so you can see that this how this looks. So here's


how it looks. This is a character level language model, right? So the it starts


off with a relatively empty empty sentence, let's say so the underscores


are just empty. These are variables that are not chosen yet. And then it's going


to fill in a bunch at the beginning, you can see that right here. And it's going


to fill in some more, right. So here is going to fill in some more, you'll notice


that all of the ones that existed, they should still exist, do they? Do they? I'm


not even sure like, here, the X still exists, the I still exists, this I still


exists, this I still exists. Yeah, okay. So all of the ones that were there, they


are still there. But there are just more now. And then more are imputed more are


imputed until you finally come to the fully imputed sentence. And you can see


that these are actual samples from their model. So on text on character level text,


it's not yet, like super good. The sentence doesn't really make sense. I


don't think that's actually an English word. It sounds English, but it may not


exactly be an English word, a potentially unsucked proof or inject


operational weapons in the game car, us individual. So yeah, this is it's


unclear because these are the sort of the beginnings of these types of models


whether that's the case or whether it's just much, much, much more, much better


objective to just train autoregressive from left to right. Because there's also


trade offs, right? You predict every single thing at once in your loss


function has to split between all the things that there are to predict. However,


if you just train left to right, then your loss function can focus fully on


what the next token is right in the given order. So you gain the ability to


decode in any order you want. But that has a trade off, namely a performance


trade off, because the model that specializes in one particular in one


particular order will always beat you. So let's go back. And I think that's, you


know, that's the entire point I've sort of found you can simplify this


relatively much by essentially saying, you know, this is BERT training, but you


decode one after another. And you can, I'm pretty sure the way this, this is you


can't you could take, you could take the pre trained BERT checkpoints and sort of


decode like this. However, the problem is, of course, these BERT checkpoints, they


have been trained with like a fixed percentage of tokens masked out. So they


usually say it's like 10 to 20% of tokens masked out. However, in order to


really get these models to produce samples, they also had had to have seen


cases where, like this case where zero percent, sorry, not zero, 100% of the


tokens are masked, right. So that way you train this is you mask tokens, like


BERT, and then you predict all of them at once. So the model would have to have


seen every single proportion of masked tokens. So that's not what exactly what


what BERT is trained for. But in essence, you could do it. So what's the


background? The background is essentially that these models, what they usually do


is they say, look, the whole sample has a given probability, I can decompose that


probability due to the multiplicative rule into products or in the log space,


sums of probabilities. And this here, this part here is what the order


aggressive models take. They say, look, if I have a bunch of nodes, then the


probability of, for example, this node is conditioned on everything that's before


so I can factorize this into products, where every probability is conditioned


on the ones before. And these models, they essentially go and they say, well,


there's no reason, no particular reason why you have to factorize in this way,


you can in fact factorize in any order that you want. And if you do that, if you


recognize that you can factorize in any order you want, you can also say that you


can also say that the you can essentially not only train in the order


that you decode in, you can already train for all the orders at once, right?


So if if my chosen order is I go from here, to here, to here, to here, right?


Once I'm at the purple node, right, in this particular order, I would go here


next. But in many other orders, right, where I came from, from here, in other


order, I would go here next. And in yet another order, I could choose, I would go


here next. And these orders I sample uniformly, okay, so I can reasonably


assume that the next time I see the sample, I'm in one of those other


orderings. And therefore, the expectation of my loss function is just the average


if I were to predict this one, or this one, or this one at this time. And


therefore, if why do I have to wait for the next samples, I can simply say right


now, well, I'm simply going to predict all of them at the same time, and then


take the mean as my loss function. So the mean classification error as my loss


function, rather than just predict the one in the order where I happen to be.


Left to right models don't need to do that, because they are always left to


right. So the next time they see the sample, they will have to only decode the


exact same next variable. However, these models, we train them to work in arbitrary


orders. And therefore, we might as well predict all of the orders at once, and


take the mean of the loss function as a loss function. And there again, you see


the trade off. This allows us then to decode in any order we want. However,


also, there's a trade off now only one over the number of remaining nodes is


the portion of the loss function that is really trained on the order that we're


eventually going to have. And all the others are essentially superfluous. Well,


they might help for generalization a bit, but, you know, the you significantly


reduce loss mass on the order that you actually then care about at the end when


you sample. So here's how you sample. It's pretty simple. That's what I said.


So you initialize x empty, example, one order. As I said, you don't have to


commit to one at the beginning, but that's how you specify you sample an


order uniformly. Then you go through the through the ordering through the


permutation here, Sigma is the permutation of nodes decode. This is very


complicated written. So the they build these masks right here, you can see they


built these masks. And essentially, M is just whatever has been decoded so far,


n is whatever is whatever one node is to be predicted right now. So what you do


is you build a categorical distribution. You put the masked x into your neural


network, build a categorical distribution. So this here means you predict all of


the nodes at once, given what you've predicted so far. So m times x is what


you've predicted so far, that goes into a neural network, that's essentially the


learned part of this. And the neural network will output a distribution, a


categorical distribution for every single other node there is. And what you


do then is you choose the one the n, you know, that's the entry in the ordering


that you chose, you choose the one that you want to decode. And you simply


augment, amend the sample that you have by the one you want to decode. This is


written very complicated in a very complicated way. So optimizing training


these models isn't too hard either. What you're going to do is you have a data


point that I guess you sample from the data set, you're going to sample one


particular time step. So notice here, we go over all the time steps, because we


actually want to get a sample. When we train that's much like transformer


autoregressive models, actually there we can train all the time steps at once.


But the individual training sample is just we select one particular time step


in one particular ordering, right? So we select an ordering and in that ordering,


we select the time step. And typically what you do is so you have a picture, you


have pixels, what this amounts to is we say, okay, we're just going to mask a


bunch of these pixels right here, we're just going to black them out, right? That


will correspond to some time step in some ordering. So we're just going to


assume we've predicted all of the ones that we haven't masked. And now we're


trying to predict all of the ones that we did mask, right? All of these ones


we're going to predict at once. And yeah, that will. So you notice that there is


no n right here. The n specifies the one pixel you want to predict next. But


during training, we simply mask out a bunch of pixels. And then we predict all


at once. So again, we have the M, which is what we've predicted so far, we input


m times x into the neural network. So the neural network will predict the


distribution of every single thing that we haven't predicted so far. And rather


than selecting n from it, we now select one minus m. So everything that hasn't


been predicted so far, and then we average that and that will become our


loss function. Okay, now, given that we know what the pixels are that we've


masked during training, we can actually compute this loss function. And, you


know, that's, that's it. That's how you train. Pretty simple. As I said, this


should remind you of BERT. And yeah, so they have several extensions to this,


which I just briefly want to touch. So they now they say, well, if we if we


sort of allow a bunch of times these dependence, independency mistakes, so you


know, given that we have like, I don't know, a million pixels in an image,


right? Can't we just sort of assume that you know, the pixel up here, and maybe


the pixel here, they're kind of independent from each other. So couldn't


we just sort of sample, sample them at once. So we can sample multiple pixels


at once. If they're kind of far away from each other, we're just kind of fine


with that. And yeah, so we trade off speed, predicting multiple pixels at a


time by we trade off speed and accuracy essentially, because now the pixels that


we predict at the same time, they have no knowledge of the other pixels in the


same time step. That's the problem we've talked about before. And then they go a


step further. And they say, well, rather than deciding, you know, we want to


decode five pixels at a time instead of just one, what we're going to do is we're


going to give the algorithm a budget. And they say, look, you have an entire


image, we have 20 steps. So you need to decide. This is the visualization right


here, if 20 steps you need to decide, do I want to go like, do I want to go. So


here is like one pixel, then two pixels, then three pixels, then five pixels, then


the rest of the pixels, right? These are five time steps. That's your budget, you


decide. So they use a dynamic programming algorithm. Essentially, they build up,


they go through their as far as I understand it, they go through their


training data set. And they compute what they call loss components. So here is


your your budget. And here is the number of nodes in the in the here is the


here is the number of nodes in your data points. And so you can say, okay, for step


number three, if I were to decode five steps in step number three, right, how


much would that cost? And then you can try to find in classic dynamic programming


fashion, a path through this matrix. And, you know, at the end, this path is going


to give you what how many pixels you should decode at what step. So for


example, here in step one, we decode two, then we decode one. I don't know what


this actually means one, no zero. That makes no sense. And then we decode the


rest. But you know how dynamic programming works. And this isn't this is


from a different paper, actually. But they just say, you know, we can use this


given that we train for any order at all, and predict all at the same time. This is


an option. So you can technically trade this off. What they also do is this depth


upscaling. And what they do in the depth upscaling is they say, well, you know, if


we're trying to predict a pixel value for a pixel, right, the pixel value is


like 256 classes. It's a big thing, right? Let's not have the model. So the


model needs to sort of commit to one of them. You know, in immediately like


that's my pixel value. What if what if we could do the following? What if we


could have the model just predict which half of the pixel values it's in right?


Are you bright in the blue channel? Or are you not bright? Are you dark? Okay.


And then we do this for all the pixels. So all the pixels in the image, they


simply first in the first iteration decide, am I light? Or am I dark? Right?


Am I light? Am I dark? Am I light? Am I dark? And so on. And then once


everyone has decided on that we go over the image again. And we say, well, okay,


now, okay, I should have filled all of them. Just imagine all of them filled


in. Now they say, okay, now you pixel who previously decided you were light.


Now that you see all the other pixel and their crude decision, you know, what


sub part of the light do you fall in? Are you very light or just a bit light?


And then so we go through the image multiple times, right? It can even be in


different orders. And the advantage here is that you first let the other parts


make crude decisions, and then you don't have to decide out of the blue, right?


So you know, sort of approximately what all the others are before you refine,


and then you refine, refine, refine, until you get to the final choice. So


this is I think this is a neat idea. They specify exactly you know how to do


this. However, I can't help noticing that as you can see the ordering here by


which you decode, so you first predict the the crude part, then the not so crude


part, then the not so not so crude part. And finally, you predict the full part,


I can't help but notice that this is again, a fixed order, alter aggressive


model, right? This is, this is again, like this is exactly what they're trying to


run away from. So they they just introduce it again, in a sub part of


their model, which I find to be funny, right? And on the on the other hand,


this this only works really, this my other problem with this, this only works


if this isn't really a categorical variable, right? Pixel value, pixel value


pixel value is a continuous variable, you can be anywhere, we just discretize it,


right? And that's why this works the, you know, decide on your crude and then go,


go more, less and less crude, go more and more detailed. If you have something like


true classification, right? Let's say into tokens of a vocabulary like A, B, C, D, E,


it makes no sense to ask them, well, in which half of the alphabet are you? The


model can't do a crude decision, it already needs to know to answer this


question for you. So unless you have a way to really split the vocabulary in


meaningful fashion, it this doesn't make sense. This is really, this is really a,


a workaround around the artifact that they need categorical variables for


their model. And therefore, they discretize the, the, the brightness here


of the pixels. And you know, that's a result of that. So in any case, I don't


want to dive too much into the results, you've already seen them, they don't,


don't do large scale. As far as I can tell, they do c410 generation, they also


do lossless compression. What they can do is with their model, they have a


pretty good handle at the trade off. So this gives you the applet, so the user


of the model, a good way of trading off performance for speed. And you can do


this on the fly, right? You can do, you can say, I want less performance, I want


more performance, I have less of a budget to infer the sample or more. And


you can change from from time to time. And yeah, these, these models, as I said,


they're young, therefore, they have a way to go. We've put so much work into GANs


and whatnot, and, and auto aggressive text models, that the fail, like the fact


that these here are not state of the art yet, they might, it might just be an


artifact of that, or they might just suck. Who knows? All right, thank you so


much for listening. As I said, join our Discord to get in on the paper


discussions. They're usually very, very entertaining. And I'll see you next


LAION-5B: 5 billion image-text-pairs dataset (with the authors)

Hi, this is an interview with people from Lyon whose flagship projects are data sets,


specifically data sets to train models like Dali or clip. So pictures and text that goes along with


the pictures, they scrape these from big internet scrapes. The first data set had 400 million images


and their newest data set has 5 billion images. These are unprecedented scales to be open sourced


as data sets. The creators of Dali or clip open AI, they never disclose their data set, they never


put it out there in public and Lyon does. So this is a big service to the community. And I was super


excited to have them on here. Another thing is just how grassroots this movement is the founder


Christoph is also here today is a father and a teacher and does this on the side just as a hobby


and sort of wants to demonstrate a little bit how anyone can take part in open source research. Now


multiple times during the interview, his kids would actually come in and be like, Daddy, play


with us and so on. YouTube is very strict on this. I cannot show the kids even though the kids


themselves would have loved to appear in this YouTube video. So you know, kids, please, I'm


very sorry. It's a shame. When you're 18, open invitation, open invitation for the channel.


I thought this was really cool and inspiring. In addition to learning what Lyon is about. Enjoy


the interview. Let's dive right in. Hey, everyone, today I have the team behind Lyon 5b with me,


Christoph Schumann, Roma, Beaumont and Kate Gordon are here who contributed to this project in


various ways, which I hope they'll just tell us about in a second. This is a giant data set. It's


over 5 billion image text pairs. So not just images, but image text pairs. And along with that,


an open clip model, open sourced clip model that matches the performance of open AI's clip model,


which is really cool. These, these big companies, they rarely give out their biggest models,


if any, if you know, if at all, and if they give out their biggest models, they usually don't give


the data set behind the model. So it's really cool that we have like a large data set. There has been


some controversy around your smaller data set that you released, I want to say half a year or a year


ago. I hope we can get into all of that today. But first of all, thank you very much for being here.


Welcome to the channel. Welcome. Nice to be here. Yeah. Just maybe tell me a little bit. What is


Lyon? And what is Lyon 5b? So it all started like 10 months ago, I guess, on the eluth.ai server,


when we talked about how could we eventually replicate Dali? And where could we get like


200 300 400 million image text pairs. And there was this idea of going to common crawl,


and looking for all the image links and only take those that have an alternative text. And we have


been talking about this in the multimodal channel there together with Aran and Ben Bang. And they


got a little bit distracted with a project of GPTJ. So they ended up focusing totally on GPTJ. And


I was sitting there and was a little bit upset and thought, hmm, why don't they pursue this? Because


because I compared to them felt like someone who is not that a good programmer. And then I thought,


okay, screw it, I'll just do it myself. And I sat down and wrote everything down in one call up and


began crawling from common crawl and filtering with clip. And then more and more people joined


me at first two comms. He, he was the first to join me. And so we call it crawling at home. Because


at first, we had some call up notebooks and some GPU somewhere on from some people on the discord


servers. And they were all like downloading and downloading, filtering, and uploading the results


to a rented server. And yeah, and after a while, more and more people joined like Richard, who is


not here at the moment, but he's also a very valuable, cool contributor, Richard Benku.


And we optimized the code so that we could filter and crawl with one 3090 in one day,


30 million image text pairs after the filtering, not before. So in the end, we ended up like at


the peak was like 30 and 60 or 100 small mini servers downloading the images, sending them to


Richard's, Richard's GPU in his bedroom, filtering everything and spitting out in the quality of like


conceptual captions 12 million was the biggest then at the time, and 12 million image text pairs


of decent quality. And we could generate with one 3090 within one day, 30 million. And at this point,


we said, Oh, wow, we should really scale this up. And I asked someone like we already had some


people on this court who gave us the CPUs GPUs. And so it grew and grew. But then it was clear


that we could get with only the nations we got from the community could get to 400 million,


what would be like the scale of open AI clip data set, because clip was trained initially on


100 million image text pairs. And I said, Okay, we can get to 1 billion if we would get like maybe


$5,000 of donations for paying for small CPU servers and maybe some GPUs somewhere. I don't


know. And I asked on the Luther AI server, and within like 10 minutes, someone said, Oh, if it's


only 5000, I will pay it up front. Someone has like a startup. It's Jack from doodlebot AI.


And yeah, he ended up giving us in the end, like $10,000. So he was our first official sponsor.


And I have to say the AI.eu also provided us with some compute, but the first sponsor who gave us


money. And then I said, Okay, I don't want to have this money on my bank account. And we probably


for now and for the future should start a nonprofit. And then came Jenya who is not here at


the moment, Jenya Jitsev, he's a lab leader of the deep learning laboratory at the Jülich


supercomputing facility. And yeah, we had been in touch and he said, Okay, we will join with our


two people because we want to train models like Dali or Clip on the Jülich supercomputer, like


Juvls. It's a giant machine with almost 4000 A100s. And he cannot directly access it and train


it Dali, but he can access it for proof of concept small projects and then apply. And so we said,


Okay, let's start a nonprofit. And we take this as a shell for basically getting money, getting


resources officially, and then spending it for creating cool data sets and training models and


giving them away for free, no fees 100% open. Because we were, I mean, we were a little bit


disappointed by the promise that OpenAI made by the name of OpenAI. And many people had been


joking about closed AI. And I totally understand that if you get $2 billion of funding that you


have some strings attached and that you have some protocols and problems and that they have security,


safety concerns. But we said, Okay, we don't have the means to do all the basic research, but we can


try to do what they were doing, what Microsoft is doing, what Google terrain is doing and just


taking the code or replicating the code and releasing such models for free. And then we


started a German nonprofit, a Verein, Gemeindeziger Verein in Germany. And yeah, ever since everything


took off, we released the 400 million data set. And less than one hour later, I got mail from


Thomas Wolf from Hugging Face and I got also in contact with many more people and everyone


wanted to talk to us. And now we also get some monetary support from Hugging Face that also


enabled us to do the big data set. And we have Stability AI who is providing us with GPUs and


will provide us in the future with more GPUs. We have an ongoing application for 600,000 GPU hours


on jewels. We don't have like the result yet, but in one month we should know for training a big


clip model and applying this to some downstream tasks. So yeah, everything is moving very fast.


And one year ago I was just like a family daddy and a computer science teacher. So I'm a computer


science teacher. And everything developed very quickly. And now Romain, who is also like an


awesome guy with much of experience and the cool tools like image to text, image to data set tool


that you already introduced in your ML news, I remember. And Kate, who is a really brilliant


computer science student who is really into clip and he helped us to train a clip and replicate


the results of the Vision Transformer 32 base. And we matched roughly with a small variation,


sometimes a little bit better, sometimes a little bit worse on several data sets,


the performance of the original clip. So yeah, everything's looking really nicely.


We have no intentions of going for profit. We agree that we want to stay open.


We agreed that we want to stay non-profit for several reasons. And everyone who likes to


contribute or to talk to us, maybe someone has some questions, maybe someone is curious about


something, everyone can join our Discord server and just ping us. Cool. So I want to dive into


sort of the biggest criticism that I would have with this project in that your data set essentially


crawls common crawl for image text pairs. And I'm going to guess that's image and the associated alt


text or whatever text you find with the image. And then you have this filtering step is what you say


you can do a lot of images on a single GPU. But you're essentially using OpenAI's clip model to


filter image text pairs, which clip deems to be, you know, fit together well. Right. So I isn't,


does that like how much of a how much of a bias does that introduce into a data, especially now,


if you say, well, we train a clip model on this data set, right? And we are able to match the


performance of OpenAI's clip model. One could ask, you know, is this, are you essentially


replicating the result? Or are you simply matching their performance, because the data set is already


essentially filtered to, you know, the data points that are conducive to that model. So could you


dive a little bit into your choices there? And how much do you feel that is an important step


this filtering? What does it like? What's the, what does it give to the data set? To use that,


and you have plans to maybe switch that up or improve that part? So no one claimed that this


would be perfect. But before I did this, I started with JFCC 100. And I filtered this also, I filtered


basically on Colab and yeah, whatever. And I checked a lot of image text pairs manually.


And I just got the feeling after looking at thousands of images and text pairs, that


at point 28 was a pretty good threshold. Like that, if you go above the threshold with the clip


B32 from OpenAI, then it really seems to match pretty well. It's still a little bit noisy, but it's


rule of thumb. And if you go above point three, it's even a little bit better. Not perfect,


but a little bit better. And this is what we have. This is not the ultimate solution for everything.


But I think because we are going so big and crawling over so many images that are made by


humans, the annotations are made by humans, that in the end, we will still get a lot of new information


in. And it could be that some people, maybe some names of people that the original clip has not


learned or some concepts, some nouns or some adjectives that has not learned could go below.


This could always happen. But yeah, I mean, from the standard benchmarks that we ran, the results


are pretty good. And everything is work in progress. Yeah, I don't doubt the quality


aspect of filtering with OpenAI's clip. What I'm a bit worried about is that you're essentially


replicating how this model sees the world, right? This model isn't perfect either. And so it will


sort of replicate its own vision of the world into your data set. And especially if you then train


a clip model, right? That would that would be replicate. Have you tried just training a clip


model on let's say an unfiltered data set? Or what could also be possible if you have many different


such models that somehow estimate quality of images and text that you could build some sort


of an ensemble? I don't know if you have plans in the future to to replace this filtering step or


make it better? Is that something you have on your radar? I guess one thing we do have is the


unfiltered pairs. We have actually 10 times this, like we have 50 billion unfiltered pairs. And


yeah, there could be some work that could be done on analyzing these pairs and trying to see if it's


different. But the problem of just using them is then you lower the quality a lot. So I don't know


if we do what it would do. But yeah, it's definitely an interesting point. We don't fully


have the answer on that. I think this is one of the points that will become more apparent when we


start to train the larger clip models. So at this moment, it was like line 400m. So that was the


initial data set that we had just that subset. And getting in the range of OpenAI is at least


sufficient enough to prove that we've at the bare minimum been able to replicate the exact


inferences of the model and get into that convex hole, so to speak, of its confidence threshold.


I think the more interesting result will come into play as soon as we hit the 5 billion scale


and we get up to that larger threshold. If we're able to push the numbers that OpenAI got before,


it could also be in response to the fact that we have maybe different image towers and text towers


to share that. But if we can outperform what OpenAI did within their original models,


it could be a sign that the data set was able to get like just enough stochasticity to go outside


of like perfect confidence. Again, it's in the future and it's not a result that we have, but


we're optimistic and seeing what it lies. Did you like how big is your data set? Just give me some


some numbers in terms of like gigabytes. Like what can I expect if I work with this thing?


240 terabytes. 240 terabytes? Yeah, if you download it in 384 resolution.


And you have different... So you collected different images. Can you give me some numbers


on that? Like what kind of resolutions do you have? How long are the descriptions usually? Just


kind of some so people can imagine a little bit what this looks like.


I think if you could open the blog post. The blog post, yeah.


Yeah, so like for example, the English part is 2 billion samples. And then if you count only the


ones that are bigger, both in width and height, then 256, it's like a billion. And then half of


half is resolution. And yeah, so it's a lot of images which have a decent resolution. But if


you want to train like a, like let's say a highly quality, high quality generative model or maybe


segmentation model, maybe you want to use a high resolution subset. Yeah, in terms of


caption length, yeah, I want to add the precise number in that section. But yeah, it's around like,


I think it's around 200 characters. But yeah, that's a good question. I should add that.


I computed it at some point, but I think I didn't add it in the blog post. Yeah.


And yeah, you have this language distribution as well.


Which is interesting for the MBG language, that's it.


Do you, oh, I saw it just a second ago.


Yeah. It's at the same time.


There you go.


Yeah, there. So it's a long tail actually, because like we have like 100 languages. And yeah,


the first one we have a lot of samples, but then yeah, you have this long tail of many other


languages that are available. But yeah, for example, you have 70, you have a 7% of the


multilingual data set, which is French. That's interesting.


Do you always have one piece of text with an image? Or do you sometimes have multiple? Because


a lot of these data sets that are captioning data sets and so on, they provide kind of multiple


labels for one image. It's just one image, one piece of text. Okay.


And that is it always the alt text of the image? Or do you sometimes like grab text around?


This is like work for the future. So in the future, we want to build an audio text data


set with a similar approach. So currently, we have some people working on training a small


or mid sized audio clip model on existing data sets. And once we have one of sufficient quality,


we could go through all of common crawl, filter out all links to audio files, and try to somehow


get something like the alt text. Because usually there is no alt text, but we could like look if


there immediately before the link or after the link is some text that has sufficient audio clip


similarity. And there are many ideas, but if anyone would like to join us and work on this,


everyone can join. We are truly open. Just get onto the Discord server and say here.


So yeah, go ahead. Yeah. And two things that you had been talking about previously.


So what could we do to make clip recognize more things that had not been in the original


clip data set. And one interesting perspective for this that is still work in progress, but


that could maybe work is we are experimenting currently with a training clip with a frozen


image encoder. And one idea that we have is to train a masked image of encoder, something like


SimMIM or the MAE from Facebook meta. And then we could train it on many, many images without texts.


And so the basic idea is that if you have a really good image encoder that can be trained


in a self supervised manner without any text, there, then the limit is the sky. Because like


in theory, we could get like 50 or 100 billion images from common crawl. We do not pursue this


at the moment because like 5 billion is enough for the next few years, I guess. But we could


train a really good image encoder in a self supervised fashion. And then we freeze it and


we can train it with text, train the text encoder. And I guess in this case, we would have


much knowledge from the self supervised training about what is actually an image. And we wouldn't


need the clip filter data. We could take any data set. And then we could train it on the image


with the clip filter data. We could take any data set. And this could help with this. So we're exploring.


We are cooperating at the moment with a CLOAP team with Andreas Furst, who is the first author of the


CLOAP paper. Like this is an improvement of the original clip architecture with some Hopfield


layer magic. So let's see what happens. So tell me a bit about what it takes to, because these


are unprecedented scales for most people. By the way, there's a nice overview here over the


entire acquisition pipeline, which is really nice distributed and all. And then you train this clip


model. Now the clip model you have currently, you already said it is on the 400M data set,


which is the let's call it the old, it's not super old, but it's your previous data set,


which is on the scale of clip and you trained a clip model on this. What does it take to work at


let's call it at that scale, right? ImageNet is 1 million images. And that's already considered


like a rather large data set for most researchers that have like a GPU or something like this,


right? 400 million is almost, I would say most people probably aren't working with this


size of data. Is it easy? Is it hard? Like how do you go about training this model?


So there's like two large contexts for this. This is whether or not you're in like your large


HPC cluster or if you're in more so just like your generic data farm. So at least these results were


supported by Jules Booster and the foundation, which upholds that. There it's also a very large


institutional barrier of even like getting to the batch size that they offered. So in terms of data


set alone, you have to have everything like stored on disk and that is a nightmare in itself getting


it collected. And that just in terms of memory is probably not accessible to most researchers.


Then you get an extra layer, which is the exact batch size of clip. There have been other papers


that have shown that these large multimodal contrastive models are like extremely batch


size dependent. Basic has a really good table on this and it's hard enough to get to your data set


alone, hard enough to get the infrastructure just to support that. But on top of that, can you get


your massive A100 cluster to actually spin this up? And one thing they don't talk about is the


massive engineering struggle that goes into actually doing contrastive loss on this. Let


alone if you just take a 32,000 by 32,000 matrix, it's like two gigabytes in FP16 or four gigabytes


if you're doing full precision. And that just becomes a nightmare of overhead. And so the


wonderful team that I've been working with, this model is just as much mine as it is theirs.


We've been putting a lot of our time into just how to optimize the small things. Like for instance,


when doing contrastive learning, you don't actually need entire global batches. You can do


only certain calculations that are necessary for your local gradient routine, so on and so forth.


But to achieve this scale, there are a lot of challenges that these large research labs don't


like talking about because they're not as pretty to write on the paper. But this isn't very


accessible immediately for like everyday researchers. And we think this is something


very important for other people to get their hands on. And so hopefully this will inspire


more companies to give out the compute necessary to accomplish results like these and inspire


further researchers to uptake in this direction. You also mentioned that your original plan was


to train something like Dully, right? And CLIP is an important component of Dully. Is this still on


your radar to eventually train something like Dully? Because there are other projects going on.


I know there's like mini Dully and other people trying to replicate Dully. Like what's your


thoughts on replicating Dully? Yeah, there's so much going on and it's incredible. So there had


been from Lucid Rains the PyTorch Dully project and we actually tried this on Jumwit's booster.


So we got this to run on, I don't know, maybe 256 A100s for 10 minutes. And it would work in theory,


but the thing is... My son is here. One second. Daniel, I need...


He has rubber balls. Okay, Daniel, I need time. Okay. Kids are important. So this is really


awesome about all this. You know what I'm doing, like on the Discord servers. I'm doing this when


I'm on the playground. I'm doing this while I'm playing Minecraft with my kids. I'm doing this


when I'm at the shopping center, like from my mobile. So I can do this in my free time. And


this is really amazing. But what was I talking about? Dully. Dully. Yeah. So the thing is with


Dully, we could have pursued this and we had to make the decisions. And first we wanted to apply


for a compute on Jouvel's last August for like a half a million GP was for creating Dully. But we


missed the deadline because we were so busy with line 400. And then I had a realization.


Others are working on Dully. Dully Mini is there and MinDully and you have like RU Dully and now


the Fusion models. And I said, hey, Clip is actually not that amazing on the first glance,


but on the second glance, it's far more amazing because you can use it to guide generative models.


You can use it to make huge datasets. You can use it to create them semantically meaningful embeddings.


And this alone is very interesting because like I had this idea and Luther people had also this


idea that maybe one could like take images and texts and do sequence modeling on the clip embeddings.


So you wouldn't do the sequence modeling on the image tokens or on the text tokens, but maybe on


the abstract ideas. So I compare it like it's not 100% accurate maybe, but it's like a metaphor.


So if I'm thinking about I want to go to the fridge and get some food and want to do this,


I'm not really imagining everything in full HD resolution and I'm not thinking, oh, I will go to


the fridge. So I'm more like having the idea in a kind of mixed embedding space, idea space.


And so one thing that we have in mind is like something in the future, maybe not now,


but if it would eventually work to take embeddings from audio, from video, from text,


from all modalities and bring them into the same embedding space and then somehow


bring a transformer to model them. This would be really interesting because you could like


train it on a text, on video, on everything and could do it in a very, very efficient way.


And Eleuther people had been working on this. They got many not number errors from feeding in


the direct clip embeddings because it's probably just like too big, too unstable with all the


noise in the clip embeddings. But I have the hunch that clip is really powerful.


And I didn't realize this when I first read about it, but I think so the idea you have GPT kind of


models, they have sequence learners. They can model sequences of whatever, of images, of texts,


of all kinds of data. And you have something like clip that can take different modalities,


basically any modality and convert it somehow into a shared embedding space. And I think these


topics are a little bit disconnected at the moment, but in the future, there's very much


room left to the ceiling to combine them, maybe do something like front-of-the-station of the


clip embeddings or whatever. I have no clue exactly, but I could really imagine that in the


future, if we could get all modalities into a semantic shared embedding space,


and find a sequence learner to model this, I have no idea. Maybe I don't dare to dream of AGI


or so in this connection, but I can really see similarities that in my stream of consciousness


when I think, okay, I want to go there, then happens this, and I do action X and action Y.


This is not so different. Yeah. Well, there is a debate of whether you need to actually interact


with the world to achieve AGI, right? I think that's the big hurdle. The other thing is there's


this model or this paper called CM3. I don't know if you've seen that. They are doing something very


similar to what you just suggested with actually quantizing the images out of the image.


They're using an autoregressive model in order to model that. Maybe that might be some ideas.


Maybe I can say a few words about your initial or your previous question about the size of things


and how do we handle it. I think maybe I have a slightly different perspective because for me,


what was interesting in this project was that it was a very interesting project.


I think I have a slightly different perspective because for me, what was interesting in this


project is to be able to do all of this with actually little resources because, yeah, it's


pretty big, but for example, the 400 million dataset, just with some Python code pretty


optimized, you can actually download it with only one machine and three days, which I think,


and at this scale, you only have like 10 terabytes of data, so you can actually store it at home and


it's not that expensive. I think that's pretty interesting because I think that was one of the


things that made it possible for many researchers to get ION400M and start applying to various


ideas. We had a bunch of papers trying to... They took it and trained some generative models,


trained some contrastive models, that kind of things. The story is a bit similar,


but of course, a bit more costly with these new datasets. I had to make everything distributed,


so now it's like 10 nodes and not one to download it in a reasonable time, but still,


it's in the domain of reasonable, you can have it without being a very large company.


Yeah, and following up a bit on this idea is... One of the things we did as post-processing of


these datasets is like downloading everything and computing all the key problems out of that


and then putting that in a kind of index, and that's the UI, the demo. I think one of the ideas


is that you can export the datasets, you can look for cats or whatever you want,


but you can also use that kind of index to extract new sub-datasets that are much smaller


and that can be interesting to train, let's say, smaller things and solve more specific problems.


Maybe you want to find all the pizzas from the world and get inspiration for a restaurant.


Or you can, for example, try to build some kind of subset out of Lion400M or Lion4Bay. For example,


Christophe has been starting a project to find all the humans in the dataset and see what's there,


what can we understand from that. And I think what's interesting is that all of this democratized


research, like it becomes possible to actually build that kind of stuff without having too


much resources. And I hope that it makes it possible and that people play with all the tools


under the datasets. I see you're storing the dataset on S3, which does, I know like Eleuther


stores their dataset on the i, which supplies these resources. I know S3 has significant


charges for egress, right? If people download this, that you incur quite some cost. I think


they have like 20 cents per gigabyte, which would be like 200 bucks per terabyte. So at 200 terabytes,


someone downloading the dataset would cause you something like $30,000, $40,000 or so.


This is what your sponsors are there for? Or do you have like a deal with Amazon?


We are very lucky. So we are very lucky. Our sponsor for computer at the moment, our main sponsor for


the GPUs and for the S3 storage is Stability AI. And their plan is actually to gather resources


from different companies, investors who actually want cool multimodal models openly available


because they want to use them, but they don't want to build an ML team or hire people or so.


And he has many connections, Emad, he's the CEO or the founder of Stability AI.


And he has a very good deal with AWS. And we won't share the AWS files that we have because


we don't own the copyright of the pictures, but we are sharing the metadata, the URLs. And so everyone


on his own, his or her own liability and risk could download them from the original sources.


We recommend that if you do this, you make sure that the dataset is shuffled nicely.


It's already shuffled, I guess. And so when we started the project, we got problems because we


didn't properly shuffle them. And sometimes some webmasters complained that we were downloading


too much from them and the dataset where we were renting the machines got some complaints.


But if you shuffle it properly and you download it over all the 5 billion image taxpayers,


there is no problem usually. And with the wonderful tool image to dataset that


Romain programmed and that now also supports distributed downloading with a swarm of


CPU workers, one could download it for relatively small money. I mean, Romain,


can you tell more about this? Yeah, for sure. That's a big thing. I think that makes it


possible for us to share the datasets. Like Lion400M is 10 terabytes in images,


but the metadata is only 50 gigabytes, which is quite handleable. And same for Lion5P,


the image is 240 terabytes, but the metadata itself is about one terabyte, which is handleable.


And then, yeah, you can use that image to dataset tool to get the data, which works.


Well, of course, there will be some link hearts and you will start losing a bit of data with time,


but it's a pretty reasonable given the total amount of data. And about the cost, yeah, to do


another Lion5P, if you use some other AWS instance, I think the cost should be like a thousand dollars,


which is not nothing, but it's not like a 40k you are mentioning about, I guess.


Yeah. Okay. So it won't bankrupt you and it won't bankrupt me if I download this data.


Yeah, exactly. Yeah. I see. And for the future, there's a new direction that we're exploring at


the moment or the Hivemind project is exploring. So they are working on some code that would allow


you to directly stream the images from the URLs. So you download them, you buffer them somewhere,


and if you have a decent internet connection, this should actually work. So last time, Alex B


from the Hivemind project, he's also on our Discord, he told me that they could reliably


train like 50 to 60 images per second. And for a small model, this would not be sufficient. So we


would get a bottleneck. But if you go to something like a vision transformer, capital G or capital H,


the training takes so much time that it wouldn't matter. So you could like train a capital H vision


transformer with this, and you would need only maybe 100 gigabytes or so of storage in your machine.


That is interesting that the models that get so big that essentially the bottleneck shifts


away from the internet connection to the cluster forward propagation. That's pretty cool. But you


mentioned a good point in terms of releasing these kinds of data sets and the not technical


challenges, but let's call legal challenges, social challenges, and so on. You already mentioned,


there's obviously issues with copyright. So any image that you have, if you want to reproduce it,


you technically need to have some sort of a, a license to it, or you'll be a criminal in some


country in the world for sure. So you only have the links, you solve that part pretty well.


But there has been there's been criticism, I think, with respect already to your earlier


data set is specifically I remember about two weeks after it was released like insanely fast,


there was a paper, like criticizing, it was it was framed in a weird way, like it was half


criticizing your data set and half criticizing the large companies for not releasing their


their tools to filter these data sets. And could you maybe summarize a little bit what that


criticism was of your data set? And what what was the issue?


So basically, the issue was that the authors said, if I remember correctly, that our data set is not


properly filtered. And that if you go to our web demo or to the raw data, you could find stuff like


sexual content or hateful content or really disturbing content in it because the content


is not manually filtered by humans. And that training on this data could eventually lead


big models to behave in a toxic way or maybe in a biased way. And I don't think they criticized


only us for this problem. But they said that we were at the moment not careful enough about these


topics. And I guess that's one reason why these big apart from competitive advantage, right? A


reason why the large companies might not release a data set like this, because inevitably, there's


even like there is legit adult content in ImageNet, right? So that's one reason why we're


seeing this. And I guess, these larger companies, they might not release the data set also because


yeah, copyright issues, because of these types of things. I also remember, they specifically


refer to the fact that a lot of adult websites, they use this alt text to do search engine


optimization. And they use this alt text to do search engine optimization. And they use this


alt text to do search engine optimization. So what they would put in the alt text would be


just terms that a lot of people search for if they search if they frequent these websites. And


that would make it such that a seemingly on like, either a seemingly unsuspecting image would go


together with offensive terms or seemingly on offensive terms would would be like, associated


associated overly with adult themed images. You know, they had some, some examples right there.


Sorry, but I interrupted you.


So to put everything in a appropriate light, I want to make some things very, very clear.


First, we do not recommend anyone to train models with the raw lion data sets,


and put this into production without really careful either filtering or end thinking about


how to make them safer. So this is just a research data set that could also be used by companies for


research purposes, or maybe for pre training and later making really, really thoughtfully sure


that it's safe. This is first. The second from the initial version, I already had some filters in


that tried to generate tags for non-certified work and to filter out obviously illegal content


through clip scores. And this time we improved the non-certified work model to become really good.


We have now a clip embedding based classifier, where you can run inference over thousands of


images within a second, if you have the embeddings. And it has on a test set. So I made in November,


a manual test set for non-certified work. And the test set has around 3000 images. And it gets an


accuracy of 96, above 96%. So it's already pretty good. And it's really fast. And thirdly, we are


also cooperating with TU Darmstadt with Christian Kerstling and Patrick Schwadovsky, I hope I


pronounce his name right, to use their existing offensiveness classifier because they have an


offensive content classifier based also on the embeddings of clip that also detects things like


violence, hate speech, things like bad animals. And it is really conservative. So it tends to also


filter out like Halloween costumes. But we will soon provide also these. And I think what we are


really doing by releasing all these samples and filtering them out in the first place is we


generate a huge opportunity for safety researchers to create openly available, non-certified work


classifier data sets. So everyone who wants to get toxic content out and non-certified work


content out is invited hereby to work on our raw data to generate subsets and train better


tools in the future to filter those things out more reliably than we can currently do.


And I remember your not safe for work classifier initially was already pretty good. So in this UI


you have right here, I think you have it maybe not here, but I remember you had a not safe for


work button, safe mode here. Obviously, I can't show this here since this is going up to YouTube.


But I tried to reproduce some of the results in that paper. And for the kind of egregious results,


you really had to actually untick that that box and select the correct sub model right here,


because you have you have different sizes and also different models of clip that you


had. Now that is probably gone now. But I remember I could select a different smaller clip model and


the really egregious results I had to untick the safe mode box. I had to select the smaller clip


models, which would probably be less nuanced, and more more prone to these kinds of things. And then


I could reproduce it. So yeah, I'm certainly I'm certainly in favor of people, you know,


looking and saying, you know, look, all the text is often used for search engine optimization,


and that, you know, can play into that can can kind of poison the data set. Yeah, but I also


feel there's a big opportunity to use this in a constructive way. Although, if you if you like,


the implication is because you filter with clip initially, and you still get these images in your


data set. That means clip itself must have been trained on a lot of data like this, right? Like,


it also means that open AI hasn't managed to to filter out these types of images, right by


implication, which is pretty interesting to think about. There's something related to that which is


interesting is so to train the safety model, Christophe mentioned the training set. But for


the model, we tried several things. And the first thing that Christophe tried was just training


hand to hand efficient net model. And it worked pretty well. But then the issue with that kind


of model is when you need to spend a lot of GPU resources to do the inference. So then we also


tried to use a model a small model based on clip embeddings, which is then much faster, like you can


run the whole inference of a lion 5p in one day with just CPUs. And what's interesting is that it


works almost as well as the efficient net model, which means that indeed clip has that knowledge


like it can tell if you add a few layers of dance, a few dance layer on top, it can tell you whether


it's unsafe or not, which actually is a good feature, like you want it to be able to tell you


that. So yeah, that's a and yeah, in that way, if you uncheck or check safe mode, it will enable or


not this inference over the clip embeddings and live filter out what the model considers as unsafe.


And there is a big opportunity in actually having clip models that are trained on toxic data,


because it helps latitude detect this and maybe even to generate synthetic data sets to combat


this. So I have been in contact with Jonas Andrudes from Aleph Alpha, the CEO of Alpha Alpha, and they


have their model Magma. Magma takes as an input a clip, the clip output of the frozen clip,


and projects this into a GPT-J and then can generate captions and do visual question answering.


And I have seen very interesting results where Jonas showed me where had been


toxic memes about racial discrimination. And then Magma was asked, why is this toxic or why is this


eventually offensive to me? And Magma generated plausible sounding explanations for this. And I


bet this was cherry picked. But nevertheless, if you would have like potentially toxic or offensive


content, you could take any VQA model, maybe that's based on clip, so you wouldn't have to train it


again. And then generate potential candidate explanations, why is this toxic or why is this


not suitable for work or things like this. And you could take these candidates, show them humans,


and let the human just click okay or not okay. And by doing this, this kind of work,


one could generate easily with far less human resources, huge safety data sets to explain


basically why something is potentially harmful or offensive or whatever. So I think to have such


kind of models for the research community, this is a really good idea. And if you have


if there maybe could be some bad actors, I am very sure that they would find other ways to fine tune


safe models that we think are safe, but maybe are not. So I think the illusion of believing


that my model is perfectly safe, just because I excluded all the harmful data from it is a little


bit naive, because there could be gaps in the filtering, or harmful actors could take them and


fine tune them easily. So this is a false safety. Instead, we should rather train the research


models with a huge disclaimer. And be aware that true safety only can come from really careful


thinking and engineering. I think this is a common way in I don't know, like psychotherapy or


something like this, that actually exposure to danger and exposure to what you're afraid of,


and so on, is the best way of handling these things. And, you know, I think as these models


get bigger, I'm more and more convinced that we should eventually apply. Of course, if I have a


linear classifier, there's not too much to do. But I think these large models, they're capable enough


that if they actually encounter such data, if they incorporate it, and so on, they're large enough,


I believe that we can teach them to discriminate internally. Oh, as you say, like, you know, this


is this is probably not a picture that I should serve at this particular, you know, for this


particular search query right here, I'm being used at a wedding to portray, you know, pictures of the


wedding pair, the bride and groom and the one where, as a child, they smear poop in their face


might not be super appropriate or so. Yeah, I think this is in my that's just my opinion. But


I think this is a good way to go. Do any of your sponsors have any kind of like, concerns or strings


that, you know, when maybe they see criticism coming your way? Was this ever an issue with any


sponsor? Or do you do you have did you have like sponsors that were like hesitant because of these


things? No, we don't have so many sponsors, we have doodlebot AI, we have Huggy face, right, thanks to


Huggy face. And we have stability AI. And I think when they read these concerns on Twitter, they


probably instantly had opinions that resonate with our cool. So where can people get started with


this? Like I'll link everything in the in the description. What do you think is the best entry


point for people if they just kind of want to check out what you're doing? Just come on our


Discord server read through all the channels that exist. We have channels for data set creation for


audio data set. Now there's audio clip effort going on. We have Dali, several Dali channels,


we have several clip variant channels about clope and lid and d-philip and d-clip and what all of


this exists. We have some channels where just people post the generated art, the generated


results from the available Dali variants and glide variants and so just join. Basically, I mean,


you could just reach out to us and ask me or someone else if there's a project where some help


could be needed. Or you could propose your own project. And if it's cool, we can try to connect


you to some of our sponsors to get GPUs or whatever. Cool. Anything else you want to get out


to viewers, listeners? Yeah, don't hesitate. Just like even if you're a high school student or


university freshman or whatever, like anyone can join. Like Theo Combs, who was the first to join


the project when I started. He actually, I always believed that he was something like a master


student or so. And later it turned out that he's a 16 years old high school student from London.


And yeah, he didn't know anything about deep learning at this time. Now he catched up.


But he was really good at doing all the server communication and he learned on the fly.


So we have many, many stuff. And if you have your own idea, if you would like to try to


train style again, or fine tune a Dali version, or whatever, just ask us.


All right. In this case, Cade, Roma, Christoph, thank you so much for being here. Thank you for


doing this. For anyone, yeah, check out the data set. It's pretty cool. It's a nice contribution.


Very, very cool contribution to the community. Thank you. And I hope this continues.


Thanks.


Deep Learning with Python, TensorFlow, and Keras tutorial

What is going on everybody and welcome to a much-needed update to the deep learning in Python with


TensorFlow as well as now Keras tutorial. It's been a bit over two years since I did just a basic deep


learning video in Python and since then a lot has changed. It's now much simpler to both like get into it


but then also just to work with deep learning models. So if you want to get into the more


nitty-gritty details in the lower level TensorFlow code you can still check out the older video but


if you're just trying to get started with deep learning that's not necessary anymore because we


have these nice high-level APIs like Keras that sit on top of TensorFlow and make it super super


simple. So anybody can follow along if you don't know anything about deep learning that's totally


fine we're gonna do a quick run-through of neural networks. Also you're gonna want Python 3.6 at


least as of the release of this video hopefully very very soon TensorFlow will be supported on


3.7 and in later versions of Python just happens to be the case right now it isn't I think it's


something to do with the async changes I'm not really sure. Anyways let's get into it starting


with an overview of how neural networks just work. Alright to begin we need to have some sort of


balance between treating neural networks like a total black box that we just don't understand at


all and understanding every single detail to them so I'm going to show you guys what I think is just


the kind of bare essential to understanding what's going on. So a neural network is going to consist


of the following things like what's the goal of any machine learning model well you've got some


inputs let's say x1, x2, x3 and you're just trying to map those inputs to some sort of output let's


say that output is determining whether something is a dog or that something is a cat so the output


is going to be two neurons in this case so it's just boom two neurons. Now what we want to do is


is figure out how are we going to map to those things we could use a single hidden layer let's


say we're going to do some neurons here and that's our first hidden layer. Now what's going to happen


is each of these x1, x2, and x3 these are going to map to that hidden layer each of the input


data's gets connected to each of the neurons in that first hidden layer and each of those connections


has its own unique weight. Now from here that first hidden layer could then map and connect to


that output layer the problem is if you did this the relationship between x1 and dog or cat and all


the other ones those relationships would only be linear relationships so if we're looking to map


nonlinear relationships which is probably going to be the case in a complex question you need to


have two or more one hidden layer means you just have a neural network two or more hidden layers


means you have a quote-unquote deep neural network so we'll add one more layer and then we're going


to fully connect that one too and then once that's fully connected again all unique weights each of


those blue lines has a unique weight associated with it and then that gets mapped to the output


and again each blue line has a unique weight associated with it. So now what we're going to


do is talk about what's happening on an individual neuron level so again that neuron has certain


inputs coming to it it might be the input layer x values or it could be inputs coming from the other


neurons so we're again going to call the inputs x1 x2 and x3 but just keep in mind this could actually


not be it might not be your input data it might be data coming from another neuron but regardless


that data is going to come in and we're just going to get the sum of that data so it's going to come


in and be summed all together but remember we also have those weights each of the inputs has a


unique weight that gets put you know multiplied against the input data and then we sum it together


finally and this is kind of where the artificial neural network comes into play we have an


activation function and this activation function is kind of meant to simulate a neuron actually


firing or not so you can think of the activation function like on a graph you know you got your


X and your Y and then a really basic activation function would be like a stepper function so if


X is greater than a certain value boom we step up and we have a value so let's say here this is


zero here the value is one so let's say this is our x-axis one two three so if X you know after


being all the inputs are multiplied by their weight summed together if that value is let's


say greater than three well then this activation function returns a one but today we tend to use


more of a sigmoid activation function so it's not going to be a zero or one it's going to be some


sort of value between zero and one so instead we might actually return like a point seven nine or


something like that so coming back to this neural network here that we've been drawing let's say


here on this final output layer you've got dog and cat well this output layer is almost certain


to have just a sigmoid activation function and what's going to say is maybe dog is a point seven


nine and cat is a point two one these two values are going to add up to a perfect 1.0 but we're


going to go with whatever the largest value is so in this case the neural network is you could say


79% confident that it's a dog 21% confidence a cat we're going to say we're going to take the


argmax basically we're gonna say we think it's a dog all right now that we're all experts on the


concepts of neural networks let's go ahead and build one you're gonna need tensor flow so do a


pip install dash dash upgrade tensor flow you should be on tensor flow version 1.1 or greater


so one thing you can do is import tensor flow and then actually tensor flow as TF and then TF dot


version will give you your current version so mine is 1.1 oh now let's go ahead and get started so


the first thing we're gonna do is import a data set we're going to use MNIST kind of the hello


world example of data sets with machine learning it is a data set that consists of 28 by 28 sized


images so it's like the resolution images of handwritten digits zero through nine so it'll be


like a zero one two three and so on and it's a handwritten kind of unique image so it's actually


a picture and we can graph it soon enough so you can see it's actually an image and the idea is to


feed through the pixel values to the neural network and then have the neural network output


which number it actually thinks that image is so that's our data set and now what we want to do is


unpack that data set to training and testing variables so this is a far more complex operation


when it's actually a data set that you're kind of bringing in or that you built or whatever for the


sake of this tutorial we want to use something real basic like MNIST so we're gonna unpack it


to X train Y train and then we're going to do X test Y test and that's gonna equal MNIST dot


load underscore data so that's gonna unpack it into there now just to show you guys what this is


we're gonna use matplotlib you can pip install or just look at it with me but we're gonna import


matplotlib dot pyplot as a PLT and what we're gonna do is PLT dot M show and we're gonna do X


train and we'll do the zero with so one thing we could do just just for the record let me just


print so we can you can see what we're talking about here so this is just going to be an array


it'll be a multi-dimensional array which is all a tensor is by the way so this is this is here's


your tensor right okay so that's the the actual data that we're gonna attempt to pass through our


neural network and just to show you if we were to actually graph it and then do a PLT dot show


it's gonna be the number and you can just excuse the color it's definitely black and white it's a


single color it's a binary so one thing we could say is the color map is equal to PLT dot CM for


color map binary replot it and there you go it's a it's not a color image so anyways back to our


actual code up here once we have the data one thing we want to do is is normalize that data so


again if I print it out you can see it's data that seems to vary from zero to looks like we


have as high as 253 it's 0 to 255 for pixel data so what we want to do is scale this data or


normalize it but really what we're doing in this normalization is scaling it so we're going to


just redefine X train index test but it's gonna be TF dot Keras dot utils dot normalize and we're


gonna pass X train and it'll be on axis one and then we're gonna copy paste and we're gonna do the


exact same thing for X test and all this does let's just run that and then we'll run this again


and you can see how the five has changed a little bit looks like I got a little lighter and then we


come down here and we can see the values here are now scaled between zero and one and that just


makes it easier for a network to learn we don't have to do this but at the end of this we probably


won't have time but if you want on you know comment those lines out and see how it affects


the network it's it's pretty significant okay so the next thing we're gonna do now is actually


build the model so the model itself is gonna start as TF dot Keras dot models dot and then


it's gonna be the sequential type of model there's two types of models the sequential is your your


most common one it's a feed forward like the image we drew so we're gonna use this sequential


model and then from here we can use this like model dot add syntax so the first layer is gonna


be the input layer and now right now our images are 28 by 28 and it's like multi-dimensional array


we don't want that we want them to be just like flat if we were doing like a conf net or something


like that we might not want it to be flat but in this case we definitely want to flatten it so we


could use that we could use like numpy and reshape or we can actually use one of the layers that's


built into Keras which is flatten so so we're gonna do add in what we're gonna add is TF dot


Keras dot layers dot flatten so one of the reasons why you you want this to actually be a layer type


is like when you have a convolutional neural network a lot of times at the end of the convolutional