So they train the encoder and decoder with this straight through estimator, because this


OpenAI DALL·E: Creating Images from Text (Blog Post Explained)

nearest neighbor computation isn't exactly differentiable.


They also train the codebook to match the outputs of the encoder.


So you can train that, or you can just take the the exponential average of the encoder


outputs.


And that's the VQ VAE, which is developed more in VQ VAE two.


So this is VQ VAE two, I've linked the papers, VQ VAE.


I was writing a three, two, the version two of it does the same thing, but in multi scale.


So here you can see that in the encoder, you, you, you take the image and you put it at


multiple resolutions.


So this is large resolution, this is low resolution, then you use the vector quantization to encode


this into this grid and encode this into the codebook vectors.


So again, here, maybe I've read red, red, this is red, and this is the green one, and


so on.


So you each square has to choose one of these 8000 vectors to represent itself.


And then you do this sort of hierarchical thing, where you use the deep a decoder on


this level to produce a slightly higher resolution image.


But then you quantize again, and you use a decoder at a next level to produce an even


higher resolution image.


So you can see that this hierarchical models, usually, these hierarchical models, if you


want good high resolution images, you sort of need them.


So you can see that the the top decoder here outputs something quite blocky.


And then every, every additional one adds a sort of details to the image.


It's pretty impressive as such.


And you can see that the training right here of the VQVA.


These are these are papers from last year or the years before.


So this has been known.


What Dali does is from what I can gather from the blog post, right here.


The images are pre processed to 256 to 256 during training, similar to VQVA each image


is compressed to a 32 by 32 grid of discrete latent codes using a discrete VAE that we


pre trained using a continuous relaxation.


Okay, there's a lot of there's a lot of stuff here.


So the VAE is pre trained.


And they're saying, they're saying also down here, that their model uses maximum likelihood


to generate all of the tokens one after another, it's decoder only and so on.


So probably this whole pipeline here is pre trained, they pre train a VAE, a discrete


VAE.


And then they simply, the Dali model simply has to learn how to produce the tokens, right,


the Dali model simply has to learn how to produce these hieroglyphs.


And the box is fixed.


The box is not changed, it's possible that they also train the decoder here.


So the decoder, but I don't know, I can't tell this from the blog post.


What's certainly is that they what's certain is that they don't train the encoder.


So what you would do in a single step of Dali is you would have your text right here, blah,


blah, blah.


And you would have a partial image, okay, you would input this text and the partial


image to Dali.


The partial image is any image where you've blacked out the bottom right, and they do


the bottom right simply, it's the same as you do left to right by text.


So you do so sort of top left to bottom right.


And yeah, it's it's good because you can always flip an image, maybe not actually, but it's


just a bias that you have to provide the model with in order to do autoregressive training,


right?


So here is the image of that cat.


Right?


That I, and you black out the bottom right, you can black out the whole image if you want


the model to produce images unconditionally.


All right, so you black all of this out.


Cool.


So now, what you do is these here, they are already, they are already words, right?


You tokenize those token, token, token, and you go into your vocabulary of text, right?


So there's a vocabulary of text somewhere, there's blah, and you encode all of these


using that vocabulary.


So this is maybe word 34.


So this is word 34, 34, 34.


You go to your image, you rasterize this according to your definition, okay.


And then you go and run this through this encoder that you trained.


So you run it through the box.


And the box will tell you for each of this grid outputs, the box will tell you, well,


in my, in my vocabulary of image pieces, this here is number two, this here is number four,


this is two, again, this is 35, and so on.


So you do this left to right, top to bottom, and then you put it right here.


Okay, so this is followed by an image of 242 35.


And what you ask the model to do is simply to predict from all of this, and the model


knows that these are this is text, and this is images from all of this predict the next


token, which would be this token right here.


So you want to predict this one, right here, what is it?


And that's how you train the model, right?


And once it gets that you can try, you can ask it to predict the next one, and so on.


And in this way, you can let it generate an entire image at inference time.


And you know, you can train this, they say all these tokens are generated autoregressively.


Now, in my understanding, this is all the model does, because once you have that token,


so if the model says this is number seven, you go back to your box.


And you say, please, or it's a different box.


Like this is the encoder.


This is the encoder of the VQ VAE.


And now you go to your decoder that you've also pre trained, right?


This is a different box.


And you ask it, I have this image, right?


I have 242 35 and seven, please generate an image for me for that.


Or maybe you want to want to wait until you have the complete image, right?


So you have the complete image, and you give this to your decoder.


These are now that these hieroglyphs, right?


So you have the box and the box produces an image and the box says, Well, okay.


This cat here, probably it reproduces the ears fairly well, because you can describe


them sort of exactly, maybe you also want to copy that over or something.


But then it says, Well, it's a cat.


So I'm going to, you know, maybe this, if the model has done a good job, there should


be some sort of a cat, right?


And the model, you know, maybe in these hieroglyphs, it's even described how the cat looks like


the cat looks straight ahead, as whiskers, as eyes and so on.


Okay, so I'm going to guess that the part on top that is trained, and the part on bottom


is pre trained with the option that the decoder part could also be trained at training time


at the same time they train this language model on top.


So they make some further inferences right here, they say, each image is compressed in


latent codes using a discrete V that we pre trained using a continuous relaxation, we


found that training using the relaxation obviates the need for an explicit codebook, EMA loss


or tricks like dead code revival and can scale up to large vocabulary sizes.


And this is the part where I am a bit confused.


So clearly, they say they have a vocabulary in the visual domain.


Okay, there are 8192.


Well, I'm, I'm don't know my powers of two 8192 different words in the codebook.


So there must be a codebook, but they say there, this obviates the need for an explicit


codebook, so I don't really know what to make of that.


I can tell you what a continuous relaxation might look like.


So this is from a different paper that they linked of the concrete random variables.


So if you have an operation such as this, like a discrete random variable, you need


to take an arg max of it.


What you'll have is, you'll have some sort of logits, right?


There may be like this, and you take the arg max of it, which means that you put it into


a distribution where it's just one value.


And this is sort of the same operation as we do in the VQ VAE, where we assign each,


each output of the encoder to the nearest codebook vector, we say you can only have


one of the codebook vectors, that's it, right?


Now, what you want to do when you relax this is you want to say, well, instead of that,


what you could do is you could just kind of take that codebook vector a lot, but also,


you know, take a little bit of the others.


So more than doing a hard assignment to a codebook vector, right, so here would be the


output of your encoder and you hard assign it to the nearest neighbor.


You want to say, well, I'm going to soft assign it to all the ones it's sort of like the difference


between k nearest neighbor and and a Gaussian mixture model, as I understand not not what


they do here, but it's analogous to that.


And with that, they don't need an explicit codebook.


And I don't know what that means.


What I can imagine is that they don't actually train the codebook vectors, maybe they just


quantize to some prefixed schema, or I just don't understand what they do.


Yeah, here's an illustration of these discrete random variables.


So you want to get to a point when when you sample the variable, as you drop your temperature,


it more and more approaches this fixed sampling, like you can be either here or here or here


with the sort of masses that are indicated by the size of the circle.


But as you increase the temperature, you go more to a mixture.


So yeah, you can be at the corner, but you can also be kind of in this region or in this


region or in this region.


As you increase the temperature, you can see the the distribution becomes more of a mixture


distribution.


And the mixture distribution, any mixture distribution with a temperature other than


zero, of course, now, all of a sudden has sort of a defined gradient, whereas these


discrete random variables, they do not have a gradient.


And that's the reason why the VQ VA needs to do this straight through estimator right


here, because this hard assignment to the codebook does not have a gradient defined.


With the soft relaxation, you do have a gradient.


And maybe they just mean they don't need, they don't need this hard assignment to the


codebook.


I'm not sure or maybe they just they quantize in a different way.


Maybe they go back to a continuous latent space.


Yeah, I can imagine they they might go back to continuous latent space, but somehow, somehow,


they still do this a form of quantization.


This could be a fixed quantization, like you say, okay, you can choose any of the basis


vectors and some some mixtures that we define between them, or they define it via moving


averages, or they define it via batch statistics, or I don't know.


If you know, let me know in the comments to the video.


Alright, so this was my take on what the model does, and what is probably behind it.


Now let's look at some more examples right here because these are fun.


So they, they say it can sort of control attributes.


So you see, these, it's for example, a pentagonal green clock, and you see, it's not always


pentagonal, it's sometimes hexagonal, and sometimes heptagonal, and whatnot.


But in general, what it does well is sort of color, and also kind of object description.


So lunchbox it gets and green it gets what it can't do super well is stuff like counting.


Okay.


So I have sort of a hypothesis, I have multiple hypotheses about here, just see, watch in


all of these examples, how the text prompt is phrased.


So it says a pentagonal green lunchbox, a green lunchbox in the shape of a pentagon.


This is quite unusual way to phrase the prompt.


And by the way, all these criticisms that I'm leveraging here, most of them are actually


admitted and discussed in this blog post.


It's actually it's pretty cool and pretty self, let's say self critical of them.


So it's this is I've, you know, I thought of these things, and then I read the little


text, and then they, they already describe what I concluded.


It's sad.


But yeah, it's pretty cool of them, because the current climate is sort of making research


look as as cool and flawless as possible.


This goes a bit against it.


So they say that the images here aren't cherry picked.


And I totally believe this.


So they have a little trick that they do.


They output, I think 512 images from their model because they can sample and then they


re rank them using this other model that they've released this clip model.


And this clip model is a pretty good re ranker.


So you give it a piece of text and an image and sort of tells you how well they fit together.


And so the outputs that you see here are re ranked by this model.


So you see are strictly the best outputs, according to that model.


So it's not cherry picked by humans, but it's cherry picked by a very good model.


And the second thing is that the text prompt here is absolutely cherry picked, right?


By the way, this is phrased, you can see that it is very, very brittle, probably the model,


I can't test it, but probably it's very brittle in how exactly you phrase this text prompt.


And I'm going to guess they have tried a lot of things before they've released these few


examples right here that they show, and they, you know, made sure that they work.


So yeah, just keep in mind that this is very brittle.


And we already know this from like GPT-3.


We know that the input might seem the same to a human just phrased differently in some


cases, and yet the model will output completely different things.


And we know that a lot of these GPT-3 examples are very, very constructed in terms of the


input prompt.


So yeah, the other thing is the model, as I said, it can do colors and it can do colors


and textures pretty well.


So we've already seen the things made of things.


So the sphere made of noodles that actually probably exists, the sphere made of guacamole.


However, it's not super good at counting, for example.


And I have a sort of multiple hypotheses.


So these image models, they tend to be very good at sort of style and texture.


Style and texture are the domain of these image models, like anywhere where there's


like a convolution.


And by the way, they use in the VQVAE model, no, not in the VQVAE, in this transformer


for images, they don't do full attention.


What they do is each one of the image tokens can attend to each of the text tokens such


as this, but the image tokens, they can only sort of attend in the grid, layer by layer


at in one layer, they can attend sort of to the row of other image elements.


In another layer, they can attend to the same column.


And in even another layer, they can attend to sort of the the surroundings of them like


a convolution.


So they can attend to, let's say there are a couple of neighbors right here.


So it's not full attention yet, and every layer, every image token can attend to all


the text tokens.


So yeah, in these models, what you'll typically see is that textures and style is pretty good.


However, global correspondences are not as good.


And that's what you see a lot in these face models, where the left and the right earring


don't match and things like this.


So global correspondences are not so good.


And you would actually expect that objects aren't as good as well, right?


So here, this is still a clock, this is still a light bulb, this is still a stop sign, right?


So it somehow gets the objects correct, which, in my hypothesis, it shouldn't, because this


is some sort of a global structure.


However, I think that's just a matter of how the data set is collected.


The data sets are probably we humans, we take pictures of objects, right?


So the the fundamental structures in these data sets is the object.


So it makes sense that it learns that we humans we don't, we don't take pictures, and we often


don't describe the count in them.


So I can get that the model has a harder time to learn that and actually focuses just on


the object as a global thing.


The count would be a global thing, right?


But it's not that prominent in the data.


And the rest is a local thing, like the color, the texture, and so on.


Yeah, the cube made of porcupine.


So you can see here that this this counting, so two is often quite good.


Actually here it mixes up glasses and glasses, right?


So two often works.


However, if you go if you go past two, it often gets it wrong.


So five, you'll get anything from three to seven clocks and so on.


So I'm going to also guess it's very brittle, like they're not here.


Yes, they're sitting on a table.


But if you take a object that's not that often on a table, like a club, you'll see that it's


pretty unrecognizable whether or not it's on a table.


Five, four clubs.


So you know, the model is prone to ignoring part of its input if the likelihood in another


part is larger.


Also, it can't do things like this, you know, a stack of three cubes, a red cube is on the


top sitting on a green cube, it often gets the order wrong, like it gets the cubes on


top of each other.


However, it often gets it wrong when it comes to you know, the order the global things,


as I said, anything global that is not what the object is, tends to be weak, anything


local tends to be strong in these models.


And that's just a matter of how they're built and how the data is.


So they say the image can render new views.


And here is where I'm not as convinced.


So here you have like an extreme close up view of a cub, cub, cabi bar, sorry, of a


fox, they're close up, sometimes they're extreme close up, right?


And you can see that it gets like forest it gets it gets pretty well.


But then you say, okay, ground level view, like, and then you say, okay, an aerial view,


maybe some of them are aerial views, some of them aren't.


What's pretty cool is things like a okay, a fish eye lens view.


I mean, that's, that's pretty cool.


And a, they had some of them a bottom view, or a rear view.


Yeah, the rear view works better.


So it does understand these these kinds of things like what's the rear of a fox and what's


the front of a fox, though, as you can also see, not always.


Texture, it's very good at texture.


So here, something made of voxels can do that perfectly.


An owl made of voxels, like this looks like it comes straight from Minecraft, right?


Absolutely, absolutely cool.


Even x ray sometimes doesn't always get the bones right.


But yeah, as I said, style, structure, very cool.


So here is an example of a completion.


So they give the the text prompt a photograph of a bust of Homer and the image, the top


part of the image and they say, well, it can describing a well known figure, it can complete


the figure.


I don't agree that it completes Homer like it completes it probably just sees this bust


and this and it just completes, you know, whatever fits.


I don't, I have not studied Homer as a historic person or busts of him.


But you know, I disagree that this depicts largely the same person very often.


You can see here there is sometimes there is even, you know, there's completely unrelated


stuff.


There's that lady with the pearl earring by Vermeer somewhere in there, and so on.


And what I also like in this kind of this, this one, you know, the game draw something


where or you know, pictionary and so on.


There are people when they can't draw something, they're just kind of write it on the picture.


It's like, ah, screw it.


And I'll just write it like, this is Homer.


This is Homer.


Now, I don't care what you say.


But you know, it does, you know, it does.


So when you say Cleopatra, it goes more into the into sort of the female direction.


Medusa, it has some though I'm pretty sure Medusa has the snake, the snake hair, no,


maybe Venus.


Yeah, somewhat somewhat.


Um, it they test a lot of things like can it do mirror reflections.


And you can see right here, they say it can do reflections on the ground pretty well.


But it can't do reflections, for example, in a mirror, because in a lot of these pictures,


the object like here would actually have to be in front of the mirror.


However, in the fewest amount of pictures, the object mirrored is actually also in front


of the mirror.


So that kind of global correspondence isn't given as much.


However, there is a fair bit of reflection on the ground, so to say.


So you know, that's pretty cool.


But it's also probably very, very common in data sets.


Yeah, cross section view of a walnut.


So they sort of implore, sorry, explore the model, what it can do.


And here you can see that, you know, if something is common in the data set, you know, like


the cross section view of a human head, there are a lot of pictures of that right in the


data set.


However, if it comes to cross section view of a where, where did I see the airplane,


there is an airplane somewhere, it's less, it's less so.


So you can see that this is still it is so here, it probably doesn't really know how


that looks because you know, they probably on the image on the internet, even on the


whole internet pictures of cross sections of airplanes or any sections of airplanes


are not really distributed often.


So it sort of just focuses on airplane and then with cross section, it probably knows


that it should somehow display some of the interior.


So it just kind of produces some stuff that matches this thing.


As I said, if if it can't make the likelihood high, of all of the things, what it tends


to do is just focus on one of the things and just make that likelihood high, which is reasonable,


you know, for a model, a macro photo, macro photographs of stuff.


These are pretty cool.


This is what you would find in some image galleries.


Absolutely.


Then it can do various things like style transfer.


And here is where it shines, right?


So you can have different paintings of different objects in different styles.


So here you can like have an owl sitting in the forest in the morning.


And you can have this as a painting as a painting in the pop art style and so on is very, very


impressive.


So I absolutely implore you actually to like as a postage stamp.


These are these are these are absolutely amazing.


And yeah, you can have stuff like stained glass windows.


And this is Yeah, it's where the model shines.


And even here, a storefront that has the word open AI written on it.


So just right now, just look at how convoluted this text prompt has to be for them to get


this to work.


It's impressive, but the text prompt has to be repeated and reformulated a bunch of times


and so on.


My personal favorite is the Pytorch chips.


They're crunchy, you get a piece of back prop in every package.


So you can see it sometimes misses like this is perch chips and so on.


It sometimes misses, but it is pretty cool that it basically can do OCR, right?


It or reverse OCR.


You can you give it a piece of text and it sort of makes a picture with that on it.


It's very, very impressive, even though as we said, like the global, the global correspondences


are not always there.


They do implore like fashion, a skirt, like here that the yellow skirt, then you know,


these mannequins.


And here they have a loft bedroom with a white bed next to a nightstand.


There is a fish tank standing beside the bed and they give sort of the beginning of the


image and here's what the model comes up with.


And you know, you can imagine that there are a lot of pictures like this in the data set.


So the model might be pretty good at stuff like this, though I have found their king


bed next to Yeah, let's say the nightstand with the telescope.


The telescope beside the bed, it just, you know, that beside like there's a telescope,


sometimes it's on the bed, sometimes it's next to it.


There's some weird telescopes around.


Well, this is a lot of telescopes.


That's a weird telescope.


But you know, the quality is pretty impressive.


This is absolutely nitpicking that I'm doing here.


Combining unrelated concepts, we've already seen the armchair in the shape of an avocado.


We also have a snail made of harp, though my personal favorite is the penguin made of


garlic.


The penguin made of garlic.


This perfect, right?


Absolutely adorable.


And just qualitatively like this, this would take a human like you would pay a high quality,


highly educated Photoshop artist, quite a bit of money to get this sort of output, right.


And these models, they they shine at this sort of style transfer, texture stuff.


And it was here, yeah, you have the illustrations, you can have any kind of illustrations, like


the illustration of a baby shark with a mustache holding, there's holding an umbrella somewhere.


Playing it, running, riding a unicycle.


It's just, it's just nice.


And as I said, this is the same model that can do all of this stuff.


And these are samples.


They're just samples.


They're not cherry picked.


They are re ranked.


Remember that.


So they can do hybrids of images, hybrids of different giraffe and turtle and so on.


And they do sort of implore the model a little bit more, where they, as I said, they give


this cat on the top and they say they want the exact same cat on the top as a photo colored


blue on the bottom.


So you can see that doesn't always work, right?


But in a surprising amount of times, it actually does work.


Sometimes it's just like a blue pot.


But you can even see, it's not the finished model yet.


However, it is a step into the direction that shows us that this is definitely, definitely


possible.


They can do some of these progressive matrices where it fills in the bottom right.


However, they do mention it's very, very finicky with respect to whether or not, for example,


if you invert the color.


So if you look at the bottom right of any of these things, if I invert the colors, the


output sort of changes, and it's often also not right.


However, sometimes it is actually right, which is crazy, because in some of these things,


you have to do some crazy sort of inference that we usually we usually do these things


in IQ tests.


So I don't know the debate about what is intelligence goes on.


They say it has geographic knowledge.


However, I'm not sure it has geographic knowledge as it just associates words with particular


images like they say, okay, this is a photo of food of China.


Okay, maybe you just not sure this classifies as geographic knowledge is, is, yeah, also


this temporal knowledge, a photo of a phone from the 20s.


Okay.


You know, and then the different time periods 60s, 70s, 80s, future and so on, like distant


future, like, wow, these phones.


I particularly so I like the usually this stuff.


It's it's pretty okay, right?


But it's not temporal knowledge.


It just associates a bunch of tokens with some sort of style of computer.


Today's computer, the future computer, the distant future computer, please know, please


don't please, please don't give me that.


I don't want to I don't want that.


I love the action movie poster, because so the style is correct.


It just says action movie in the future.


Yeah.


They do get sort of the kind of some of the styles.


It just says action movie.


Like this is like a like a naggy naggy child.


Like I'm hungry.


Hi, hungry.


I'm dead.


All right, so they also have a summary right here.


And they do show what it means that they they use this clip to rerank.


So on the left here, you can see just eight samples straight up from the model.


And they're not too bad.


But you know, you increase the quality by sort of sampling more and then taking the


best eight as you go to the right here, according to the re ranker.


So I'm going to guess they decided on 512 because that was sort of, you know, it gives


you already pretty diverse, pretty good, pretty high quality outputs right here.


All right.


So just lastly, shout out to the the the authors right here.


The primary authors are Diter mesh, Mikhail Pavlov, Gabrielle go and Scott Ray, with a,


I guess, the secondary supporting authors and most of open AI behind them.


Though I don't know how they work.


I would encourage you to go look at the model.


It's pretty cool.


Try out all these inputs.


As I said, these are the inputs are simply restricting you because they don't trust you


with their model yet, right?


In the real model, you can input any piece of text that you want.


And you will get out an image.


And the fact that you have to select the stuff here is simply because that's the stuff they


tried.


That's the stuff their PR department has signed off on, right?


And so you get to see that because, as I said, that they're not like, this is at the same


time, this is a PR dilemma when you release a generative model, because it, you know,


it could release, they discussed this a little bit in the blog post, you know, it could release


like very problem problematic images in a classifier, it's not as pronounced.


It's also sometimes dangerous, but not as dangerous as if you have a generative model.


That's the first thing.


And the second thing is there is, I mean, there is money in this.


Definitely, definitely money to be made in this.


So you know, we'll see whether or not we get the full model or not.


Alright, with that, that was it for me.


I hope you enjoy the blog post.


I hope you enjoyed the video.


Why Kernels - Practical Machine Learning Tutorial with Python p.30

What is going on everybody and welcome to part 30 of our machine learning with Python


tutorial series and another video for the support vector machine and really kind of


in general kernels and all of that.


So what we've been talking about is how to handle for non-linearly separable data and


up to this point we've been talking about the kernel and how we're going to do some


sort of magical thing with the kernel of going into plausibly an infinite dimension and doing


some calculation there or actually having a calculation done there for us so we don't


actually have to visit that dimension and yet we can get back the answer somehow.


So that all sounds really hand wavy so I think it's important that we can actually kind of


work this out and see how that actually works so that we have a little more faith in this


whole thing because it's kind of crazy.


So first of all when you're going to write out a kernel generally let's say you've got


X, we're going to call our feature set X and then we're going to call the new dimension


or the new space our Z space.


So you're going to have, you're going to say usually something like this, you're going


to say K of and we're going to say X comma X prime.


So just for the record that like slash looking thing is a prime, I'm going to do my best


to keep it like a really slanted slash so it's not confused with an X supposedly to


the power of one.


But anyway so X and X prime and that K is just your kernel and that is going to equal


this new Z space where you've got Z dotted with Z prime.


Like I said I did not want that to be a one so maybe that, I don't know.


Anyway that's a prime.


So first of all what is Z?


Okay right so you can kind of think of Z as some sort of function that's being applied


to its X counterpart.


So Z equals let's say a function of X and then Z prime is also a function of X prime.


And when you do a kernel, let's say a kernel on something comma something, what are you


actually doing?


You're doing, you're running first of all you're going to convert X to Z right?


The constituents will be converted right?


This gets converted to a Z.


This is getting converted to a Z prime and then what are you doing?


Well it's the dot product, the dot product or the inner product between this and this,


whatever that happens to be.


And that's a kernel.


As long as you're doing those two things it is a valid kernel and it's important to note


that function and function are equal to each other.


You have to do the same thing right?


You can't just kind of do one thing to X and then another thing to X prime.


It needs to be the same.


It needs to be symmetrical.


So keep that in mind as well.


Also before we move on I don't know that I'll ever really show it again but a lot of times


you might see rather than you know K of X or whatever you're going to see the kernel


being denoted as a phi or phi or however you want to pronounce that.


So you might see it in an equation like Y equals W phi X plus B. Okay something like


that you're going to see and that's just going to denote that hey this person is using a


kernel and not just the straight up formal SVM calculation.


First of all inherently we can see right out of the gate again what does a dot product


or an inner product what does that produce?


It simply produces a scalar value.


That is it right?


And so all we need from this magical Z space well really it's this one and this one right?


All we need from that Z space is actually the inner product right?


And that is the kernel.


So the question is of course can we calculate the inner product of that Z space without


actually knowing that Z space and needing to process that Z space?


So first let's consider a really simple feature set it's just going to be a nice two dimensional


feature set kind of like what we've been working with really up until this point.


So let's say your feature set is and I'll even let me write that out.


I'm not going to write that out because that's messy.


So your feature set you're going to have X1 comma X2 that is your feature set.


Your feature set is a vector and then let's say we want to take ourselves out to some


sort of Z space and the way we're going to do it is with a second order polynomial.


So we're going to convert this to a second order polynomial.


So how are we going to do that?


Well when you do that you're going to start with one then you're going to just take the


X1 then comma X2 right?


So it's going to create a new vector for us right?


And this new vector is going to come out to be six dimensions right?


So you're going to start with one for second order then it would be just one X1 X2 and


then X1 times itself X2 times itself and then X1 times X2.


That's all you do to just we're going to take it out to that second order right?


So let's go ahead and write that out.


It probably would have been easier the first time anyway.


So first of all our feature set like let's say those X's are I don't know little X's


but the big X right that feature set is equal to X1 X2 and all that.


Then we're going to say this new Z space is going to be Z equals and again it's just a


second order polynomial so we're going to take it out to and actually I think we'll


go to another page.


So I'm saying X equals I just wanted to have some more space here so it could fit everything


hopefully.


So X is equal to X1 comma X2 this is a vector and then we're taking it out to the Z space


and we're going to say actually Z and Z is just a second order polynomial of X.


I've already kind of explained that now we're actually going to write it out.


So that's just going to start with the number one comma again this is going to be a vector.


It's going to be comma just the first two original values that we had so that would


just be X1 comma X2 comma then it's going to be the first value multiplied by itself


so just X1 squared.


X1 squared.


Let me do the same thing so X2 squared comma and then it's just X1 times X2 so X1 X2 and


again this is our new Z space.


So this is our new Z space where Z this new Z space the function that we've run here is


second order polynomial.


Okay and I'd already went through that logic but it's just one you know the first value


second value the third value or really the fourth value will just be the square you know


the first value times itself then the second value times itself and then finally that sixth


final value is just the first value times the second value.


Second order polynomial.


So that is our Z space and remember we said that K let's we'll write it out we said K


of X and X prime and sometimes you might see K of X and Y or some people might even write


X1 X2 okay just don't worry about the notation just understand when you see K and then two


values that's a kernel.


So K recall K of X and X prime we said was going to be equal to the dot product of Z


and Z prime right and Z and Z prime are going to be identical right because they're going


to be the same this same notated vector at least.


So what is the dot product of Z and Z prime?


Well what is the dot product?


What is the inner product?


Well it's just each value times itself so in theory we would know we would know Z prime


basically I'm just going to write it out really quick but Z prime is equal to this right it's


just rather than X1 it would be I suppose I'll write it out so that's Z and Z prime


same thing it's just you know obviously just throwing in the prime and then K X of X prime


what were the two things we were doing we're applying first that function to X which we've


just done to convert it to Z but then that final thing is it's just it's the dot product


of Z and Z prime so you can really think of it now as K X X prime equals Z dot Z prime


right and what does that equal?


Well recall the dot product is just each kind of constituent of the vector multiplied by


that same place constituent so what we're going to have here is you know one times one


so that's just simply one plus and then it's X1 times X prime one so X1 times X prime one


plus X2 X prime two so X2 X prime two plus X1 squared X prime one squared so X1 squared


X prime one squared plus X2 squared X prime two squared plus X1 X prime one X2 X prime


two.


Okay so that looks obviously really messy those primes are pretty hard to truly denote


out here but hopefully if you follow along you got the whole primes and stuff like that


if not you can check out the text based version on pythonprogrammer.net and I'll use something


like latex or something like that to make it very nice and neat so it's totally obvious


what's prime and what's one and all this but hopefully you recognize that you would never


write to the power of one.


So what we've done here is actually we have gone to the Z space right because you and


I went to the Z space and actually performed the calculation because we actually did the


conversion to the Z space and then performed a dot product of that conversion to get our


answer okay and as you can see that was actually a really long and very tedious my hand kind


of hurts after writing down all of those things right and so your processor also kind of hurts


because that's a lot of stuff.


So the next question is can we do this kernel without actually writing it out or without


actually having to visit this Z space and the answer is yes and this is again this is


called the polynomial kernel.


So the polynomial kernel is the following so you're going to say k x comma x prime and


this equals one plus x dotted with x prime to the power of p you can have that power


of p be whatever you want so basically to write this out in further notation let's say


you've got a number of dimensions and then p is basically whatever you want it to be


what we just did a second ago was p equals two and for we'll say however many dimensions


you want but let's say n and n was also equal to but you could in theory have p equal many


more but first let's write this one out and then we'll consider the difference of adding


and changing and manipulating p and n here momentarily.


So first of all do you see any Z in this in either of these calculations I don't there


is no Z okay so what would happen though if we for example let's say we wrote out let's


expand this first so this would be basically one plus x one x one prime and then basically


it's plus dot dot dot all the way out to x n x prime n to the power of p whatever that


happens to be.


Now imagine though how much more difficult is this equation if we changed n to fifteen


and p to a hundred now obviously you know by hand or something that would make this


significantly more challenging but by a let's say a calculator by python or something like


that by a processor how much more difficult is this equation not much more difficult I


mean it might it might cost you an extra cycle or a few cycles or something but for the most


part that's not that much more difficult as opposed to here where we're actually going


to the Z space and in this case that was just a second order polynomial this could get way


more complex we're just simply going to like in this case we've we've gone to the Z space


and done the dot product and found our answer and it's it's pretty ugly already just looking


at it whereas over here we actually don't have to have to do that much and in reality


if we were to if we were to work this out we would find that we we had the exact same


problem and then you can even you could pull this down into their own little separate separate


vectors and it might give you it'd give you a vector like one comma b x one squared x


two squared square root of two by x square root of x oops square root of two x two and


then the square root of two x two times x one x two and it's important to know that


so that square root of two is simply avoiding it's just a square root of two times x one


times x two so keep that in mind so what would happen when this is a vector and it was multiplied


by itself what would you get in return like what would that scalar value be right well


that would be your second order polynomial dot products between the two x's having been


transformed to the Z space dot product those two things you have the exact same scalar


except you went through a lot less work so another kernel is going to be our RBF or radial


basis function kernel it's a lot more complex and you can't really conceptualize it as easily


as we could the polynomial kernel regardless the kernel is k x x prime is equal to the


exponential so it's an exponential function of the negative gamma times the absolute value


of x minus x prime squared so in theory this can take you out to infinite dimensions and


just in case anybody is not familiar x you know an exponential function like this so


exp x equals e to the x so what's happening here is plausibly you could go out to an infinite


number of dimensions now as you may guess that could get you into a little bit of trouble


also it's infinite dimensions I struggle to say fundamentally but in reality it's really


not really I guess it's hard to explain you can save that for your doctoral dissertation


if you'd like so anyway one problem though that you might get by going out into an infinite


number of dimensions as you might guess is that maybe just maybe you might wind up maybe


your data shouldn't actually be there maybe there's nothing there in your data but if


you're willing to go out to an infinite number of dimensions it's likely that at some point


you're going to find some sort of linearly separable case and when your data really really


wasn't meant to be linearly separable in the first place so in the next tutorial what we're


going to be talking about is when you're using a kernel to kind of force your data into some


sort of linear separability when you do that how might you know when you've made a mistake


and then also not only that what can you do to avoid this exact same mistake which actually


can be pretty common even with a polynomial kernel but really RBF and just for the record


RBF or radial basis function this is your default kernel this is the one that pretty


much is going to be used depending on whatever library you use but with scikit-learn for


example the default kernel with the support vector machine is the radial basis function


simply because it's going to work most of the time fundamentally again and theoretically


you can actually kind of you can make data sets that RBF will not be able to linearly


separate it's possible mathematically to do that but in all reality it is almost certainly


unlikely to ever be the case that RBF won't be able to separate your data so anyway you


can get into trouble like that so you want to watch out when using the RBF kernel so


anyways of course though after you know just because you fit a support vector machine you


know you've optimized your support vector machine you can also one way one way you can


find out right away is what we're going to be talking on the next tutorial but also once


you test it and you know you might actually have optimized it you go to test it and you


have you know 50% accuracy and you're like what the heck that's kind of that's probably


why you ran into some trouble is what we'll be talking about in the next tutorial so anyways


and what we'll be talking about might help you remedy this or maybe your data actually


just is not separable it's not good data so anyways if you have questions comments concerns


up to this point feel free to leave them below otherwise as always thanks for watching thanks


I'M TAKING A BREAK... (Channel Update July 2020)

Yes, you read that right. I am going on a break. Don't worry though, there will still be videos, just not as many.


I've decided to basically reduce the upload frequency a little bit, mostly because I am going on a break,


but also because I kind of want to have time to do other things, but we'll get to that later.


So how's our little channel doing? We've just passed 1 million views.


One million times someone thought, well that's kind of worth watching and only about 900,000 times where they were severely disappointed after clicking on a video.


I think, I still think that's a net gain honestly.


The channel just surpassed 30,000 subscribers, so technically in log space we're already halfway to a 100,000.


It's only a matter of time and I think I've said this in the last update,


but this is just absolutely overwhelming how many people are interested in machine learning research and topics related to it.


So that's pretty cool and encouraging.


Thank you everyone who has already subscribed and especially the people that leave comments, the people that share the videos.


This means a lot and I think it's awesome and it's quite motivating to continue doing this honestly.


I'm having lots of fun.


Along with that I've gained almost 5,000 Twitter followers, I think more than 5,000 Twitter followers,


which is strange because Twitter is weird, but you know, so that's pretty cool I guess.


I wonder if all of those are subscribed to the channel.


In any case, I just want to highlight again that the community around machine learning research is in the absolute largest part a very, very positive community.


You people are absolutely great.


The comment sections are just so much better than anything else on the entire internet, including paper reviews at major conferences.


Really, this is a half joke that the comment section is better than the reviews on papers,


but it is actually very often true.


People are discussing ideas in the comments that are valuable and creative and asking interesting questions and helping each other out.


And that also counts for our Discord server.


So if you're not on our Discord server, we do have one.


There is a channel for beginners question.


There's a channel for discussing the videos that are on the YouTube channel and people are generally very, very helpful there.


It's a vibrant community and I can only recommend that if you're looking to contribute to the community and be part of it, it's a great place.


That being said, I'm also on a number of other platforms such as LinkedIn.


I finally made a LinkedIn account.


I was always kind of skeptic.


I don't know how LinkedIn works.


Like what is the difference between follow and connect?


And then people like write little messages while connecting and it says like, I'd love to connect.


But then you accept them and then that message pops up and then it's saying, I'd love to connect.


But you've already connected at that point.


This is weird.


Like how does LinkedIn work?


Someone tell me, what is it for?


Like I get it.


It's like professional social networking, but it's just it seems weird to me.


OK, but there is an entire community there.


And I do post my videos there.


I'm not like super active on LinkedIn.


I have to say that.


I'm also on BitChute, Minds, Parler.


So the reason why I'm mentioning these things is that with recent developments, especially around this Jan Lecun video, there were some developments that potentially threatened the existence of this channel.


And I don't want to make it a single point of failure.


So I would appreciate it if you follow me on at least one other thing, at least one other point of contact.


So that in the case that something might happen, which is unlikely, but you know, can, I still have a way of distributing this content.


All the links are in the video description.


I'd love to see you there wherever.


So with respect to the Jan Lecun situation, he has left Twitter now de facto.


And people wanted me to kind of make a follow up video, asked me about it.


But I feel, you know, I have I have nothing more substantial to say and just to make a video for video sake is not really a thing I want to go into.


So it's kind of sad, but these kind of news and drama videos, they do get a lot of attention, not like outrageous, but they do get.


I want to keep this channel mostly about the machine learning research.


And I only want to make videos when I really do have some information to add.


You know, Jan Lecun is an adult and he's able to make his decisions of whether he wants to leave Twitter or not.


It's probably for the better for his mental health.


So with respect to the drama videos, I always kind of say that I'll pull you in with the drama and then before you know it, I educate you.


Ha, checkmate.


So that's how this works and how we ultimately end up with many more machine learners than originally wanted to be.


We gotcha.


So in other news, Flatland, round one of Flatland has officially started.


And I've previously made a video about Flatland.


It is a NeurIPS challenge.


It is a challenge where you have to route trains around a 2D map.


And I wanted to make this kind of a community project where we do research together, hopefully do something with machine learning, with reinforcement learning to tackle this problem and to crush the challenge.


And this is looking extremely good.


So on our Discord server, there is a core group of people that is really engaged.


And this is one of the reasons why I kind of want to reduce my upload frequency because I want to have time to participate more in these community efforts.


I really want to have time to do more myself in the Flatland research group that we have.


We do open research.


Anyone is welcome.


You're still very welcome.


Join our Discord server and join us and contribute to the code.


There is a core group right now that is really pushing forward.


So just to highlight a few of them, there is Novik, Edward Durek, Dolas, Frostbite, China CEO, trademarked, AI Adrian, and I'm Peter.


And I forgot Nuber Ponij.


I'm so sorry, Nuber Ponij.


You're beautiful.


These people are helping each other reach more and better performance, profiling code, implementing parts from other papers.


And it's just great to see that people can collaborate on this, even though it's technically a competition.


But, you know, I guess we're competing against nature.


For anyone I haven't mentioned here, I'm very, very sorry.


Any list of names is always prone to leave people out.


I do not want to diminish your impact.


So right now, this DQN algorithm is able to reach about a 95% success rate with three to five agents on the map.


So three to five trains.


But round one has just started and we see that some of these environments have many, many more agents in them.


So there's still a lot of work to do.


So we need you to come and contribute and join the fun.


It is fun.


And as I said, I will be working on this myself more as well because it's fun.


So again, big shout out to anyone on Discord that has contributed in any way.


This is just awesome.


We've just had recently our first Flatland Town Hall with entirely community generated content.


So these people came together and basically joined in making one PowerPoint presentation and then presented to each other their knowledge of the environment.


And amazingly, we also had the Flatland organizers come in and tell us their perspective about the challenge, the environment and what's challenging and what's changed and so on.


This was just unprecedented for me, the amount of contributions there.


The first Town Hall is available if you join Discord.


It's linked there.


It's recorded.


You can still see it.


And as I said, there's still plenty of time to join the competition together with us.


Okay, next thing.


You've probably seen the SpineNet video and the thumbnail there was excessively beautiful.


So the story behind it is on the Discord server.


I've asked people to help me with this thumbnail, which was originally rather boring.


And I just kind of wanted to know which subtitle I should put there.


And then one of the Discord members, Lucas Ferreira, just gone ahead and drawn up this very beautiful image of a SpineNet robot that has the SpineNet as a spine.


And this is this kind of stuff is just awesome.


So again, big shout out to Lucas Ferreira and the absolutely amazing thumbnail that this has generated.


And also the contributions to anyone that comes on the Discord server and into the Beginners Question channel, ask some question and usually get some form of help.


Now that being said, please don't just come and we'll solve your problem like try to search for a solution before going into that group of very well meaning people.


Because, you know, if too many people just expect them to solve their problems, it won't be as well meaning anymore in the future.


Okay, so how does this channel go forward?


I want to make the content a bit more diverse and kind of branch out. And as I already said, the upload frequency will be sort of lower after my break and also during my break.


But I have some ideas of how to generate kind of more interesting content or different content.


So here are my ideas. And this is a list. And please tell me what you think of it.


And you can do this at this video and you can do this at any point, you can give feedback about what kind of videos you like, what kind of style of videos you like, anything really, I'm happy to listen to people and incorporate all the feedback that I can.


So I want to do some more channel updates, maybe more frequently, maybe once every two, three weeks, just to let you know what's going on, kind of what's going on with the channel, what's going on with the community.


This should be fun. So another idea I had is to look at kind of historical papers. And I think I got the idea from a comment in my comment section.


So shout out to whoever lifted me to that. It's a great idea to basically go back to historical papers, and just kind of see what people back then knew and didn't know and predicted and were right about.


And especially I wonder what kind of choices that they make that survive until this day kind of arbitrary choices that someone made in some paper that just stuck around.


It's interesting to see and there will be a series of kind of classical papers that I will extend from time to time. And I hope you enjoy that.


Also want to do more a bit of live coding videos. Lots of people have requested that I'm not the best coder in the world. But I have done my fair share of machine learning research, hands on, I have lots of more stupid ideas that might or might not work out.


And I'm very happy to implement them live. Then next thing I want to branch out in topics, maybe more exotic things. Causality is a big thing, quantum machine learning, whatnot, more practical applications, robotics control, also fairness, a lot of people, especially after the young look on video asked me to look more into the fairness literature.


I am naturally very interested in that and approach it from kind of a technical but also a societal standpoint. Throughout all of that, I would like to include the community more so you I don't really know how to do that properly yet.


So here's the first thing that I'm going to include you. If you have a good idea of how to do more community inclusion in the channel's content, please tell me because I think the community has lots of stuff to give. And it would be a shame if it were only me always doing all the things where other people would be much better at it.


Okay, the last question on this is a question that I get very often. And that is, how do I select papers, and there seems to be a misconception in this. So let me tell you, I select papers by what interests me. And that alone, if I make a video about a paper, it means that I found it interesting enough to read it.


And I found it interesting enough to make a video about it. Now this can be accelerated by sending me appropriate amounts of protein and carbohydrates. But generally me making a video is not an endorsement of a paper. It doesn't mean that this paper is the most important paper or influential.


It doesn't mean anything beyond I find it interesting. If I ever get sponsored deals, I'll let you know that a video is sponsored. And that's that I will not change how I select papers, I will not go by some kind of impact, I would like to branch out my content, I see the danger in only kind of covering what the big companies do.


But honestly, they do a lot of interesting stuff. They do a lot of stuff. And I'm constantly looking at research that's kind of outside the box, whatever is interesting, you know, that's what it's going to be, I will not start going by some impact factor.


And I will not start politicizing my paper review selection, any time soon, you know, I've already had multiple people, high profile people come to me and say, well, with your platform, couldn't you once a week, take a paper where the first author is an underrepresented minority and review that and, you know, I appreciate the sentiment behind it.


And I see I see where it comes from. But if you consider the practical implications of something like this, like I, I'd have to, you know, go through papers, Google, the first author kind of try to find a talk or a picture of them, and estimate whether the melanin content in their skin is high enough for this to qualify now and something like this.


And just the thought of this, how someone could do something like this and not start to vomit is just beyond me. I don't know what to say other than that.


So let me say this, if your thinking leads you to a place where it's necessary to treat people differently based on the color of their skin, you're wrong. Like, that's my opinion. But you're wrong.


The answer to bias cannot consist of more bias. That's that. I do not care how the person that wrote a paper looks like. If your papers on this channel, it means your work was interesting to me. And I hope that can be my contribution to making the community more fair and just.


Okay, last thing. Lots of people have asked me if I had a Patreon or something like this. And I've sort of resisted that kind of stuff until now. Mainly because I knew that the day would come when I reduce my upload frequency and I didn't want to kind of trick people into thinking that I was going to continue this forever.


Again, financial support is not my main goal here. And it is completely absolutely and utterly voluntary. And so I just want to have that out there. So I have made a Patreon page, I do have some reservation with respect to Patreon, because of free speech issues and so on.


So I've also made a subscribe star page, both are equal, both have equal tiers, all the tiers are equal, there's no option where it just where you could just put an amount, which I would like. So I just tried to make a bunch of tiers, all of them are equal.


So I had to ask myself, what do I give as a benefit? Because I don't want someone to have to pay for like extra content, because the entire goal of this channel is to educate people, including people that don't have money to go to good universities that might live in other parts of the world, where education is not as available, where resources are not as available to give extra content to people that pay seems to be so I thought, okay, what I could give to the people that are not as available.


And one of the things I would like to say to people that do support me on these pages is you will get a PDF of my scribbled one note document of the papers that are review. I mean, it's not very helpful because I mostly scribble and it's going to be like subdividing the pages weirdly, maybe it has more of a symbolic value.


But I'm not going to get into that in you know, at least there's something.


I have a bunch of crypto wallets. So if you'd rather want to use that support me, you are welcome to do so all the links are in the description of the video. Again, financial support. Very, very, very optional and very voluntary, though of course I do think anyone that does, I am also going to experiment with ads on the videos.


And as creators, we have kind of different options of which ads are displayed, and how often and so on. I find mid video ads annoying I find non skippable ads annoying and so on. I'm really counting on you here to give me feedback after various videos of how much the ads annoy you which ones annoy you which ones don't.


I'm really counting on you. Okay. Okay, last thing I am planning, planning on a line of merch, mainly because I think it's funny. But I don't know if that's going to work out but you know maybe if you have fun t shirt ideas or so just let me know.


That was the update. As I said, I probably won't be reading comments too much, but I will catch up after the break, and I hope you continue enjoying this channel, even with kind of the lower upload frequency and the new types of content that come in.


JOIN ME for the NeurIPS 2020 Flatland Multi-Agent RL Challenge!

Hi there, today I want to talk to you about something that's very near and dear to my


heart.


And that is the flatland environment.


Now the flatland environment is a train simulator that has been developed by the Swiss train


company.


And I ride the trains every day.


So when I heard that there is a new ribs challenge to use the flatland environment to make the


train system in my country better.


I of course was very excited to do that.


So out of purely egotistical reasons, I'm going to present to you the flatland environment.


And I invite you to join me in solving this as a group together.


So the plan is basically that we as a community sort of do this challenge and completely in


the open, with absolutely no aspirations of winning or doing well or getting any of the


prizes just for the fun of it.


And we'll see how far it will come together.


Okay, so let me demonstrate the environment itself.


So as you can see here, this is a visualization of the environments.


There are these agents in the environments and they have to reach certain goals.


And of course, they can't crash.


If you look here to the left, there's a bunch of them crashing right now, which is not good.


And your task is this is a multi agent reinforcement learning problem.


All of these agents have to reach their goal and as fast as possible without any collisions


along these tracks right here.


So you have basically have to specify for every single agent what their next action


at each time step is.


Now the simulator is completely given to you.


This is you can you can use it.


And basically, it's a planning problem for multiple agents.


So at each step, you have to decide, does the agent move up, down, left or right, depending


on whether they can do so depending on the tracks and whether something is in their way


or is not in their way.


And you know, every agent should reach their goal at the closest possible or at the shortest


possible time, of course.


Now there is this NeurIPS 2020 flatland challenge.


And basically, you can submit your solutions to their evaluator.


And there's a leaderboard and everything and I thought it would be fun to participate in


this.


Now I don't exactly know what the what's the exact connection to NeurIPS and so on.


But I don't care, honestly.


And this hasn't started yet.


The timeline isn't really open yet, but it will start soon.


But I think we can already start working on it.


So the plan here is basically to just kind of, I have no idea of traffic scheduling,


no idea, absolutely clueless.


But I know a lot about reinforcement learning.


And even though they say the challenge has already existed last year, in a very in a


slightly different form, I think it was just one agent instead of multi agent.


And they said, usually you have to combine the reinforcement learning with like some


traditional stuff in order to perform really well like screw that.


No, I'm totally up for that.


But it would be fun to just blast it off with RL and go there.


So here's my proposition.


I have opened a Discord server for you to join where you can join and basically people


can discuss solutions to this problem.


I'll make a GitHub repository in public where people can submit poll requests to and I'll


be sort of the merger and whatnot of these.


And we together sort of develop solutions.


Now my idea is that people would sort of independently try things and then kind of suggest things


and if they work, we can merge them and whatnot.


And there's just a lot of discussion in the Discord server.


I myself will not be like super active on the server.


It's meant for the community basically together to discuss things and whoever wants to do


that.


So I just want to make some things clear from the beginning.


I will be the dictator of this project, the 100% authoritarian, no compromises dictator.


If anything is supposed to make it be decided, I may elect to hold a vote and I may not.


If we win something, I'll decide what to do with it.


So just this because otherwise there's just trouble, right?


Are we going to win?


Probably not because anyone could just come to our GitHub repo, clone it and then tune


it a little bit more.


Right?


So I have no aspirations of winning right here.


So as I already said, I'm not going to be super active in this Discord.


It's meant as a method for the community among itself to communicate.


Third, if you decide to put in work, don't expect others to do so.


Expect nothing.


If the project doesn't work out, we scrap it.


If people get tired of it, we scrap it.


If there's some other problem, we scrap it.


No expectations.


Never get mad at anyone else for not doing as much work or anything like this.


This is purely you participate because you yourself want to learn something, want to


have fun.


And if someone else does the same thing, that's all the better.


Okay, I will have a mainly supervisory role in this in that I will look at things that


are happening and advise and occasionally I of course will participate myself.


So I hope the framing of this is clear.


This is not me throwing a hundred percent at this.


I just thought it would be cool to do something as a community together.


And this challenge, it seems like, you know, there are other challenges like MineRL where


everyone needs like a billion GPUs to even get competitive.


This seems like small enough that we could actually make a difference here and hopefully


do something very cool.


Alright, if you still want to participate, even though I really, really, really try to


talk you out of this right now, I will leave a link to the Discord somewhere in the description


and a link to the Git repo as well.


And I hope that some of you will be motivated enough to come join and have some fun.


Alright, I'll see you there.


An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Paper Explained)

Hi there, today we'll look at an image is worth 16 by 16 words transformers for image recognition at scale.


So this paper is a bit special. Andrej Karpathy tweeted this out, and I'm going to guess many of you have seen it already.


It's a paper that's under review at iClear. iClear, of course, uses open review.


So all these submitted papers can be seen and can technically be commented on. And as you can see, it's anonymous.


And good thing it's anonymous because the double blind review process relies on anonymity.


So we can really evaluate this paper, which is a very interesting paper at its merits without, you know, having a clue who would be writing something like this.


Now, out of pure out of pure randomness, I just happened to have this in my like, control C, control V memory.


I just pasted this here. I don't know why, but this is this other paper called Big Transfer,


general visual representation learning by Alexander Kolesnikov, Lucas Bayer, Siahua Chai and others of Google Research.


I've actually made a video about this. So if you're interested, totally not related at all.


I mean, yeah. So disregard the fact that the paper that we're discussing here uses a GFT 300M data set that is not available to the public, only to Google.


That is and actually this other paper also trains on that. Disregard that also largely disregard the fact that their model is called VIT,


while the other papers model is called BIT. Disregard the fact that they train on the exact same data sets, as you can see right here.


I mean, this here is ImageNet then C4100 pets, flowers, and the VTAP, VTAP, this visual task adaptation benchmark.


I've done a video on that too, by Google. But they do have actually the ImageNet reel here, which is a just a set of new labels for ImageNet,


which comes out of a paper by Google with largely the same authors as this paper.


I mean, disregard the fact that the color scheme for the VTAP evaluation is exactly the same as is the histogram plotting.


And of course, we don't even want to bicker about the plotting style with these bubble sizes.


And so I mean, anyone could do this, anyone, anyone in the world could just randomly have this much overlap with these models.


And of course, anyone just has the money laying around to train on 2.5000 TPU v3 days.


And, you know, compared with 9.9000 TPU v3 days for the BIT.


I guess you could just pick those numbers out of the paper. But what do I know?


So, no, don't worry, peer review is totally fine. Like, like, I mean, yeah, so I hope I've made my point.


This is by these people. And, you know, people say, you know, we need anonymous on archive because the danger is that people upload their paper in archive and then we can see who they are.


I think this should prove to anyone that an anonymous archive is like, it's the crappiest.


Why? Why? Like, why would you ever work against the core incentives of people?


Like, clearly, these authors have an incentive to make known who they are.


And clearly, we as readers have an incentive to figure it out.


And to completely work against these incentives just seems so, it seems dumb, it seems counterproductive, and it doesn't work.


As you can see, what do you want to do? Standardize the plotting styles, standardize everything, standardize the citations.


I mean, come on here. You go like, when we compare, oh, no.


Where is it? When they compare against things, they say, oh, our first point of comparison, our first point of comparison is the big trend randomly just big transfer by these authors that we have no relation to maybe or maybe not.


It's, it's ridiculous. You can't shield this, this fake anonymity.


This is actually counterproductive, and it only helps the big labs, the this anonymity criterion.


All right, let's actually dive into the paper after this rant. Well, yeah, don't worry.


Peer review, very pristine, very good, very anonymous, double blind, for sure.


So the paper says, while the transformer architecture has become the de facto standard for natural language processing tasks, and we know this, you know, this is from the first attention is all you need paper to things like BERT, GPT, GPT2, GPT3, transformers have revolutionized NLP.


Say its applications to computer vision remain limited. Envision attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks, while keeping their overall structure in place, which is correct in computer vision,


convolutional networks have been so incredibly successful since Alex net and then of course, resnets being the major contributor there. I mean, even this big transfer paper right here, all it does is scale up resnets and then feed in more data.


So CNNs are extremely, extremely powerful in computer vision.


We show that this reliance on CNNs is not necessary. And a pure transformer can perform very well on image classification tasks when applied to when applied directly to sequences of image patches.


And they go on saying that they outperform CNNs while requiring substantially fewer computational resources to train. Well, you know, substantially fewer in these regimes of 1000s of TPU days is something that is a bit ironic, honestly, but, you know, it's, it's, it's, it's pretty cool.


So what's the deal with transformers and images? classically, transformers are of course, things models that operate on the sequences specifically, actually, they operate on sets.


So you'd have a set of words, which you can characterize as tokens, which I'm just going to characterize as, as bubbles, and then the transformer would somehow take all of these in and do something with them. And something in this particular case, is attention, and attention is a quadratic operation, which basically means that you have to calculate the pair wise inner product between each of these between each pair of the


of these bubbles, which becomes a very, very large task very quickly. You see, I even have trouble drawing, I think I drew this twice. However, this this already with five, it is many, many, many interconnections. And you can imagine that if you are in NLP and have a paragraph, that's maybe 500 tokens long, you need 500 squared connections.


So this one thing is the limitation of transformers, they work really, really well for NLP. However, they are limited by the memory and compute requirements of that quadratic attention. Images are therefore much harder for transformers, because an image of course, is a raster of pixels.


And there are many, many, many, many pixels to an image, right? So usually, even in image net might be image net counts as a large images in computer vision applications, but even the image net, they're like, what 250 by 250 pixels, which are small.


By human standards, we are used to looking at, I don't know 1000 or 2000 pixel side length on a regular basis for it to be clear. I mean, even the rasterization of this PDF, you can see is you will recognize it as blurry. And that's that's way, way more resolution than image net images.


So the just the rasterization of images is a problem in itself, even for convolutional neural networks. But if you want to feed this into a transformer, you have to think that every single location here, every single pixel has to attend to every single other pixel, which the image itself is 250 squared big.


So the attention will cost you 250 squared squared, which is impossible in current hardware, even for Google, right? Maybe they can do it. But so people have resorted to other things, doing things like only local attention.


So only attending to the kind of area around them, which of course, is the the foundational motivation behind convolutional neural networks is that you learn kernels that are local, and then you you kind of slide them across and over the layers across the layers once once you go from layer to layer.


So the first layer, this part might attend to like a cone around itself. And this part might attend around a cone around itself. But then the next layer, the thing that attends in the same cone will have a larger effective receptive field, right?


So in this, the receptive field grows by depth. However, transformers are able to attend within a single layer to everywhere. And this paper solves this by not going into direction of, hey, let's do local attention over pixels.


But they say, let's do global attention by simply going over image patches. So they divide the image into these patches, as you can see here, and one patch is in this case, something like 16 by 16.


They unroll these patches into a sequence, which is a in first instance, it's a set. They combine this with a positional embedding. So the transformers naturally, they have no idea what what is where it's not like the transformer in a way is a generalization of an MLP of a feed forward network in a feed forward network.


What you have is you have you have just you have connections between these different inputs and outputs, okay, and these are fixed. So the this node here will always attend to this node here with the weight that's specified by this particular connection.


However, in a transformer, this W isn't a fixed number in a transformer, this W is computed on the fly. So and that's dependent on what these exact nodes are.


And therefore, the M while the MLP knows where information comes from the transformer doesn't the transformer computes on the fly and therefore is permutation invariant. And that's why a lot of applications add to the inputs these so called positional embeddings, where they simply say, look, this here, this here is patch number one, this here is patch number two, this here is patch number three.


And you can do this in a sophisticated way in images is specifically you can say this is position one, one, this is position one, two, one, three, then you go on by saying this is two, one, two, two, and so on.


Now they in the paper claim that they've tried this and it doesn't help. It's it's much easier if they just say this is one, two, three, four, five. And the these are learnable embeddings. So the the you don't actually feed the number one.


But what you have is you have a table. And the table will say, we'll have these indices 12345 and so on. And each one is associated with a vector. And these vectors are learnable parameters. So whenever you say this is the first patch, what you actually do is you go here, you grab the vector to the number one, and you put the vector along, sorry, up here along with the patch into the transformer.


Now the patch itself is still a small image, right? It's a 16 by 16 image. So you have to get that somehow into a form where the transformer can understand it. One way of doing it, of course, is simply to unroll it and say, gee, this is a 16 by 16. What's what's 16 by 16? It's like 256.


I think so. I don't know.


I guess to its 250. It's a 256 dimensional vector. However, they find that if they first put that through a linear projection, that helps before they put it into a transformer. So there is one single matrix. And this one single matrix is called E.


In this case, embedding, haha. They take a patch like this, they unroll it. So here you have the image, you unroll it into a big vector, you multiply that vector with the embedding matrix. And that's what goes into the transformer, along with the position embedding. In this case, we have position embedding, whatever seven, you go grab seven right here, you concatenate that here or add it to the position embedding.


And you put that into the transformer. And from here, it's a standard transformer. This is just out of attention is all you need standard transformer. And what you do is you have a special input. This is a learnable embedding. It's like the BERT embedding, the CLS embedding, and you take the output of this thing finally, in order to classify and this is just a standard classifier.


So it's really simple architecture, except for the bottom part here. It's a transformer, one of the inputs is decided to be special. That is not associated with any patch, but is a learned input, the output of that particular dimension or of that particular input you take as a classification.


Okay, so there are more outputs right here, but they are discarded, of course, because so in the last layer, they're actually not even computed, I would guess what in the last layer, only this thing is computed. But in the other layers, everything is always computed, right? So you have many, many transformer layers in here.


Transformer layers are of course, made up from these blocks right here. Sorry, not the embedded patches, but this thing. Okay, and you see the multi headed attention, that's the expensive operation.


So the paper completely completely discards the notion of convolutions. They have a variant where they, I believe, replace this patch embedding here with a convolutional embedding.


But I don't I don't think it helps much. They really want to show that convolutions are necessary. And I don't want to go too much into the details of the paper, because also it's it's also subject to change, you know, an open review, you can revise it and so on.


But the experiments show, as you can see right here, that this visual transformer, this vision transformer outperforms the the other like the convolutional networks by a pretty significant amount often, like sometimes small, but sometimes also large, and costs less to train than these big convolutional networks, at least of this one other paper, right?


So it costs less to train. Here you see, of course, if you go 16 by 16 patches, then that means you will have so if you divide your image into patches that are themselves bigger, that means your your sequence of patches will become smaller and therefore your computationally more efficient.


If you go with 14 by 14 patches, but also the the age, I believe is more layers. There is actually a table up here. Yep, so the huge has 32 layers. And that is has double the amount of parameters, all of that gives you a higher computational requirement still lower than the big transfer paper.


Okay, so the idea here is you train on these big data sets like this JFT data set. So you pre train on that this is a weekly labeled data set of 300 million images, and then you transfer to the other data sets, which just happened to be the same data sets that this paper used plus the other data set that the same authors created after this paper came out.


Don't worry about it. Okay. They also test on this visual task adaptation benchmark. And you can see that especially specifically in these natural images subclass, they actually both of these models make gains, but then overall, the visual transformer outperforms the conv nets.


So what's the what's the deal here? What's the deal with transformers? And that's something I want to talk about, I don't want to go too much into the rest here. Of course, you can visualize the attention, you can see it's doing something sensible. And you can visualize the positional embeddings that are learned, which is pretty interesting.


And you can see that the positional embeddings come out pretty sensible, you can see where they pay attention to mostly and the seems like this positional embedding, it largely recognizes where it is in the image, even though you never tell it, you simply let it learn.


But it it relates to other positional embeddings that are in the same row or column, largely. And that's all sensible, you can see the filters it learns. So this is analogous to visualizing what convolutional networks learn. And you can see it does something sensible, it does something that we're very much used to, if you look at conv net visualizations, you'll see exactly filters like these.


So it learns, let almost like the same thing as convolutional neural networks, right, but it's not specifically programmed to do so. Also, you can see as you increase the depth of the network, the mean attention distance, so the distance over which the attention goes increases, and from like the middle of the network, you pretty much have global computation.


And this is also like, this is almost like the drawing I made of the CNN, right, where you you would have the different heads. So some heads would immediately at the beginning, go out a CNN, in this case would look like a line, a CNN would look like a line that's like this.


And the additional benefit you get in the transformers is, of course, that at the very beginning, you can already pay attention to things that are very far away. You cannot do that with convolutional networks, or when you use local attention. So all this branch up here, that's kind of the game that transformers can make, they can attend to very far away things right at the lower layers.


So, so what's the deal with transformers, it seems like transformers are coming for everything. So first, they, I guess they, they were attention was introduced in LSTMs. So LSTMs with attention were the cool thing to do. And I think still are in some places in NLP.


But then transformers completely replacing LSTMs in NLP. And now transformers are coming for vision, they have been paired with vision, as the introduction here said, but now they are replacing convolutions, sorry, they've been paired with convolutions, now they're replacing it.


And here's what I what I think about this. So what do you have in LSTMs and in convolutional neural networks were good inductive priors. So, technically, if you think about it, if you have something like an MLP, a feed forward network, like we looked at here, the, the, the notion should be that it could technically learn any function, right?


A feed forward network can technically learn any function. But it's it's kind of unstable, and so on, you know, if you shift by a pixel, all the inputs are all weird, and so on. So a convolutional neural network for images seemed pretty good, because it has a good inductive prior, the good inductive prior is this is that probably what one pixel cares about is its immediate neighborhood.


And then what that neighborhood as a whole cares about is its immediate neighborhood, right? So that's sort of how we look at images like you integrate over small regions, and then you connect the regions to each other and so on. So this is a very sensible inductive prior for images, as well as the LSTM for language, if you have a language, right, having an LSTM, having the inductive bias of let's first process this thing, then, you know, remember some general


woo woo woo state, then in in go to this thing, and then incorporate that into our memory what we already know, right, then that kind of updates our latent belief. And then we go to this thing. And again, we incorporate that that's how we read. And that's that's how we do it. And so the inductive prior of this model is actually very, very solid.


And inductive priors, or inductive biases, the name already contained it, it's a bias, we bias the model towards solutions that we think in general are relevant are useful, right? We, we tell the model, look, we know you could learn everything from data, no doubt about it, we have the theoretical results, you could do that.


However, you don't have enough data. And we want to make it a bit easier for you. So we tell you that certain things like CNNs, like convolutions, generally tend to be useful. So we restrict the model, and we bias the model towards a certain solution or LSTMs. These are bias biases that we introduce in the class statistical sense of bias, right? So these are the


biases that help the model become very good at task. However, now we are in a regime where we have lots of data, and lots and lots of data. And we know bias, why is it called bias? Because it will buy us our estimator, our estimator will not be the perfect expect expected value matches the actual underlying thing.


estimator. Therefore, we know that if we have enough data, a biased model will perform worse in the end than an unbiased model. It's only in the not enough data limit that the bias model can perform better, at least, I mean, I'm simplifying here. But now transformers come along and transformers are basically transformers aren't an another architecture transformers are basically


a general compute thing. They're even more general than MLPs. Like people think that MLPs like this MLPs are the the on most unbiased thing ever because everything's connected to everything. No transformers are actually more general, because not only is everything connected to everything, but these connections are always computed on the fly. So a transformer is like the main thing that's connected to everything. So if you're a transformer, you're a


general, because not only is everything connected to everything, but these connections are always computed on the fly. So a transformer is like the most general thing there is in terms of deep learning that we have right now that we can train. Yeah, I'm making bold statements, but that's how I think about it. So the the if the CNN and the LSTM are more specialized MLPs than the transformer,


is a less specialized MLP. And therefore, it's not necessarily in the architecture of the transformer that makes it so special. It's just the fact that it is a general computer. And if we we are now able to feed enough data into it, such that it can actually learn the things and it can, it can not only can it learn the useful biases, right, we give, we give useful biases. And you can see it learns


the same thing as a convolutional network or very similar things, right, it learns these filters, and so on, that before we would have, we would have given this thing here as like a wavelet filter, that was our in even before CNN, we, we fed in like wavelet filtered things, and this thing would be on top of the list. So it learned it can learn that from scratch. But probably this thing is not exactly a wavelet filter, it's actually something that performs


slightly better, right, that we couldn't have come up with as a as a bias to build in. And that's why it works better. Because it can learn almost the same things, but it can do so a bit better because it has that much data. So I believe the world is still open transformers aren't aren't the end transformers are simply one general computer, there can be others, there can be something even more general than a transformer. And their world is still


wide open to build in inductive biases that are actually better than CNNs or LSTMs also to build inductive biases in transformer. Or if you go in the other direction to alleviate because what you see right here and in the formula, you see this pretty well. There are inductive biases in the transformer. And if I had to guess, I would say the ones that are to go next are the skip connections in here. Now the skip


connections are very important for us to be able to train these architectures. Because if you read the resonant paper, the residual nets paper, that's kind of where the gradient flows back the rationales that you can go very deep and each layer only has to kind of calculate the delta that you have to do to the input instead of transforming the input as such and so on. It makes a lot of sense, but it is a strong


inductive bias. And it pulls through all of the layers as you can see here, right? All of like the skip connections is pulled through all of the layers. This is a very strong inductive bias. And we tell the network maybe it's sensible if you only calculate the diffs in each layer. If I had to guess, this is one of the next big things to go. If we have yet an order of magnitude, more big data sets, and we figure out how to train


big networks without these big skip connections. Alright, so it's not like, as I said, it's not like transformers is like the very, very good architectures in the same sense that LSTMs and CNNs are very good architectures. It is the fact that transformers are so general, they are actually able to make use of the big data that we just now have that we didn't have before and of the big compute, such that these inductive biases of the old model


I'm out of Academia

Howdy diddly doo. Hi everyone. If you're wondering what the ridiculous thing on my head is, then


I'm out of Academia

that is my official graduation slash successful defense hat. I'm not yet allowed to technically


I'm out of Academia

use the title doctor but let's be honest who gives a crap anyway titles. Um, I'm a huge


I'm out of Academia

fan of this hat. My lab mates made this for me and I thought I'd share a little bit what's


I'm out of Academia

going on right here. So the everything on here is kind of like a meme and therefore


I'm out of Academia

that that has to do with me in some way. First of all, you see my name, which is made up


I'm out of Academia

out of letters of our lab homepage picture, which is like the cringiest lab homepage picture


I'm out of Academia

you've ever seen, where everybody's just kind of made to hold the letter. And it's just


I'm out of Academia

it's very I love cringe by the way cringe is the best there's obviously the meme of


I'm out of Academia

me being a youtuber and having followed or not followed my own advice. There is me as


I'm out of Academia

Schmidhuber in Schmidhuber attire. I went to his talk dressed in his style to to honor


I'm out of Academia

him. There is two plus two equals five which I made an extensive video about. I made the


I'm out of Academia

first neural network in Minecraft. Not technically true. I made the first analog neural network


I'm out of Academia

in vanilla Minecraft that could also do back prop and weight updates. It's very specific


I'm out of Academia

but it's the first there are the hugging face. That's a transformer. I don't know if you


I'm out of Academia

can see this. That's a I don't know which one that is. That might be a Decepticon. There


I'm out of Academia

is the Asphalt set which is my kind of side occupation as a fitness instructor. There


I'm out of Academia

are the sunglasses. I also like cats. There is I'm always chilling for for Vin as an editor


I'm out of Academia

though I use neo Vin. Yeah, also the pronouns, you know, gotta have them. I'm you know, happy


I'm out of Academia

they're here. There is crypto because I'm also always chilling for crypto sometimes


I'm out of Academia

for the wrong ones. But you know, you can't always win. There is cheese and chocolate


I'm out of Academia

which is my standard lunch depending on the season. If I'm doing keto, it's no chocolate


I'm out of Academia

but you know recently you just I'm Swiss after all. There is Yeah, there is the skeleton


I'm out of Academia

and the sword from Minecraft again due to my extensive research into the technicalities


I'm out of Academia

of redstone. And Ili Cafe five years five years of that coffee will you know get you


I'm out of Academia

through a PhD hopefully. There are the tweets who that got me into trouble. Yeah, there


I'm out of Academia

is there's also trigger happy Gandhi asking you earn a decade just for a PhD. Yes, yeah,


I'm out of Academia

we are like the best paid PhD students on the planet. It's it's fantastic can recommend


I'm out of Academia

there is a deep judge logo which is the thing I'm going to do next which is illegal tech


I'm out of Academia

startup. If you if you need legal tech, please buy our stuff. And so on the inside, you'll


I'm out of Academia

see Joe and obviously the Donald. Oh, I'm gonna have to reattach that again. Yeah, so


I'm out of Academia

so because I lost a bit of money betting I bet on the you know the really old dude and


I'm out of Academia

it turned out the really old dude one. So I lost. Yeah, so this is this is sort of a


I'm out of Academia

bunch of memes throughout my PhD. I'm gonna reattach the VIM. You know, you don't want


I'm out of Academia

to that dropped. So yeah, I you know, thanks to all my lab mates. This is this is really


I'm out of Academia


Python 3 Programming Tutorial - For loop

Hello guys and girls and welcome to another Python 3 tutorial.


This tutorial is going to be covering the for loop.


The for loop is used for pretty much the same tasks as the while loop.


They're completely interchangeable.


Typically you're going to see the while loop being used for...


It sounds kind of funny, but I suppose I would say finite or infinite tasks.


The for loop is usually used to be a little bit more variable for uncertain kind of time frames and stuff like that.


But that said, the for loop can actually be used for the same tasks as the while loop.


The while loop can do all of the things a for loop can do.


They're completely interchangeable performance-wise, action-wise, and all of that.


For this reason, I will just say that it just comes down to personal preference.


I prefer the for loop myself. A lot of times the for loop is much more condensed or condensable than the while loop.


But again, it really just comes down to personal preference and it just doesn't matter.


Anyway, what I'd like to do now is show you guys some examples of the for loop and where you're going to most often see it.


Because some people really, myself included, I do actually use both.


It just depends what the situation is.


Anyways, let's go ahead and get started.


First, what we're going to want is generally you're going to see the for loop being used to iterate through a list.


What iterate means is just go through a list.


Let's make an example list.


We're going to name this example list.


We're going to encase this list.


I'm just going to type in random values. You can do the same.


You can try and copy my values if you want, but it's not necessary.


I'm just putting some values in here.


In a previous tutorial with the variables I was showing you guys, that if you add multiple variables here, it's going to unpack.


It's going to try to unpack all of these values.


If you didn't have enough on either side, if they weren't equal, it would throw an error.


But if you're just assigning a single variable here, this is totally fine to do.


It's going to assign this entire list to this variable.


If you did example list var, that would throw an error.


You would need as many variables as here because it's going to try and unpack the list.


Just as a quick aside.


Now what we're going to do is we're going to say for x in example list, what do we want to do?


The way to think about this is basically for each i.


Maybe what I'll do is actually to make it a little more understandable.


For each number in example list, what do we want to do?


What it's doing is it's creating a variable here.