Also, every time there's a new major version, you'll rename it to something more ridiculous or,


Full Self-Driving is HARD! Analyzing Elon Musk re: Tesla Autopilot on Lex Fridman's Podcast

memorable and beautiful. Sorry, not ridiculous, of course.


If you see the full array of neural nets that are operating in the cars,


it kind of boggles the mind. There's so many layers, it's crazy.


I mean, what is he actually saying here? It's hard to decipher Elon because obviously he's not,


you know, a deep learning engineer. So he sort of probably gets like the pitch from from Andre,


and you know, some diagrams or something like this. But as of now, we don't know if there are


many neural nets, but it's unlikely because he says like, it's mind bogglingly many,


and you'd have to sort of train all of them. I couldn't really imagine how you'd put like


mind bogglingly many neural networks into a single neural net.


So I'm going to guess that they have a couple. And these are just kind of big and complicated.


And that's exactly what we saw in in Karpati's talk when he explained how they how they go vision


only and so on. If you haven't seen this, watch my analysis of that. He's about to explain a bit


more in depth of what's going on.


We started off with simple neural nets that were basically,


image recognition on a single frame from a single camera, and then trying to knit those together


with it, you know, it with the C, I should say we were really primarily running C here because C++


is too much overhead, and we have our own C compiler. So to get maximum performance, we


actually wrote our own C compiler and are continuing to optimize our C compiler. So we


actually wrote our own C compiler and are continuing to optimize our C compiler for


maximum efficiency. In fact, we've just recently done a new river on a C compiler that will compile


directly to our autopilot hardware.


If you want to compile the whole thing down.


I mean, he's going to talk about two things kind of interleaved right here that have on the surface


not too much to do with each other. So apparently, there is a C compiler that compiles directly to


the hardware, which makes sense, right? These cars have the property that you have to be super duper


efficient than power saving and whatnot. And you know, running Python on top of that might just,


you know, the overhead of that might just be too much, you can in fact, save a lot of energy,


a lot of time and so on by going building a compiler that uses the hardware as optimally


as possible. Now that being said, this has little to do with, you know, how you build


the neural network system other than the neural networks will be faster if you compile them down


correctly.


And so there's actually a lot of work done by some very talented software engineers at Tesla that


at a very foundational level to improve the efficiency of compute, and how we use the


trip accelerators, which are basically, you know, doing matrix math dot products,


like a bazillion dot products. And it's like, what are neural nets? It's like,


compute wise, like 99% dot products. So,


yeah, I mean, he's, he's obviously correct, right here, though it has to be said, you know,


for anyone who's listening to this, your neural network isn't slow because because you don't have


the right compiler, it is true that if you do it correctly, you compile your network down to like


a format that is optimal for some hardware and you run it with you know, the correct libraries and


and you set up everything correctly, you can probably get like, maybe if you did, if you did


it terribly wrong, and then you do it terribly right, you can get up to a 10x speed up, I would


guess maybe, you know, 5x 10x speed up something like this, best case, however, usually, usually,


the first thing you should investigate is whether or not the architecture you're using is the


correct one, you can get like many, many more times a speed up by simply changing the architecture to


something more appropriate. So Elon says this here, because obviously, this is the last step.


And you know, they need to they need to get every every millisecond they can out of these systems.


But just for most people listening, this is sort of the sugar, the icing on the cake,


you should first care about the cake and try to make your architecture, you know, more optimal,


maybe use less layers or anything like this change from this operation to that operation,


analyze your bottlenecks. And only once you have everything through and you have the exact model


you want, then you can care about doing all the engineering things. One of the things we're


moving towards now is no post processing of the image through the image signal processor. So


like, what happens for cameras is that almost all cameras is they there's a lot of post


processing done in order to make pictures look pretty. And so we don't care about pictures


looking pretty. We just want the data. So we're moving just roll photon counts. So the system will


like the image that that the computer sees is actually much more than what you'd see if you're


represented on a camera, it's got much more data. And even in very low light conditions,


you can see that there's a small photon count difference between, you know, the spot here and


that spot there, which means that it can see in the dark incredibly well. Because it can detect


these tiny differences in photon counts, much better than you could possibly imagine. So,


I mean, that is, again, like that is a third issue next to the, that the C compiler, and what the


neural networks do is essentially saying that if you remove the post processing within the camera


sensors that are usually built into, let's say, cameras that you could buy on the market, then you


get the raw data. And since you don't have to look at the pictures, the raw data is much more useful


than the post process data since it's a machine anyway, that analyzes the signal, and therefore,


you might as well make it machine friendly. I think it is a good lesson for maybe other fields


as well to think about, you know, what parts of the pipeline are just there to make it, you know,


because because humans are involved and try to remove those. But, you know, it doesn't really


add to what's the what's the deal with the neural networks, which I think was the original question


here. And then we also save 13 milliseconds on a latency. So from removing the post processing


an image? Yes. Yeah, it's like, because we've got eight cameras, and then there's roughly, I don't


know, one and a half milliseconds, also, maybe 1.6 milliseconds of latency for each camera. And so


like, going to just basically bypassing the image processor, gets us back 13 milliseconds of latency,


which is important. Yeah, I think this, you know, besides getting the raw data, this is also, again,


they need to squeeze out sort of the last mile here, or the last milliseconds here. And this is


another thing they they can practically do. So getting rid of jitter is extremely important.


And that affects your control decisions and all those kinds of things. Okay.


Yeah, the cars is going to fundamentally maneuver better with lower jitter.


The cars will maneuver with superhuman ability and reaction time much faster than a human.


I mean, I think over time, the autopilot full self driving will be capable of maneuvers that,


you know, are far more than what like James Bond could do in like the best movie type of thing.


That's exactly what I was imagining in my mind. As you said, it's like impossible maneuvers.


It's like an impossible maneuvers that a human couldn't do, you know, so


Well, okay, it's, it's, it's, it's two things, impossible maneuvers are impossible. And things


that humans could do, you know, are things that humans could do. I also, I have no doubt that


at one point in the near future, self driving cars will be able to do things that humans couldn't do.


The question is more, are there going to be things that humans do that the cars couldn't do,


right or can't do, because that's the actual gap you're trying to close. Look at Boston Dynamics,


or so if you hard code stuff, and you have extremely, extremely good sensors and actuators,


you can do many things that humans couldn't do. But on the other hand, you know, it's the things


that humans can do that the machines can't. And those are the problem.


Well, let me ask sort of looking back the six years looking out into the future,


based on your current understanding, how hard do you think this this full self driving problem?


When do you think Tesla will solve level four FSD?


I think Elon gets asked this question every year. And every year he says next year. So


I mean, it's looking quite likely that it will be next year.


Tada. This is the thing with Elon Musk, he always promises things like next year or


in ridiculously short amounts of time. And I wonder how long it's going to take for people to


to just, you know, stop believing him, I guess many people already did. But it's still, you know,


a thing to consider that on one hand, obviously, if you do it too much, then people are simply going


to say, Oh, well, probably in five years, if he says next year, but on the other hand, he's also


able to sort of it's a motivating thing. It's it's a cool thing. It it drives momentum. And


that itself accelerates the development of these things, people being ready to just flip on a beta


version, and so on. It's a bit insane. But I do think his optimism and a little bit salesmanship,


also a lot of benefits besides the obvious negatives.


So the interventions, you know, per million miles has been dropping dramatically at some point, the


and that trend looks like it happens next year is that the probability of an accident on FSD is


less than that of the average human and then and then significantly less than that of the average


human. So it certainly appears like we will get there next year.


There's a lot of hedging going on here. But you know, you can this is this is actually a nice


method, I think of of making these types of predictions, you see that the rate of disengagements


is dropping at a certain speed, you can extrapolate maybe a little bit and say, Look, you know, here's


going to be the sort of threshold where we're better than a human. I think that's a quite a


sober analysis, if done correctly. And I also think people who are, you know, it's obviously good to


be skeptical of, you know, fully self driving systems. But on the other hand, you also have to


think if they're a lot better than humans, it makes makes total sense, right? That's the


sense, right? It also makes total sense to have them and not engage them all the time, right? There


might still be situations you want to drive yourself. The question is a little bit, can you


just continue the trend? Or is there a sort of an okay, you solve the easy problems. And that is


what makes the rates of disengagements go down now. But now come the more and more hard problems,


and sort of it gets exponentially harder to continue that trend, in which case, we're not


going to be there for a long time, then there's going to be a case of Okay, well, we now have to


prove this to regulators and prove it to you know, and we want a standard that is not just equivalent


to a human, but much better than the average human, I think it's got to be at least two or


three times higher safety than a human. Yeah, probably more like 10. Like knowing, you know,


no regulators and and how the public perceives these types of things. Of course, right now,


they're they're cool. But then it's really easy to publicize in a few accidents that few stupid


accidents that happen if you build machine learning systems for the real world, they are


going to make stupid mistakes, it doesn't matter how accurate they are, on average, they're going


to make stupid mistakes that a human would never do. And people are just going to point at it and


never forget that one instance. And I think it's pretty easy to sort of scare people publicizing


those kinds of things. And therefore, yeah, you have to be like massively better than humans, I


agree here. There is a some fundamental like leap that really deserves the 11. I mean, that's a


pretty cool number. Yeah. 11 would be a single stack for all, you know, one stack to rule them all.


And but there's just some really fundamental neural net architecture changes that are that


will allow for much more capability. But, you know, at first, they're going to have issues. So


like, we have this working on like, sort of alpha software, and it's good, but it's


it's basically taking a whole bunch of C C++ code and and leading a massive amount of C++ code and


replacing it with a neural net. And Andre makes this point a lot, which is like neural nets are


kind of eating software. So it's interesting what Elon says right here, this upcoming version 11


of the Tesla software seems to have kind of a rewrite in what he calls the creation of the


vector space. And specifically, he says you replace a whole bunch of C and C++ code with neural


networks. And I guess what what that means is that they used to have certain heuristics for


what he calls creating the vector space, right. And remember, creating the vector space means


seeing and understanding so what objects exist to where are they? How are they moving and so on. And


you want to get that out of your your cameras and whatever other sensors you have. So it seems like


until now, they had a bunch of neural networks that were would do you know, their their stuff,


I can imagine they had maybe single frame neural networks or kind of short frames, one after another


neural networks that would recognize sort of bounding boxing the objects in the image. And then


they would use sort of an algorithm heuristic algorithm that they wrote themselves to stitch


that together over time, maybe they use algorithms to do some kind of inferences, like what he


mentioned with the object tracking, and so on. And it seems to be that what they want to do is just


end to end train one big neural network that just does it all, you input all of the sensor data,


let's say from, you know, not only just right now, but you know, from the from the recent past,


you just input it all in there, and the neural network will spit out this finished vector space,


this finished scene understanding graph. And this obviously can see where it comes from. This has


been the story of deep learning so far, replacing more and more classical heuristics with an end to


end learning system. And it also matches exactly with what Elon is saying, namely that right now,


it doesn't seem to work quite well yet. But in time, it will get there. And again, this has been


the story of deep learning in pretty much everything we've tackled since the beginning of


deep learning, and to end systems ultimately came to beat the heuristic systems. But it takes time,


it takes work, it takes data, obviously, massive amounts of compute, you know, over time, there's


like, less and less conventional software, more and more neural net, we're just a little software,


but it's, you know, still comes out the lines of software, but let's more more neural net stuff,


and less, you know, heuristics, basically, if you're more, more, more matrix based stuff, and


less heuristics based stuff. So by the way, the reason why this is the case, the reason why it


works to replace heuristics with neural networks with data driven systems is that the world is


always more complicated than you can encode in any heuristic. That's why we use machine learning in


the first place, because we can't just program the algorithms that do image recognition, or speech


recognition, or whatnot. So the only representation of this really complex world, like the actual


underlying world that is so complicated, is the data. And therefore, our best chance to create


systems that deal well with the world as such is systems that actually learn from data from the


real world. And that's why it often works to replace the heuristics with data driven systems,


if you have the data, and if you have the compute, which Tesla obviously does,


we call it the giant bag of points. Yeah. And it's like, so you got a pixel and, and, and


something associated with that pixel, like this pixel is probably car, the pixel is probably


lane line, then you've got to assemble this giant bag of points in the C code and turn it into


vectors. And we're just a pretty good job of it. But it's, it's a, it's, we want to just,


we need another layer of neural nets on top of that to take the giant bag of points, and distill


that down to vector space in the neural net part of the software as opposed to the heuristics


part of the software. So the translation of this is probably, if I understand Elon correctly,


what they were doing so far is sort of semantic segmentation or pixel based pixel labeling,


I can also imagine that they estimated things like depth maps and so on just from pixels. But then,


as I said before, it was heuristics, it was sort of classical algorithms. And these aren't, I mean,


classical, these are advanced algorithms, right, that take point clouds that take sort of


segmentation maps and depth maps and all of that and turn them into objects. These are mostly


heuristic based, but very sophisticated algorithms, but it is clearly a good or a,


let's say a modern move to ditch all of that and also teach the neural networks to just handle it


until you have the semantic result that you want, namely the space of objects,


the scene understanding graph. It's really outputting proper vectors to the C, C++ control


code, as opposed to the sort of constructing the vectors in C, which we've done, I think,


quite a good job of, but it's, it's a group kind of hitting a local maximum on the how well this


you can do this. So this is this is really, really a big deal. And just all of the networks in the car.


By the way, whenever you hear him talk about C and C++ code, just replace that with human human


authored code, right? The difference isn't necessarily the language you use, the difference


is more like who writes the code. And when he says C and C++, it's humans, very smart humans,


but still humans that write the code out of, you know, their thinking. And whenever he says neural


networks, it's some sort of a data driven systems, which obviously human author in the first place,


but probably also is is as well implemented in in C and C++. So the training, the amount of work


done with like, we've written all this custom software for training and labeling, and do auto


labeling, auto labeling is essential. Because especially when you've got like surround video,


it's very difficult to like label surround video from scratch is extremely difficult.


Like take a human such a long time to even label one video clip, like several hours,


or the auto label it, basically, we're just apply a heavy like heavy duty,


like a lot of compute to the to the video clips to pre assign and guess what all the things are


that are going on in the surround video. And then there's like correcting it. Yeah. And then


all the human has to do is like tweet, like say, you know, change, adjust what is incorrect. This


is like, increase increases productivity by effect 100 or more. Yeah. So you've presented


that. I mean, we've we've discussed this in the last video that I did about Karpath. He's


talk and this to me is is, you know, I think too few people are currently doing something like this.


Essentially, it's active learning, right? It's sort of if you're not sure about something,


ask the human, it has a slight twist on it in that they probably always ask the human, but they


suggest a label, which is super powerful, especially in something like semantic segmentation, where


you need to annotate every pixel or you need to place bounding boxes around the human. And


you have to place bounding boxes around many objects. It's really different if you simply


have to check and adjust a little bit versus if you know there's there's a data point and you


have to place the labels yourself, I think we're going to see quite a bit more of that in sort of


the the near future. A lot of people are already doing something like this. But I think still too


few are so quiet in Tesla's primary mission direction of accelerating sustainable energy. But


it is a an extremely useful thing that we can do for the world, which is to make a useful humanoid


robot that is capable of interacting with the world and


All right, the rest of them talking about AI is talking about the Tesla bot, which is a bit more


far fetched, I have to say the Tesla bot just on its face is way more complicated than a car,


especially if it is supposed to not only you know, be on the factory floor, in which case they just


build like a robot arm, right? These are like the most useful things in a factor on a factory floor.


But if it's actually to sort of interact with humans or in a human way, navigate not only


unknown terrain, but also society potentially, I mean, this is just this is just futurism at this


point. And there's really nothing we can legitimately say about what's possible,


what's not possible where this is. And obviously, they like we don't we don't have a prototype,


we just have like a human in a in a suit to demonstrate the Tesla bot. So I will not comment


much further on that with respect to the Tesla fully self driving system. I want to say that


obviously, you know, for Elon Musk, there's always kind of lovers and haters. And I think you can


acknowledge both sides. He is a bit of a salesperson, he sells these things very well,


he always promises, you know, next year, we'll be ready next year, we'll be ready. And then they


never are or he over promises massively on you know, how much cost you can save and yada yada


yada. But then on the other hand, he also delivers a lot more than other people deliver. Maybe that's


just because a little bit of recklessness, but also the sort of optimism and momentum that he's


able to to come up and drive. And all of that together, I think just makes for like an interesting


person. And I think the advances itself are remarkable. Even if you say other car companies


are on the track and whatnot, Tesla has done more than all other car companies together for the


adoption of electric vehicles. Yes, you can debate whether or not that in itself is a good thing. But


just to say that it's not only salesmanship, there are also results. And I have no doubt that in the


near future, we will see self driving cars. Sure, they're not going to be accident free, but I


believe they will be much, much better than humans. And the question is simply is this next


year in two years in five years, I cannot tell you but I'm excited to see I hope you like this talk


analysis interview analysis. If you want more of these things, let me know. Otherwise, let me know


FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

Hi, today we're looking at FixMatch, Simplifying Semi-supervised Learning with Consistency


and Confidence by Kyuk-San, David Berthelot and others of Google Research.


So this paper concerns semi-supervised learning.


So what does semi-supervised learning mean?


In semi-supervised learning, you have a data set of labeled samples.


So right, you have this data set of X's and corresponding Y labels.


But this data set sometimes is very small.


Now you have a much bigger data set of unlabeled examples, just X's with no labels, right?


So you don't know what the labels of the unlabeled examples are.


But what you would like to do is you would like to use this really large data set in


order to help you with learning the association between the data points and the labels.


So for example, in this case, you would have something like an image classification data


set.


And I'm going to take the example here of medical data.


So you have pictures of lungs.


Let's draw a lung here.


That is an ugly lung.


You have pictures of lungs and whether or not they have like a tumor in them, right?


So medical data is very hard to get, especially labeled medical data.


Because you need, first of all, you need the data itself.


But then you also need like one, at least one, but ideally like three radiologists to


look at whether or not this is a good or a bad image and label it.


So it's usually very expensive to collect that data.


But you might have plenty of unlabeled data, right?


You might just be able to go through some database and find like anonymized, undiagnosed


lung scans somewhere lying around.


The same with image, like other images.


So labeling images is pretty human intensive, but the internet contains like a whole bunch


of unlabeled images.


So the task of semi-supervised learning is how do you use this unlabeled data set in


order to make your classification on the labeled data set easier.


And FixMatch combines two approaches to this in a smart way, namely consistency and confidence


approach.


Right.


So what does, what does, we'll jump right into the method.


So basically what you want to do is you want to say my loss that I optimize, right, this


is my loss, consists of two parts, namely a supervised loss, which is your classic classification


loss, right?


Plus an unsupervised loss, right?


And then you have like some sort of a trade-off parameter in front.


Now your supervised loss here, this is where, this is just the cross entropy, let's call


it H, between your predicted labels and the actual true labels, right?


And the predicted labels, they can be, you know, kind of a distribution over labels.


Now the magic of course is here in the unsupervised loss.


And this unsupervised loss, this is what's described here in this part, right?


So the unsupervised loss is going to be this H between P and Q, and we'll see what P and


Q is.


So if for the unsupervised loss, you of course want to start with an unlabeled example, then


you have the same sample go into two different pipelines.


In the first pipeline up here, what you do is you so-called weakly augment it.


And here we're dealing with images, so we have to talk about image augmentation.


So image augmentation has long been used in supervised learning to kind of give you more,


it's kind of a cheat to give you more training data.


So if you have an image, right, of let's say our famous cat, you can obtain more training


data if you, for example, by random cropping.


So you can random crop, let's say we just take this bottom right corner here, and then


we enlarge it to the original size, right?


And it is still sort of a cat, but it's just a part of a cat, right?


But usually that helps because you say, okay, my image data set is just pictures of animals,


right?


It's entirely conceivable that someone held the camera like this or like this, right?


So technically, in terms of generalizing to a test set, these both data points should


be valid.


So I'm just going to add both to my training data.


So you can see how from one training data point, you can get many training data points


just by doing this cropping.


What you can also do is you can flip it left, right, right?


You just swap the pixels left, right.


And usually these kind of, so a cat that has a little dark spot here is still a cat when


it has the little dark spot over there, right?


But to your classifier, those are two different samples.


So you can do many of those things and they have two kind of augmentations.


They have what they call weakly augmented and strongly augmented, right?


So in the weakly augmented pipeline, I think they just, they crop and they shift and they


rotate or something like this.


So you can see here, this horsey here, it's something like it's cropped here about, then


it is turned slightly to the left and then, yeah, I think that's it.


So they crop, they rotate, and then they also flip horizontally at random in like 50% of


the time.


So this is what's called weakly augmented.


The goal here is just to kind of obtain a bit more training data.


All right.


So you run this through your model, through your classification model, as you would a


regular sample, and you get a prediction.


Now from your prediction, you can take the highest prediction here, and that is going


to be your pseudo label.


So this is P of Y.


This is your distribution that you estimate, right?


So and this, and this, if you just take the max, this is going to be your Y hat, right?


And this is what they call a pseudo label.


Sorry.


You'll see why it is called a pseudo label.


So the other pipeline here is the strong augmentation pipeline.


Now in weak augmentation, we just wanted to get some more training data.


In strong augmentation, now the goal is to really screw up that picture to the point


where it's still, you know, you could recognize in the same class, but you can see here, the


augmentations, they go wild.


So you play around with the color, with the hue, you play around with the light intensity,


right?


With the contrast, you can do many, many things.


You can see this, this image looks basically nothing like this image, but you can still


kind of recognize it as a horse.


But the strongly augmented data is much more distorted than the weakly augmented data.


And that's the point.


So also you send the strongly augmented data through the model, and again, you get a prediction,


And now is that the trick is you take the label from here, and you, you take that as


if it were the true label, right?


You take that as if it were the true label.


And you form a loss from this prediction being the model prediction, as if this thing here


that also comes from the model as if that was the true label, right?


That's why it's called a pseudo label, because it is a label that you produce from the model


itself.


Now, of course, if these were to be the same picture, it would be kind of pointless, right?


That's why you see there needs to be a weakly and a strongly augmented pipeline.


I am pretty sure if you want a more basic version of this, make this just clean, so


no augmentation, and make this augmented, right?


That's that's how you can think of it.


The fact that there is weak and here strong augmentation, I think is just a classic trick


to get more training data.


But in essence, you can think of it as this is here, the clean thing, you just want to


produce a label.


And then you want the that the an augmented version of the image has the same label.


Now, you can think of it shortly, what does this model learn?


If you just have this, you remember, I think the important thing is always to remember


that there are two components here, right?


There is first the supervised loss, this is the important one, ultimately, because we


have the true labels, right?


And then second, there is the unsupervised loss, which is just an auxiliary loss that


is supposed to just kind of tune our model to the nature of the data, right?


So don't forget that this, this down here just concerns the unsupervised part of that


loss.


So if you think what does the model actually learn when, whenever you train it like this,


it basically learns to revert this strong augmentation, right?


So it basically says, hey, model, whenever I give you a weak augmented image, and I distort


it heavily, right?


Whenever I give you an image and I distorted heavily, I want the label to be the same.


So the model basically learns that whatever the image, the, whatever the image, the model


at the end of the training will be able to basically map any strongly augmented picture


to the same class as a weakly augmented picture if it comes from the same source, right?


So the model basically learns to ignore these kinds of augmentations.


That's what this loss over here does.


It basically says these sorts of augmentations, these sorts of distortions of images, please


ignore those because I always want you to output the same label here in the prediction


here as if I had not distorted or just weakly distorted the image.


So that's, that's what you have to keep in mind that this, this loss is designed to make


the model not distinguish between differently augmented versions of the same image.


And interestingly, that really seems to help with the, with the supervised loss, right?


My kind of hypothesis is, is that all these methods, what they're kind of trying to do


is to just tune the neural network to the, let's say the order of magnitude of the, of


the input data and also to the kinds of augmentations that the humans come up with.


And that's a very important point.


So the augmentations, and here we said, you know, it's a, it's kind of a rotation and


the crop, the kind of augmentation really seem to play a role.


So this paper finds that on CIFAR-10, where the state of the art, I believe is something


like 96, 97% accuracy on CIFAR-10 with just 250 labeled examples, right?


Now the usual data set size is about 50,000.


It goes to 94.9%.


So almost 95% accuracy with the state of the art being like 97.


This is incredible with just 250 labeled examples.


Crazy, right?


And it, with only four labels per class, it gets 88.6%.


So that's just 40 images with labels.


They get 88.6% of the, of accuracy compared to the 97% that you get with like 50,000 images.


That is pretty, pretty cool, right?


Simply by having all other images not labeled, but pseudo labeled and consistency regularized,


right?


So the two things that are combined by FixMatch again are consistency regularization, which


basically it means that the model should output similar predictions when fed perturbed versions


of the same image, right?


This they're really forthcoming that they are not the ones who invented this.


They just combined the consistency regularization with the pseudo labeling.


The pseudo labeling, they have also not invented.


The pseudo labeling leverages the idea that we should use the model itself to obtain artificial


labels for unlabeled data.


We've seen a lot of papers in the last few months or years where it's like the teacher


teaches the student and then the student teaches the teacher model again and so on.


So that they simply combine the two methods in a clever way.


They have one last thing that is not in this drawing, namely, they only use the pseudo


label.


They have a break right here.


And they only use the pseudo label if the confidence, if this P of Y here is above a


certain threshold.


So they don't take all the pseudo labels, but they only take the labels where the model


is fairly sure about, right?


So they have actually an ablation study where they show that this is reasonably important.


And if you go down here where they say ablation, where is it ablation?


Ablation study.


Oh yeah, something I also find cool.


If you just give one image per class, one image per class, 10 images that are labeled,


it still gets like 78% accuracy.


I think the images are chosen as good representations of their class, but still one image per class.


Pretty pretty cool.


An important part of this is the ablation study where they say, okay, we want to tease


apart why this algorithm, why this semi-supervised learning technique works so well.


And they find several important factors.


They find, for example, that their augmentation strategy is extremely important.


So how they augment the images is very important.


You see here the error of this 4.8% on the 250 label split.


If you change up the augmentation strategies, your error gets higher, right?


And so they say we use this cutout and we measure the effect of cutout.


You find that both cutout and seek-t augment are required to obtain the best performance.


Removing either results in a comparable increase in error rate.


Now you've seen before, for example, they went from this 93, sorry, 93 point something


percent to 94 point something percent from the previous state of the art semi-supervised


learning.


And here they find that simply changing the augmentation strategy changes the error by


more than a percent.


So you can just see this in context of what's important here, right?


They say, again, the ratio of unlabeled data seems pretty important.


We observe a significant decrease in error rates by using a large amounts of unlabeled


data, right?


Then the optimizer and learning rate schedule seems to be very important as well in that


they use this, they say SGD with momentum works much better than Adam.


And then they use this decreasing learning rate schedule, this cosine learning rate schedule.


So there seem to be a lot of things, a lot of hyperparameters that are fairly important


here.


You can see that the gains are substantial sometimes, but they aren't like through the


roof substantial, where you can make a good argument that it is unclear how much really


comes from this clever combination that FixMatch proposes and how much also just comes from


whether or not you set the hyperparameters correctly and exactly how much computation


are you able to throw at selecting your hyperparameters.


So that seems to be a bit of a pain point for me.


They also say we find that tuning the weight decay is exceptionally important for low label


regimes, right?


Choosing a value that is just one order of magnitude larger or smaller than optimal can


cost 10 percentage points or more.


And so that all of that seems to me that this kind of research where you're nibbling for


half or single percentage points in accuracy while a single misstep in a choice of hyperparameter


might cost you 10 times that gain is a bit sketchy.


Now I recognize they get numbers like no one else has gotten before, but where exactly


the gains come from and if the gains really come from this architecture or actually just


more from throwing computers at it, I don't know.


Alright with that I hope you enjoyed this and I invite you to check out the paper.


Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (Paper Explained)

Hi there, today we're looking at transformers or RNNs, fast autoregressive transformers


with linear attention by Angelos Kateropoulos, Apor Vias, Nikolaos Papas and Francois Fleuret.


So this paper on a high level proposes to interpret the attention mechanism in transformers


with a in terms of a kernel function, and therefore the resulting higher dimensional


linear operation can be used to formulate the linear transformer, which is orders of


magnitude faster than a classic transformer. They also show that in the case of autoregressive


transformers, this makes the transformer essentially equivalent to a special kind of RNN. So yeah,


that's what this paper is about. And I think I have some comments to make that I haven't


really seen made by others, though I have to admit, I also haven't really looked at


many comments. So I might just be telling old boring things. As always, if you like


content like this, consider sharing it out, leave a like if you liked it, leave a comment


to you know, let me know what you think I do read the comments and they're generally


comment section is very, very helpful to me and also people, you know, respond to each


other. It's fairly cool to see that the community is usually very helpful to people asking questions.


And yeah, just let me know what you think. Alright, so what's the problem with transformers


and I've done many videos on transformers and I keep referring back to them for people


who don't know what it is. But there's this original paper called attention is all you


need that where I made a video about that. So if you don't know what transformers are,


you can go look at that that should basically cover everything you need to know. But there's


many more transformers. In the meantime, there's BERT, GPT, 2GPT, whatever the number is after


that. Many sequence processing models are now transformers, many set processing models


are now transformers. So this has reached a very, very made a very big splash in the


community. So essentially, transformers come with this attention mechanism where you have


an input set, actually, but let's consider it a sequence. So a sequence of text, maybe


like I have an ice cream cone, something like this. And you want to classify the text or


you want to perform language modeling. So in language modeling, the problem is as follows,


I give you this piece of text and I ask you to predict the next piece of text. This is


this was kind of the first task that these transformers were used on. And this is what


is called an autoregressive transformer, because you always have a piece, you predict the next


piece, and then I give you that next, it gives you that entire piece, and then you predict


the next piece yet again, and so on. And this autoregressive property is going to, you know,


come in play in this paper later. But ultimately, what you have in a transformer is called an


attention mechanism. So an attention mechanism is the following each layer in the transformer,


you can imagine as having the same number of nodes, kind of the same number of neurons


as the seek your sequence is long. Now, from these input sequence, you're going to generate


for each of these tokens, you're going to generate three different things, you're going


to generate a key query and the value. So in in these you do from so usually, this doesn't


come in form of a letter, right, this comes in form of some kind of embedding vector.


And from that, you're going to generate three different things, I should probably use different


colors for for. So this is a function, you're going to produce three different things from


that, you're going to produce a key, you're going to produce a query, and you're going


to produce a value. Now, the key is, you can imagine it being attached to this lower layer


right here. So that's the key for this, this token right here. That's the key, the key


here for that token right here, it's a word piece, right. So the keys, again, are also


just you know, vectors, vector, vector, the query, you figuratively attach to the top


layer right here. So the queries, they go here for each token, and they are also vectors.


And the values will keep out of it for now. So the queries and the keys define basically


how you route the information, and you route the information by going over each. So each,


you have to imagine each token right here, this this half or half, it needs to aggregate


information from all the other tokens, right. So we're going through multiple layers of


this. And in each layer, each of these tokens is aggregating information from the other


tokens. If we do this in multiple rounds is eventually, you know, the each token is aggregating


information, eventually, each token knows about all the other tokens. But how this information


aggregation is done is very important. For example, if the token is a pronoun, it would


be very interested in information coming from any sort of named entity in the sentence,


because it very much wants to know what it is referring to, right. If you are a if you


are the the pronoun in the sentence, it is very vital that you understand which of these


things you refer to. So you'll start aggregating information for that. And then once you know,


who or what you refer to, then the other parts of the sentence can make use of that information.


So they will start requesting information from you. And so layer after layer, each token


aggregates information from each other token. So this works by let's say we're at this token


right here. What we're going to do is we're going to form the inner product between that


vector and each of these vectors. And then we're going to transfer that into a softmax,


which makes this into a first of all, there's so we do the query together with all the keys,


key, and then we run it through the exponential function. And after that, we're going to normalize


it by the sum of all the exponential functions that will give us a properly normalized distribution.


So a histogram basically, of where we are going to get our information from, this is


going to be the highest where the inner product right here is the highest. So from this token


right here, and you know, this is fairly, fairly standard. What I drew by accident is


fairly standard that a token probably wants to know a lot about itself. So you want to


carry forward the information that you already have in this particular token. That's why


your inner product is going to maybe align a lot with your own key. So the keys and queries


are learned. So each token decides what kind of information it wants to advertise to the


others. And then also each token decides what kind of information it wants to gather from


the others. And the the routing then is put through a softmax function. And that gives


you this right here, you do this for every single token. So the problem with this is


that every single token needs to do the inner product of its query with all the different


keys, and each of that has to go through the softmax. And then the value that's actually


aggregated are these values right here. Now, the values are simply a transformation of


the incoming values. Values are what's really propagated. You can think of it as just like


a one layer neural network. Ultimately, you could also leave away the values, people don't


do this. Some people do the same queries and keys, but the values are just a transformation


of your input. So the important thing is this right here, this decides how you're going


to aggregate the values. Alright, so this is has a quadratic complexity. So if you if


you have n input tokens, then this entire process will require n squared operations


because you need to form the inner products between each pair of queries and keys, right.


And it also is going to require that much memory. And this we're going to see this is


in large part due to this softmax operation because because we have a softmax, it makes


the whole thing nonlinear. And it being nonlinear basically means we'll have to, you know, store


everything, keep everything around and we have to recompute for each query. We're going


to see in this paper formulation where if we make the whole process linear, then we


will not have to do that. So let's dive into it. So here they go linear transformers. They


start off with saying each transformer layer is essentially this right here. So this is


a this is kind of a higher level of view. What we viewed so far is just this part right


here. This is the attention routing mechanism. Each layer is actually wrapped in a residual


connection and also a simple element wise or row wise feed forward layer. But these


things are usually not that much into consideration. What's really hurting in the transformer if


you go into very long sequences is this attention routing mechanism. So the attention routing


mechanism is as follows. You can see right here this is the formal expression of what


I described right here. Here you have the and notice this is an outer product. So if


I have if I have n sequence elements, the q right here are the queries. So this transforms


each of the n into a into a d dimensional space, right? And also the keys will transform


each of these into a d dimensional space. So this here is going this here is going to


be a n by n matrix, right? This is this q k t is going to be an n by n matrix. This


is x w q w k x and this transposed right here. Nope. Yeah, like this. Okay. So this is sort


of an outer product. And then we're going to take the row wise softmax. And that will


give us for each row in this matrix. So for each row in this matrix, we're going to have


these this distribution of how to aggregate information each row will give us basically


for each of the upper level tokens for each of the outputs, how we need to aggregate information


from the inputs and the information that we're aggregating are these values right here. Now,


they generalize this first of all, they say, we can also we can write it in this form right


here. Instead of having a softmax, we can actually think of any kind of similarity function


it between the queries and the keys. So here you see what we want to do if we want to calculate


output I here, the important thing is there is no longer this is an entire matrix. And


we considered a row wise softmax. And now we write this out into the individual elements


of the output. And we can we can do so we can say, okay, how do we obtain one element


of the output, we're going to calculate some sort of similarity of that particular query


see I here, I here, we're going to calculate some sort of similarity between the query


of that particular output with all of the keys. So here you can see all of the keys


of the input. And we're going to add and we're going to normalize, right, this is the normalization


that happens also in the softmax. And that will give us like a histogram of how we aggregate


the values right here. So all of this all of this red stuff will give us again, some


sort of a histogram of how we're going to aggregate information. If you look a bit like


this, and you know how the softmax is defined, you'll see that if we plug in the exponential


function for as the similarity function, then you'll get back to the softmax. Okay, these


they say, equation three is equivalent to equation two, if we substitute the similarity


function with the exponential function. Now, they go, they go ahead, and they go into kernels.


So for that, you sort of need to understand what a kernel is. A kernel is a special kind.


For the purposes that we are, you know, looking at here, a kernel is a special kind of a similarity


function, it needs to have some properties right here. But essentially, they say, well,


this this kind of looks like a kernel. And we will simply say, okay, here, this similarity.


What if we use a kernel here. So a kernel simply is a similarity function of two vectors,


if you interpret it like it has some more conditions, I know, I know, don't freak on


me. But the interesting properties about kernels is that if a similarity function is a kernel,


it means that there exists a mapping. And where do we do so, if k between a and b is


a kernel, if k is a kernel, that means that there exists a similar a function phi, such


that phi, such that the kernel between a and b can be expressed as a linear product between


phi of a and phi of b transpose. Okay, this is like, this is an inner product. So what


it means is that this can be like a super nonlinear function, a kernel. For example,


it can be and the example often given in like machine learning classes is maybe something


like this, you have one dimensional data, right, and here is the error zero. And you


have two kinds of data points, you have the x's right here, and you have the circles right


here. Now, I cannot classify this data linearly. However, however, I can transform this into


a higher dimensional space. So my function phi is of my function phi of x is going to


map to the vector x x squared. And that will transform the data into a two dimensional


space, right. And the data will look something like this. So it's going to the y axis is


going to be the square of the x axis, okay. And like this, and now I can find a linear


classifier. Okay, so in this case, right here, you can see that in this higher space, things


become linear, things become linearly classifiable. And very similarly, like this is you can define


the similarity between things right here. So the similarity function would be the square


function right here. And this would be a quadratic an example of a quadratic kernel. So this


function right here can be very non linear. I mean, it can be a linear function, but it


can be very nonlinear. But it is equivalent, it is equivalent, this means it is equivalent


to a linear function in a high dimensional space. Now to figure out linear to figure


out what this function phi is, is the big, the big question, of course, for a couple


of kernels, we know the function phi, right for the quadratic kernel, for example, we


know we just saw that phi maps this to the vector of the coordinate and it's quadratic


to its square, we know for a couple of other kernels what their associated functions are.


But in general, if it's a kernel, then we can just replace it with a linear function.


With with with this thing, and in reverse, we can just say, well, what we could do is


we could just simply define a function phi, and basically map this into map these a and


b into a higher dimensional space, where this super duper nonlinear function would just


become a linear function, wouldn't that be much easier, linear functions are much easier


to work with than long nonlinear functions. And if we know that, as long as we get the


correct phi, we do exactly the same thing as the nonlinear function, you know, that


would be helpful. So there is an entire literature in the entire like decade of kernelization


and kernelize everything kernelized SVMs to start, but then you can go way further in


this. And this is just the beginning. And this is just a very sloppy explanation by


me right here. But ultimately, they're saying, hey, instead of doing complicated nonlinear


similarity function like the softmax, can't we just project a and b into the higher dimensional


space, and then just do the linear inner product? And do the same thing as the softmax? And


the answer is yes. And no, we know that for the softmax, the particular five function


that we would need would map to an infinite dimensional space. Usually, usually this is


applied in reverse. It's like, oh, here, instead, usually in machine learning, they say, you


know, we want to do this, we want to map it into a high dimensional space, there's just


linear, but we we can't because these spaces are too high dimensional. And therefore, we


find an equivalent kernel function. And it's usually said, well, we use an RBF kernel that


corresponds to an infinite dimensional space and so on. That's pretty cool. Here, it's


the reverse here, it's, we want to do, we want the linear function, we end the equivalent


of the softmax function is an infinite dimensional function, which we can't do, right, we can't


feasibly compute an infinite dimensional space explicitly. So it's not possible to just do


the equivalent thing than in a transformer. However, you can still do something else,


you can still use polynomial kernels, you can still use any kind of kernels that have


corresponding functions that map to a finite dimensional space. And that's what this paper


does. So here, they say, if we have such a function that maps these things into a higher


dimensional space, such that their inner product, such that the similarity function in this


higher dimensional space is an inner product, then we can write it as just this inner product


right here explicitly. And then because of the associativity, you can see that here is


an I and here there's no I so we can just sort of pull this out of this sum and as well,


right here, it doesn't don't don't cross this away. These are vectors, right? But you can


see, especially here, you can see pretty clear, why is there a cursor stop, you can see that


this here, you have to pay attention to the matrix dimension. So if we use like bracket


notation, this is like this, like this, like this and like this. Okay, so here on the bottom,


you see that there is an inner product. So each output will be normalized by this inner


product right here. However, the top is going to be a vector, we know that, you know, each


output is a vector. So the top will aggregate these vectors right here, according to this


routing. Okay, but if we write it like this, you can see that what we could technically


do is we could technically compute this part here once because it doesn't contain any I.


So there's no I in that part. So we could just compute it once because we have these


two, these two layers of the attention mechanism. And these K and V, they just refer to this


lower layer right here. We could just compute that thing on the right ones, that's going


to give us a matrix, as you can see right here from the dimensions. And then we can


simply take the product of that vector right here of the vector on the left with the matrix


on the right. And we'd be done. It's one operation, right? Instead of for each thing, you know,


going and attending to each other, and then do the softmax. Without the softmax, we can


all do this in a linear fashion. So that makes it a lot easier. In fact, it makes the computation


linear in. So this is now O of n. Okay, plus, of course, the the work that you have to do


for mapping this into the higher dimensional space. But this is also not quadratic, this


is done to each of these elements individually. Okay, so this, this is now, as we said, it's


pretty easy. You can calculate the matrix on the top, you can actually also calculate


this part right here, this vector, you can aggregate over the bottom. And then if you


go through the top, it's simply a inner product with the vector of the queries, and you're


done. And this is it, in fact, in matrix form, you can simply write it down as one matrix


multiplication. Seems pretty easy. So the computational cost goes way down. And they


use the following function right here. Okay, this is their map to the higher dimensional


space. So they say for our experiments that deal with smaller sequences, we employ a feature


map that results in a positive similarity function as defined below. So right here,


you have to pay attention, you can't just pick any function. But you can you can pick


a lot of different functions, where LU denotes the exponential linear unit activation function.


Okay, like this seems, this seems fine. They also say in our experimental section, we show


that the feature map of equation seven performs on par with the full transformer, while significantly


reducing the computational memory requirements. This, you know, it seems, it seems like the


the original transformer, this choice of the softmax function, even though it's, you know,


powerful and can't be approximated with this trick right here, it was also somewhat arbitrary,


I mean, there is a reasoning behind it, but it's also somewhat like, and it's entirely


possible right that that this here is way faster. So I want to jump this causal masking


thing for now and look at the results where you can see a verify the fact that in terms


of time in terms of GPU memory, if they apply their transformer and here on the x axis,


you see sequence length, you can see that the this is log plot, right? These are log


plots, you can see that the original transformer right here has a way steeper slope than their


transformer, which is the black line right here. The blue lines are the reformers, which


we've also I've also done a video on reformer. If you want to check that out, that is also


a trick that uses locality sensitive hashing to get rid of the quadratic attention mechanism.


Now the locality sensitive hashing also means that you kind of lose some accuracy. So that's


the trade off right here. But you can see that is also linear, actually, it's n log


n depending on the sequence length, but the log n is negligible. So you see GPU memory


and time way down. And in terms of experiments, it does perform on par, it seems like it has


different optimization trajectory, but they show that, you know, there is this trade off


for the reformer where you lose in accuracy, they do not experience that trade off in the


linear transformer compared to the original transformer in their particular experiments.


Now they do their experiments sort of show that they are not on par with the original


transfer like they are on par in some of the tasks, but also in some of the tasks, they


are not on par. For example, this speech data set right here, where they do fairly well,


they actually beat the BI-LSTM baseline and the reformer, but they do not beat the softmax


transformer. So there's, it is still the case that the softmax transformer is more powerful


than the thing here and we'll give some intuition very shortly on that. But the linear transformer


is way faster. Here it's three times faster. And up here, it is 300 times faster. And on


MNIST, and if you go on C410, it is 4000 times faster. Simply by property of the longer either


sequences are that you input, the much more matters the fact that the softmax transformer


has a quadratic runtime, whereas the linear transformer has a linear runtime. And I was


also surprised here to see that the reformer wasn't that much faster. That's probably due


to the fact that it already has like a big overhead in these hashing rounds and so on,


that probably is hurting it at sort of a constant level. I guess if you were to up the sequence


length even more than the reformer would also improve a lot more over the softmax transformer.


Okay, so what's happening here? What's happening with this attention? And why is it different?


What does it makes it different from the old attention? Now I want to, I want to sort of


connect this to the kind of old and old, the old and literature of topic modeling. So if


you think of the of this transformer, again, of this attention mechanism, what you'll have


is a dynamic routing of information, right? So for each, from each output token, you get


to look at all the input tokens. If we, for example, select this one, you get to look


and you get to decide for each one, how do I want to aggregate my information? Okay,


and this is what makes this quadratic from each of the output tokens, you get to look


at all of the input tokens and decide how you want to do that. And that is can be very


long nonlinear in terms of when we use the softmax and so on. So that what makes it expensive.


What this thing is doing is the following. It takes all the keys right here. So here


we have all the keys. And it's going to map them through this five function, right? Each


key is going to map through the five function. And each query is also going to be mapped


through the five function into these higher dimensional spaces. And then an inner product


is performed between the two and that decides the routing. This is very similar to like


topic models, where if you interpret this, this right here can be a mapping of my dimension


of these keys and queries to the topics. So essentially, what's happening right here is


for each of the input tokens, sorry, input tokens here, output tokens here, the dimension


of this map defines is how many topics there are. So in, you know, in these topics modeling,


you would have things like I want to I have news articles or words, and then I define


like a set of topics. And I'm going to assign each word to a topic. And then I'm going to


assign each news article to a topic and so on. And then you kind of do this dimension


reduction. But this can be done in many ways. So let's say this is a mapping to three dimensions.


What this does is essentially this five function decides how you're going to map each of these


inputs into these three topics. So you can say, oh, this token goes here and here, this


one goes here, and a bit here, this one goes here, and so on. So again, this is a this


is a mapping into a well, in this case, a lower dimensional space. And then this function


decides how you're going to aggregate these topics are over across here. And since this


is, you know, this is now a linear multiplication between the two things. So these two are going


to be your matrices, this here is going to be your phi of k. And this here is going to


be your phi of q. So you can see the difference here between the old attention mechanism and


the new attention mechanism, right? The old attention mechanism, each token was directly


able to look at all the input tokens and decide how to aggregate the information. And here


it's sort of we have this in between, in between representation in this higher dimensional


space. And we can aggregate in only a we can distribute in a linear fashion. And we can


aggregate in a linear fashion in and from this higher dimensional space. That's sort


of how how I sort of how I imagine that, okay, so you get to distribute each token right


here into these topics. And then the the outputs they they don't see the inputs anymore, right,


you see that in the formulation, there is a sum over j. So right here, there is this


sum over j. And that means that the outputs here, they don't see the different inputs


as different inputs, they only see the inputs through the map of the phi function. So they


can only see the individual dimensions of that phi function, they cannot see the outputs


anymore. And therefore, yeah, therefore, you don't have the dependence on the big quadratic


dependence on this on this n. Okay, however, you do have a curve course now a dependence


on this dimension of the intermediate representation. And they also they say this, right, this is,


you know, reasonable. Yeah, they do derive the gradients here to save even more memory.


So you don't have to search that you don't have to, let's say store of all of these activations,


that's pretty cool as well. And they implemented in CUDA, there is code available for the linear


transformer, all of this pretty, pretty cool. Okay. So the last thing they say they make


the connections to our n ns. Now, this is a bit detached from the linear transformer.


But because they formulated how they do, they can make this connection. So this now, this


now is valid for all transformers, what they say right here. But keep in mind, it is valid


for the original transformers in practice, if you can make this mapping phi to map to


infinite dimensions, which you can't. But the analysis is equivalent. So they say, look,


if we write the attention mechanism like this, and therefore, like this, what we can do is


we can define these two quantities, right, s and z. And this is what we said before,


we can actually pre compute these quantities right here. Okay, so that reduces to this


right here. If now we are looking at a autoregressive transformer, and we said before what an autoregressive


transformer was an autoregressive transformers, you have a piece of sequence and you are tasked


to predict this next thing right here. Now, usually, if you want to train this using an


RNN, you have to, you know, run your RNN input this hidden state and input that map forward


the hidden state. So you have to do all of this forward propagation in order to derive


at this hidden at this output right here, make the output and then you need to back prop


through time right here. There is no way to what you would like to do is you would like


to say here, I have a sentence, I can actually make like five different training examples


from that sentence. So the first one is the one you just saw, I just block off the last


word. But I can also make that training example right here, right to when I just cut a second


to last word, and so on, I can actually make all of these different training examples for


language modeling from a single sentence. And what I would like to do is I would like


to train them all in parallel, right? I load my data point once I already have it, why


can't I just train everything at the same time, like predict this from this word now


predict also this from these two words, and the transformers are, you know, very well


conditioned, they are very good at this basically. So what a transformer can do is if you input


a sequence, like sorry, like the thing at the bottom, you can calculate the training


signal for all of these different things at the same time. And okay, this was maybe a


mistake, you can calculate the training signal for all of this at the same time, by using


what's called causal masking in attention. So if I have my attention mechanism, right


here, let's consider it again, let's consider these two layers. If I have my attention mechanism,


what I want to do is I want to constrain each token to only attend to tokens that came before


it in the sequence. So for example, this token right here, I'm going to constrain it to only


attend to itself and the past, because it will it will predict the next token in the


sequence and it wouldn't be it would be really easy if we could attend to the input of that


token, right? It could simply remember what that token is, and then aggregate that here


and then predict that. So if for each token, we restrict the attention to the tokens that


came before it, like also for this right here, we restrict the attention only to go backwards,


then we can train all of this in parallel. This is called causal masking, it's usually


implemented with like a mask that is like an upper diagonal. And it's a bit unclear


if you can attend to yours to yourself, because then I guess this will become the output and


you can only attend to this. I don't know exactly how it is implemented. But there it


is usually realized with an upper triangular matrix as a mask. And you apply this mask


to each layer. Now, they say that this is actually like an RNN. And with their formulation,


you can make this pretty explicit. Namely, you have these two states s and z. And in


each sequence element, it's actually like an RNN, where you update the s and z with


these quantities right here. And so it's like an RNN where these are the hidden states that


you pass forward, right. And then you can formulate any transformer as an RNN that simply


updates these two states. But you see, you need the explicit mapping of these of this


kernel function, you need this explicit mapping in order to be able to do this, because otherwise,


this is here, this is not going to be a linear addition, it is going to be complicated, you


can't do it by simply remembering the past state. So you need that formulation in order


to be able to express it as an RNN. But their analysis shows that this a transformer autoregressive


is essentially an RNN. And you can you can so you can make a connection in that and you


can actually formulate this as an RNN, which means that you can train in the transformer


fashion everything at the same time. But what is cool about an RNN, an RNN at inference time


an RNN, once it has produced, you know, this word, it can then because if you produce autoregressively,


you simply say, Hey, I have this beginning of my news article, please finish it. So the


model must output the next word and then from that sequence, it must output the next part,


the next word and then from that the next word and so on. And RNN because of the nature


of simply passing forward hidden states at inference time can simply you know, remember


what the hidden states were input those again input the output here and go on. So it's pretty


fast at inference time, which a transformer isn't with their formulation now, if they


have the explicit function phi, they can use this at inference time to be so much faster.


In fact, on their website, which I'll link, of course, in the in the description, you


can play with image generation using one of these transformers in your browser. So you


can simply start a transformer run in your browser. That's how easy this becomes. So


you can see the linear transformer with causal masking, you'll simply update these states


right here, and then pass those forward. So easy. And the backward pass, as we said, I


don't want to go into the gradient calculation, but they derive the gradient such that you


don't have to remember these hidden states and it becomes or it is linear in or it saves


a lot of more memory than before. Okay. Note, so this is the last comment from my side.


Note that this this causal masking transformers, they are they are a bit of a hack in transformers.


And because so ultimately, let's say, let's say I have this sequence right here, this


is given. And I want to predict this word right here. What? And okay, let's make it


up here. So I need multiple layers for this. So I want to predict that next word. And I


have multiple layers, right? So I want to predict this from from the outputs right here.


Let's say there is an output node right here, I want to predict that particular word. It's


true that I should only be able to aggregate information from whatever was, you know, on


the back right here. But technically, in a transformer, it would be completely valid


to say that this node right here, which is let's say that's an article, and it followed


by a noun, right, would be able to attend to that one. And then that one would be able


to attend to that one. And or sorry, the output right here would be able to attend to that


one. This would not violate the autoregressive property, right? You can, but you can see


that in the intermediate layer, this node right here is attending to a forward node.


Now, if you do things like this, you can't do this trick anymore, where you train everything


at once. Because if if this connection exists, that also means that if in this other training


sample where this is the word to be predicted, then this node could aggregate information


from that node and basically cheat. But the the technical autoregressive property is not


violated by this connection right here. And you only get this RNN formulation if you do


not have these connections, right. So the this this hack to make the autoregressive


transformers train in parallel is actually making the transformer formulation much weaker.


And therefore, that's then equivalent to an RNN. Okay, I it's not that transformers in


general are equivalent to an RNN, or at least this paper doesn't show that. It's just that


these hacked transformers are. And I think that's an important distinction to make here,


rather than saying transformers are RNNs. If we could only approximate the softmax in


these infinite dimensions. I don't think that's entirely true. But it is true for the transformers,


the autoregressive transformers that we currently train. Now, why is this connection so powerful?


It allows a token to attend to, you know, tokens forward of it. And what does it mean


to be able to attend? Like, let's say, it's really important that this token right here


attends to that token right here. What would you need to do if you couldn't do that? If


you let's, let's, let's say this is a program, right? So this token is the function f and


it needs the input, this argument a of whatever token comes in front of it, and needs to do


something conditioned on a so if a if a is one, it does something if a is two, it does


something else, right? If you if you don't have if you can't input a, then you can simply


pass on the output value, what you'll have to do is conceptually is basically you'll


have to store the entire code of the function into hidden state. If this is an RNN, right,


it can't look forward, it needs to store the entire code of this function f. So all it


needs to basically build this map, if a is one, then this if a is two, then this if a


is three, then this store that in the hidden state. And then once a comes around in the


next time step, this can be resolved, you can see that this is infinitely more complicated


than simply looking forward and outputting the value yourself. So that's sort of the


difference in power that these two formulations are talking about. Okay, so yeah, two parts


to this paper, first part, linear transformer through kernels, second part, if you formulate


it like this, it is equivalent. So a autoregressive transformer in this way becomes equivalent


to an RNN. And here are some of the output samples, you know, they're, they're pretty,


pretty good, though, if you look at the more output samples they have here, it. So here,


this, this is the linear one. And you can see, for example, as already in this very


bottom one, this one right here, it's the kind of all the other transformers get the


slant to the right, and that the the original has, whereas this one is simply straight.


I mean, I don't want to I don't want to dunk on this, like these others make a lot of mistake,


mistakes right here. But here, I also saw, you know, all of them get that this is going


to be the number three, while this one is somehow making this circle in here. So it


is not perfect. And even though it's on par, where in the tasks they see, you can see right


here that especially in this speech recognition, the original transformer right here is significantly


outperforming the linear transformer, which is the one in black right here, in fact, in


all of the tasks, but ultimately, it might not matter because they reach you know, the


same, they reach the same, they reach the same accuracy or whatnot, and the linear transformer


is way, way faster. So I can see that this is going to be a thing that people apply,


I guess time will tell. Right, I invite you to read the paper, tell me what you think,


I might be totally wrong here with any of my formulations or my intuition about what


this new attention mechanism does. Yeah, please let me know and I'll see you next time. Bye


ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (Paper Explained)

Hello there. Today we're going to look at EXT5, Towards Extreme Multitask Scaling for Transfer Learning, by researchers of Google Research and DeepMind.


This paper introduces two new things. The first thing is this EXMIX, which stands for Extreme Mixture. This is a data set. It's actually a collection.


They say a massive collection of 107 supervised NLP tasks across diverse domains and task families. So this is a big collection of tasks.


And using that collection, they train this new model EXT5, which as you can guess, is a T5 model trained on this EXMIX data set or pre trained on this EXT5 data set.


And by doing that, they can show that this model, once you fine tune it on the downstream tasks, achieves much better performance than if you were to pre train with less tasks, or just something like language modeling.


In fact, the final model they come up with is a mix of the language model, self supervised pre training task, and training on these 107 supervised tasks all at the same time.


And that seems to be a strong model. It outperforms they say, strong T5 baselines on superglue and a bunch of other tasks, some tasks which are inside of the training set.


So some tasks are going to be part of those 107 tasks, but some aren't. And even on those tasks, the model drastically improves over other models such as T5.


So we're going to look at what the data set here how it is constructed. And I like the way they approach this right here, and the kind of ablation experiments they do to show that really the scale of or the amount of tasks and the diversity of tasks is what really makes the difference right here.


At least they give some good evidence for that hypothesis, it could still be that the selection is important. But you'll see. Yeah, that's, that's the overview. Let's dive into it. So what is T5?


T5 is this model, this idea that we can solve any NLP task with essentially the same model. So a T5 model is a language model, the language model gets a few tokens as an input, and is asked to complete the sequence or to continue the sequence.


This is a standard language model, right? If you have a sentence starting here, you can ask the model to complete the sentence. However, there are other tasks such other tasks than language modeling in NLP, there is, for example, question answering.


Now in question answering, you want to come up with an answer, maybe the answer is part of the document. So this is your document, right? And you want to come up with an answer to a question that is answered in the document.


And you would point out, here is the answer, right? That is what you would do if you were to do something like BERT, like a BERT model, you'd put on top here, you would feed all the sequence in, and then the BERT model would output for these tokens, this is the answer, not with T5.


With T5, what you would do is you would ask the model in a language modeling way to continue here and give you the answer essentially give you these tokens.


So everything is formulated as a language modeling task. And this is done by using clever prompting. So here you can see a list of these tasks. This is the same for T5 and ext five, the difference between them is that ext five is pre trained on a much larger set of such tasks.


So for example, if the task is question answering, the input is this right here. As you can see, it says question, colon, right? And this here, that's kind of the pre prompt this question, colon, that that's sort of prompting the model to now do question answering.


What does the sun represent on the on the Uruguay flag? And the answer is the May Revolution of 1810.


Here is some some other.


I guess this is what dialogue modeling, person one. So true story, I once swam with Monterey, it was awesome person to colon, right, you see this format, this person one colon person to colon, this is how they represent dialogue modeling. So you don't have to build a separate model for each task, you simply have to find a way to formulate the task in a language modeling way, such that you can indicate to the model what kind of task it is by using prompt structure.


And then the model will sort of recognize what this is, you may have seen, right, you may have seen this in GPT three. Now the difference is, in GPT three, you can do prompts like this, and you probably will get an answer here that fits to person two.


The difference is GPT three during training time has also has only done language modeling, nothing else but language modeling. While, and then it simply has picked up these patterns as it went along sort of learning from what was scraped from the internet.


However, at t five here is explicitly trained on these tasks. So prompts like these will actually be part of the training set. Now, it doesn't mean that. So you can do two things. Once you train such a model, you can evaluate it on tasks that have been in the training set, be that on the evaluation sets of the respective tasks, but they would still be considered sort of in distribution


tasks. So you've explicitly trained the model for these tasks. Or you can test it on out of distribution tasks, which is here would be a task, like a different task, but you haven't trained on that task. But you use the trained model to evaluate on it. This comes much closer to something like GPT three, right?


You have done supervised pre training here, but the supervision was on a way different task than what you evaluate. So we're going to see these two different things. So that's the idea of t five, you they pair this here with language modeling.


So there is this one pre training task right here, that as you can see, masks out a bunch of parts of a piece of text, and then the model is asked to reconstruct it again, it's asked to reconstruct it in a language modeling way. So this whole thing will be your prompt with these special mask tokens.


And then this whole piece of text right here would be the output and you would train the model to produce that output, right? Something like GPT three is probably not going to produce the output in a structured way where it says, well, mask zero and then mask one and so on.


So those are the mixture. It's a 107 107 supervised tasks. And then this here I think comes from common crawl from C four, you just do self supervised training on top of that. And you mix all of that together during training.


And then you have apparently a super powerful model that if you take it and you fine tune it on the downstream task, it is going to perform quite well. That's essentially the whole model. It is not conceptually different from t five.


It's simply this fact that we now have the 107 tasks. So they have a split up of what the 107 tasks are, but mostly they fall into kind of different categories here.


They have these task families, their summarization, dialogue modeling, and natural language inference. So classification, semantic parsing, common sense, closed book question answering and reading comprehension.


So this is this is quite, quite a different selection of tasks, you can see that there's wide like dialogue modeling might require some real world sort of knowledge to be in there.


You can see wizards of Wikipedia is in that data set. But also, there is something like semantic parsing. So you've already seen fun qL up here. So here is this is the input parse to fun qL.


Give me a list of airlines in Pittsburgh, and we actually transfer or translate this prompt this in natural language input to a fun qL or funco I don't even know how to how to say that output, you can see the tasks are quite diverse, but in all of them, you have to sort of do something with the language right beyond just language modeling.


Right, so they now do a bunch of different experiments. And the first experiment is with respect to these, these task families, they wonder, they wonder, does it help or kind of hurt if we include a dialogue task for another dialogue task if we include an ally task for a dialogue task and so on, they want to understand,


how do these tasks sort of fit together? Do they help each other? Do they hurt each other? Or you know what's going on? And for that they in this here they have like some inter intra family correlations and turns out most tasks in the same family actually help each other though not all, which is weird, but the families are, of course, something arbitrary that the author simply laid on top of the of these tasks.


The of these 107 different tasks, right, so it's not necessarily the case that all the within family tasks are necessarily so similar to each other. Though the authors say that there are some of the 107 that fit no category. So the categories here are simply for evaluating some measures of how the tasks help each other.


So now, what they do is they as if I understand this correctly, they take a pre trained model, presumably a T five model, if I understand this correctly, so they take it, not maybe not a T five.


I don't actually, I don't actually have this present in my mind. However, they take some model that has been pre trained on something, maybe not pre trained at all.


They fine tune it on two tasks at the same time. So rather than fine tuning it on one task, they fine tune it on two tasks or respectively here, rather than fine tuning it on one task family, such as the NLI family, they fine tune it on two together.


Okay, so here you can like this cell right here is what if I fine tune on NLI tasks and on CLS tasks together, okay, and I find tune on them together and then I evaluate on the column so row I've column J, I evaluate the performance on family J.


So I fine tune on classification and NLI together and then I look at the performance on NLI on the on the test sets of these tasks, right.


And my question is, is that more or less than if I were to just fine tune on NLI, which is this diagonal entry right here, I fine tune on NLI and NLI given like the same compute budget, select one of these two numbers, one, it's sort of the data equivalent to both tasks and one is the compute equivalent.


You can choose but given that this cell is green right here, I think the authors have taken have chosen the top number to be sort of the more representative one.


In this case, you see that training, switch, sorry, co training classification tasks and NLI tasks benefits NLI task evaluation, compared to just fine tuning on NLI tasks.


Okay, that's how you read these, these numbers.


Or maybe I'm wrong, but that's how I think you read these numbers.


So on the other hand, you look for example, right here, you have this CMNS, I don't actually remember what that family, the name is, it gets a performance of 68.24.


If it's just trained by itself, if it's fine tuned by itself, if you add the CBQA family, it gains in perfect Wait a minute, it is green.


Column, family J column J. This is a column, right?


Could this be actually row? No, this is smaller than this. Why is this green?


Why is it green?


I'm confused.


Maybe they have to compare with the average right here.


I'm confused by this table.


I thought it meant, I thought it meant, if you co train on the two, actually, it gets better. However, the 66 is clearly smaller than the 68.


So, so maybe they consider the row, not the column.


I don't know.


No, they would do consider the column because they're in the text, they say, look, for example, adding an NLI task, you can see in the row, this is whatever happens to another task.


If you add the NLI tasks, this is very often beneficial, right?


You can see right here, it's very often a good thing to add the NLI task, for example.


But then on the other hand, the summarization task, it's often very negative if you add a summarization task.


So it is not the result from this experiment, at least they say, is that it is not clear whether or not adding more or fine tuning on multiple tasks together helps or hurts ultimately.


So that is already very special recognition. This is not yet pre training, but it is something about training multiple tasks together.


It doesn't always seem to help for the end result.


Okay, so I think I figured it out.


Yeah, I think they do in fact compare to the lower row right here. And this row is the average without regarding the diagonal.


So this is on average, how does a given task fare when you add other tasks or other task families, and then they evaluate whether there is a significant deviation, you know, to the to the top or to the bottom with the red and green numbers.


So mystery solved. Yay.


I think the lesson from this is that it is not entirely clear that co training helps here.


And now what they do is they go and they say, Okay, can we figure out like, we have 107 tasks, and we set ourselves one goal, which is superglue, which is one of the one of the common benchmarks in NLP, I think it itself consists of, I believe, 11 sub tasks or something like this, and is is most common reported in these NLP papers that do multiple tasks.


So superglue is the benchmark. Can we find a subset of these 107 tasks that when we pre train on when we pre trained on these tasks, and then fine tune on superglue, it's going to be very, very good on superglue, right?


Because given from this table, it's not entirely clear that we should include all of the tasks because they sometimes might hurt each other.


So now we're going to try to say, Can we find a subset that is just really good for the superglue downstream task.


And what the authors do is they do different things. So vanilla, I believe is you pre train on zero tasks, you just train on superglue. Okay, cool.


And then this, this random 55 is simply 55 random tasks. Now, why 55? I don't know, it might be because it's kind of half of what they have, or because it very narrowly beats, whatever they have here, the best effort, the best effort is the author saying, Okay, let's look at superglue.


I'm not exactly sure what. So let's look at superglue. And let's look at all of the sort of helpful families right here.


They say, specifically, we include NLI, common sense classification and closed book qA tasks from the ex mixed for a mixture of 48 tasks to include in a multitask pre training setup.


So you can see those are the tasks right here. If I see this correctly, these four tasks that all have the green numbers right here. So in on average, these task families help other tasks.


And as you can see, with this is 48 tasks that were selected after extensive evaluation, and simply by picking random 55 random tasks, you're already better in the downstream performance on superglue.


Keep in mind, this table here is co fine tuning or co training on two tasks and then evaluating on one of them, which is different than pre training on a task and then fine tuning and evaluating on another one.


But still, it is sort of an effort by the authors to say, you know, what, what could we do right here?


And, yeah, as you can see, the final result, obviously, is that if you use all 107 tasks, you get way better results. So, it seems to be.


This is a good indication that is really about the scale and diversity, and might not matter too much that you have this huge task overlap, although it could very well be that neither the best effort case, nor the random sampling right here have really hit sort of the tasks to include and there might actually be a subset that is even better, but it's kind of unlikely.


They've done a different thing where they compare pre training versus pre fine tuning. So pre training is I train, I train on the supervised tasks and on language modeling, all at the same time, and then I find two and once on my target task.


And pre fine tuning is I first do sort of pre training. In fact, they start with a standard T five base checkpoint. And then after that concluded, they what they call pre fine tune.


So essentially, it is another step of pre training, but starting from a checkpoint, they pre fine tune with this ex mix data set.


And after this phase, we find tune on superglue. So to compare the comparison is simply that you have like a pre training.


And then you have a pre fine tuning, which is where the this ex mix data set goes in.


And then you have a fine tuning and eval


on the same data set. And they compare the setting where you do the three in stages right here, compared to when you do these two things at the same time.


And then you find tune and evaluate, okay, the bottom row is sort of the baseline. And the top row is what they actually suggest.


What they suggest is that, hey, we should just do all the pre training at the same time on the supervised tasks on the on the self supervised tasks and so on.


So here you can see if you simply take a vanilla model, fine tune it on superglue, you get a 76.1. I'm going to guess that's the T five base checkpoint right here.


Then if you pre fine tune that checkpoint on ex mix, you do get 78.1 on superglue, which is a considerable considerable increase, right?


However, if you simply do the same, but include that, those that ex mix data sets into the pre training, you do get an even bigger boost.


So how do we make sense of that? That is it? That is a good question. I'm not entirely sure, but maybe a hypothesis.


So I've I've I've I've in the comments, but one hypothesis here is that I have is that a supervised NLP data set, you know, if you if you look at these tasks right here,


what does it really mean to be a supervised NLP data set?


What it means is that by determining a label to a prompt, you sort of as a labeler bring in additional knowledge that is not just in the text, right?


So, you know, in order for you to parse this kind of fungal query, you have to know a bunch of things. So give me a list of airlines in Pittsburgh.


You can see right here. Okay, Pittsburgh. There is like city name. Okay, so you need to recognize this is a city.


This is already information that you kind of bring in into the model that just goes beyond language modeling, right? Services.


So the recognition that an airline is servicing a city that is again, it's kind of real world knowledge.


So I think that, you know, if you compare having these tasks here versus having just self supervised pre training, it is a bit deceitful that you say, well, we train on the same amount of tokens or we train for the same amount of steps.


Because I think the labeled examples are much more information dense because they sort of bring in knowledge that is not in the tokens themselves, but kind of in just like in the in the positioning of the label with prompt.


You bring in additional knowledge that, you know, it's not the same as simply a piece of text with that many tokens. That is one of the observations that I for myself could make right here why this is quite helpful.


So it conveys world knowledge or it conveys grammar knowledge or whatnot to the model.


By means of these labeled data sets. The second thing is that I do think we have a lot of evidence that especially the beginning of neural network training are quite important.


So you never in a neural network, you never quite get rid of your pre training.


So I think that explains this right here. We know that if you pre train, it doesn't matter how many steps of fine tuning and fine fine tuning and fine fine fine tuning you have the pre training, especially the one at the very beginning will always be somewhat represented in that model, it will always influence a model quite a bit, how you exactly pre train, or on what.


It's almost like, it's almost like you have kind of like endless possibilities to go as a model from your initialization, right. And then once once you've picked one, you know, once you've picked one of these directions, all you can really do is kind of find, you know, find wiggle your way into better local Optima, but they're all essentially in the same conceptual area like in the same direction right here.


And if you were to start with a different data set, you might choose a very different direction at the beginning one much more amenable to the kind of natural language inference prompts that you're going to do.


That is just that is like a hypothesis that could be right. That will kind of explain why pre fine tuning works less well than simply pre training from the get go on the multiple tasks.


So what do we learn initialization or early stages of training might be quite important, especially for like generative models like like these. They go further and they ask themselves, you know, since I've, I've shown you that they have.


In addition to all the supervised tasks, they have this top row right here, which is this self supervised tasks that you're used to from language model, it's formulated a bit differently with the prompt input and the prompt output.


But essentially, this is masked language modeling, like you'd you'd encounter in like BERT in the form of auto regressive language modeling, as you would encounter it in in a GPT model.


So, the question is, of course, you have two data sets. So even if you somehow managed to have these 107 data sets to get roughly like equal data for each one, how should you mix the self supervised objective with the supervised objective?