that the memory footprint is large, as small is not that important. But as I go down, sparsity


How to make your CPU as fast as a GPU - Advances in Sparsity w/ Nir Shavit

gives me two things speed with neural magic gives you speed, but it also makes the model extremely


small. So you're getting a small, accurate model, right running on a very small device. And this,


you know, typically is an ARM device. And so that's, that's, that's the audience that I'd like


to say, hey, we're coming, you know, we're coming in, we're going to deliver the same things that


we can deliver for Intel and AMD. We're now going to deliver it for ARM at the very end.


If you say edge, do you mean smartphones? Do you mean security cameras? Do you mean robots,


everything? Okay, I mean, everything. I not like I'm going to do everything to start with. But yes,


yes, we're aiming in that direction. Yes. And with the danger that this is become going to become


like a marketing opportunity question, but how easy is it to get started with what you're doing?


Like, let's say I'm a I'm like, I've done, you know, my TensorFlow tutorials, I know how to build


a model and train it and so on. Like, how much does it take for me to transition or to apply what


you're doing? Yeah, so you just go to our website, go to get go to get download, deep sparse, or,


you know, our engine download our ML tooling. And, you know, immediately, you just either pick a


sparse model and transfer learn onto it with our tool. So we have recipes, you have a model, you


have a recipe, exactly what you would do if you went to Hugging Face and downloaded a model,


and download a recipe, you do the same kind of thing. And you sparse transfer learn onto it,


and you're in business. So it's not very hard. So I think this is really, and we're working on


making it even even easier. This is one of our goals, right is to make it really, really easy


to do this. And the advantage, of course, is that, you know, people are already busy, you know,


quantizing their models to get more performance. So this is like quantize, in some sense, right,


you're going to do the same kind of thing and get a lot more performance.


Is there a type of model where it works particularly well and the type of model where it


doesn't like I'm thinking, you know, convnets recursive networks, autoregressive, maybe, you


know, the big language models, like what what is it best at?


Yeah, so right now, you know, it's best at at BERT, YOLO models, we do we do computer vision,


and we do and we do the language models, but not the large language models, we haven't done the


large language models yet. So for those types of things, like the BERTs and the YOLOs and the,


you know, the whatever the variants of efficient nets and all these guys, this is, you know,


visual transformers, these are the things that that we do right now. And all our technology is


right now, you know, available for those. I'd love to do the large models, a CPU is a natural


environment for running the knowledge models, you know, these giant models, these trillion or


whatever parameter models that people talk about splitting across 16 GPUs, they fit on your desktop.


Okay, so clearly a CPU is a natural place to run a very large model. Okay. And so that's that will


be a target but right but not right now. Okay, very exciting. Is there any last things you want


to get out maybe about neural magic or sparsity in general? Well, you know, our our our whole


machine learning software stack is open source. And we'd love people to come in and help us build,


you know, better sparsity use sparsity in their models and, and tell us about what they're doing.


And, you know, that it would we have a community and we'd love you to join us. Excellent. Nier,


thank you so much for being here today. So it was very pleasant. Thank you very much. Bye bye. Bye


Learning model-based planning from scratch

Hi there, today we're taking a look at learning model-based planning from scratch by DeepMind.


So, as a recap, what is model-based planning?


Basically, a model, also called an environment model, is just kind of a black box thing, you can imagine,


where you have a state of your current environment, you put it in there, and you have an action that you want to take.


You put it in there as well, and the environment model tells you what the new state, S', here,


and possibly also the new reward for taking that action is going to be.


So, of course it's always good to have such an environment model, because you can use it to plan ahead,


but the authors here question how do you plan and propose a new algorithm to learn this planning.


For now, people have mostly used kind of heuristics to plan either things like A-star search,


where you have a maze and you want to go here, and you kind of have a heuristic, say, the distance between the two points,


but there's kind of walls in between, so you try to go there, but then there's a wall,


and you kind of explore around it.


So, these are kind of the techniques that have existed so far.


Also, we've seen stuff like Monte Carlo tree search for AlphaGo and other things like this that are not really learned.


So, this kind of paper, pros and mechanisms to learn how to plan using such a model.


So, basically they devised an algorithm or a framework, you can say, where they have this, what you see here,


this schematic, and this schematic tells you that you have this thing called a manager.


Let me just quickly bring up my comment thingy thing.


All right, so you can see here, there is this kind of manager, and this manager can decide to imagine or act.


If it acts, then it simply takes kind of the current state and all the things that happened so far,


and decides on an action to do in the world, and then it kind of trains on the action like classic reinforcement learning.


But if it decides to imagine, it can use its model of the world, its imagination model,


to perform an action and see what would happen if it did that action.


And it can then also use that, append that to the memory, and use it to further learn,


even though it didn't do the action, it can imagine what happens.


So, how can it imagine? The authors in particular propose different methods of imagining.


In this graph you see their proposed methods. The first two methods basically, so here every row is a method of imagining.


The first method, the one step imagining, simply means you have the current state of the world, which is the grey blob here,


and what you do is you always go from the current state of the world, imagine one step ahead.


So basically, you select the state to imagine from, you imagine one step.


And if you decide to not take an action after that, but imagine again, because maybe you're not sure yet what you want to do,


so you want to imagine another action, you would again go from this initial state,


so this in the horizontal direction is time, internal time, basically.


You would again go from this state, imagine another action based on it, and so on, imagine another action.


Until you're satisfied, you've imagined enough so you can actually take a real world step.


In contrast, the end step strategy, so these are hard coded strategies as you can see.


It's the learned part is which action should I take? The hard coded part is where do I base this action off?


The end step strategy also selects the first state at first, imagines one action on top of it, but then always selects that new imagined action.


So you can see here it selects this one to propose this action, and then it selects that imagined action to propose yet another action.


So you can see it kind of imagines one path into the future instead of many paths, just one step ahead.


And then lastly, this imagination tree strategy is basically the only one that's actually kind of a learned strategy,


where the manager can now propose any previously imagined or real world states in order to imagine from.


So you always have the current world state, which is the first node in the graph.


You select it. Of course, at the beginning you have no choice. You imagine an action on top of it.


But then you can select any of these two nodes to imagine from.


And here again the first is selected, an action is imagined. Then you have three nodes.


You can choose any of those where you want to imagine the next step.


And here in this example, the manager selects this state right here and decides to imagine another action on top of it until it is satisfied.


And can then actually go over to plan to actually perform an action in the real world.


So if you then decide to do an action in the real world, what you can do is you can take all of the things you've imagined and use that.


So you see in this pathway here, this flows back to the manager. At some point it decides, OK, I've imagined enough.


And we can use all of these imagined steps in order to take a real world step.


And after the real world step, the entire thing starts again.


So that's how it learns to plan. Especially interesting of course is this imagination tree strategy where it's actually learned to plan ahead.


So the model is described in detail in a formal manner and then it already goes over to experiments.


And there's this kind of spaceship tasks where you have to, there's pictures, you have to get the spaceship to move around stuff and around these asteroids and get a reward.


So you can see different imagination kind of projectives here in the top row.


You see the red ones is the kind of executed actions, the blue ones are imagined ones and you see the tree it's constructed.


So first it takes an action right here just without imagining.


Then it imagines one step but then decides to take another action.


It imagines two actions but decides on a third one.


So you see to the left in this picture you see the first action.


Then it imagines one action and decides to take an action.


Then it imagines two actions and based on these imaginations I'm going to guess it's fairly satisfied with the one that's very close to the target.


And it can then take an action.


So you see it's pretty smart in that it sees that the second imagined action is fairly close to where it wants to go and it doesn't need to imagine yet another action that then actually hits the target.


It can go over to performing the action right away because the imagination gives enough information.


So these kind of things are pretty cool to look at and check out the more experiments if you want to know.


Here is even more experiments in discrete mazes.


They feature multiple goals.


They feature the system optimizing not only for its reward but also for internal costs, having a budget for imagining and optimizing, not doing too many imaginations.


On this experiment the kind of thing that bugs me here is the fact that they didn't actually use the full imagination tree algorithm but the manager only selected from what you can see here.


So act is do an actual action.


Then SJ0 is the first imagined state and SJK is the last imagined state.


So basically the manager can only choose between actually acting, then doing this one step strategy and then doing this end step strategy in each step.


So it kind of limits the way it can plan but I'm going to guess they did this because otherwise they couldn't have trained the model.


And it seems a pretty reasonable simplification to make in order to get this to work.


Also check out the paper if you want to see how all of these different parts are implemented.


Of course you can guess most of them are neural networks and it's pretty standard so far.


[ML News] Meta's OPT 175B language model | DALL-E Mega is training | TorToiSe TTS fakes my voice

Meta builds and releases a 175 billion parameter language model, a contrastive captioning model


out competes clip and the open source Dali mega looks better and better every day it


trains.


Welcome to ML news.


This video is sponsored by weights and biases.


If you don't know weights and biases, you're clearly missing out there in the number one


tool for MLS.


Whatever you do, they track your experiments, they optimize your hyper parameters, they


make everything observable, they track your artifacts, your models, your data sets, your


inputs and your outputs of all the things that you do there with you from conception


of your idea to experimentation to deployment and beyond.


It's really cool.


They enable students, they enable professionals, they enable researchers, personal accounts


are free forever as our educational accounts, but the extra benefits of weights and biases


for teams cannot be overstated.


Everything you do as a team is shareable, you can write up reports that you can share


with your teammates, they can comment on it.


And all of that is really cool.


They're in the cloud, but they do have options to host on premise if that is important to


you.


And they're just all in all a great tool.


They work seamlessly with a single line of code that you add to your script.


And from that they just track everything they have integrations with all of the popular


frameworks.


So there's no reason really to not try weights and biases.


Here's my link that's wanderb.me slash Janik to get a little surprise intro and also to


let them know that I sent you Thank you again so much to weights and biases.


This is really awesome allows me to do these videos.


And yeah, let's get into it.


Hello and welcome to ML news.


My name is Janik welcome to the channel we discuss the newest happenings in the machine


learning world.


In fact, so much time has passed since the last news that I'm having to split this episode


into two parts.


So you're seeing part one right now.


And part two is going to be released in a few days.


So keep an eye out for that Facebook releases a giant language model the same size as GPT


three, but they're just releasing it out into the wild, not entirely as we're going to discuss.


So this is the first thing where open AI gets serious competition from open source models.


So let's talk about it.


Meta AI has a blog post called democratizing access to large scale language models with


O P T 175 B. Now as I already said 175 billion parameters is the exact size of open as GPT


three.


Remember that GPT three is behind an API so you don't necessarily get access to it.


Now open AI has been building and improving GPT three over the time that it has existed


apparently or supposedly and the model we're getting out here out of Facebook is just a


straightforward language model.


So without access to GPT three, we can't exactly tell where the differences are.


However, in the papers, the author state that O P T 175 B is comparable to GPT three, while


requiring only one seventh of the carbon footprint to develop.


Now besides the blog post and the paper, there is a GitHub repository to go along with that


which contains the code and also the pre trained models.


You can see they release models starting from 125 million parameters all the way up to 175


billion.


Now you can get up to the 30 billion model just like that to download the larger models


you have to actually go and ask them for it.


They will share it with interested researchers, but they don't release it out into the world


quite yet.


So you're gonna have to wait on that just a bit more.


What is also interesting is that they published a log book of training this model.


Now the log book is essentially where the researchers keep track of what happened during


training of this giant language model.


And so there's a goal, there's a purpose, and there's some instructions.


And after that, you can find essentially logs of what people did, what they experienced,


what they ran, what problems they encountered, and so on.


So here you can see all kinds of stuff like people looking at the plots and finding out


interesting trends in the plots like repeated patterns and some metrics, you can find logs


of stuff crashing stuff trying to auto recover and so on.


In fact, many times these people had to rewind had to restart had to get their system out


from some kind of failed state and so on.


It really gives you a nice insight into the behind the scenes of training these large


language models because all we end up seeing is usually just the shiny paper at the front


and the nice results.


But reading this gives you a much better impression of just how much work goes into this.


So big props to Meta not only for releasing the models but also showing a little bit behind


the curtain of what's going on.


Though the best take on this goes to you of Goldberg saying Meta released OPT 175B but


have you heard anything of OPT 175A?


What are they hiding?


Couldn't have said it better.


There's a new paper called COCA contrastive captioners are image text foundation models


by Google Research.


This is a model that ultimately competes with clip among other things.


So the model is trained on the configuration on the left side right here there is an image


encoder there is a unimodal text encoder which means it only takes text.


There is a contrastive loss between these two encoders and then there is a multimodal


text decoder which means that it is essentially a language model that also gets the image


tokens as an input.


So there are two losses involved right here.


One is the contrastive loss between the encoders and the other one is the captioning loss from


the language model.


There are a number of special things.


The first one is that the unimodal text decoder is also an autoregressive language model which


is pretty interesting in itself because usually people use bi-directional models if they just


want to encode stuff but also the system can be trained once and then used in different


configurations for either fine tuning or even zero shot inference.


For example the image encoder will have very good representations for fine tuning a classifier


on top of it and the unimodal encoders both image and text can be used directly as a replacement


for clip in order to assess the alignment between text and images given their contrastive


loss training.


Of course given that the model is trained essentially as an auto encoder for the text


with the help of the image the model can also be used to do image captioning and other things


to do with connecting text and images where the output is text.


Here's a bit of a deeper insight into the model you can see that the image is tokenized


in classic VIT style whereas the text is first ran through an autoregressive decoder style


model even though it is technically encoding the text.


What's special is that we put a CLS token at the end usually it's put at the beginning


it doesn't really matter in bi-directional models but in unidirectional models and autoregressive


models we have to put it at the end to get the actual representation out the representation


of that CLS token and a pooled representation of the image tokens will be used for the contrastive


loss whereas the rest meaning the image tokens themselves and the text tokens will be used


for the multimodal text decoder.


In this plot right here in purple you can see the new model is called coca by the way


and how it stacks up against other models that are either not specialized just connecting


text and images somehow or even specialized model for something.


So the difference here are pretty significant sometimes.


For example this is the table on zero shot image classification on ImageNet.


Now zero shot can be achieved by these image text models because what you can do is you


can input the image and then ask the model to simply get you the distance to all of the


class labels as text.


It's actually a pretty neat way to do classification and you can classify into an open set and


coca beats the other models by a pretty good amount especially compared to clip in the


first row and you see just how much progress is being made in this field.


Again you see there is another competitor to one of OpenAI flagship models clip so alone


today we've seen a competitor to GPT-3 we've seen a competitor to clip and what's the last


one of OpenAI flagship models well it's Dali and as it turns out Boris Daima is leading


an effort to reproduce Dali out in the open.


Now the first model Dali Mini has already been made and in fact you can try it out it's


pretty good.


So this is the Eiffel Tower on the moon.


However Dali Mini as the name says is kind of a smallish version of Dali.


The new effort is Dali Mega which is a proper large model and the replication that resembles


Dali in scale and performance.


Here you can see intermediate results this model is training as we speak.


So on May 2nd it was 29% done and you can see that it's already producing pretty stunning


images with respect to the prompts that are given.


On May 4th it was at 45% and this prompt right here by Rohan Anil was apparently pretty difficult


for the model up until this point.


It is Spider-Man on a horse and yet it doesn't look too well yet and one person has actually


responded by inputting that prompt into Dali 2 and giving us the picture out of that or


at least that's what is claimed.


And these look pretty sweet I have to say.


So I'm not sure if Dali Mega is going to match Dali 2 in its performance.


It's certainly going to be a good model but I do feel that Dali 2 with its new architecture


relying on multiple internal models combining clip with diffusion models and so on.


And what I also suspect is that Dali 2 had very high quality data at least in part.


So I guess it's going to be difficult to reach that level of performance but still an open


source model that has such a good performance is quite cool.


So this project runs out in the open you can actually look at the report and the ongoing


experiments on weights and biases.


I'll link to it in the description check it out.


Dali 2's TTS is a multi voice text to speech system that is trained with an emphasis on


quality and emphasis on quality means it's very slow just so we're clear but it is pretty


cool version 2.1 has just been released and now you have the ability to use your own pre


trained models.


And I have to say this model is extremely good like it's very good.


Now there is a page with handpicked results and there is a collab where you can experiment


with the model yourself but the author James Betker has made a custom model for me and


sent me a little sample out of that model and you just have to listen to this.


I have never spoken this text.


In fact, this is a message that I sent to him on Discord and now it's just available


in my voice.


That would be fun.


Is this the model that is called tortoise because it's very slow?


Insane.


It's me is crazy.


I mean, imagine just the possibilities that open up with the ability to just clone voices


and let anyone say pretty much anything you want.


I mean, obviously there's going to be dangers ahead.


I mean, essentially, you can't trust audio recordings anymore where a person says anything,


but there's also really cool things ahead.


And in fact, the project does include a detector model that recognizes whether or not a given


sample was created by the tortoise system.


Now knowing a bit about adversarial examples, it's fairly easy to still use the system take


the output and then modify the output such that this detector will not be tripped.


But at least it is a first line of defense against people who simply mindlessly produce


stuff and then put it out into the wild.


But let me know what you think.


This is essentially a deepfake system for voices.


I think it's very cool.


Let me know in the comments.


This GitHub repository is very cool, probing VITs, vision transformers.


It's by Aritra Roy Goshtipati and Sayak Paul and investigates visual transformers and various


variants of that like the original VIT, the DIT and dyno and applies various techniques


to investigate these models.


They've also written this up in an excellent article on keras.io that really takes you


through the research how to interact with their stuff and reproduce their results.


So the questions that can be answered like this are things like what do vision transformers


learn or where in a picture do vision transformers pay attention to when they make a given classification.


All of these things can be achieved via techniques such as attention rollout, visualizing the


attention in an image, visualizing positional encodings and much more.


So if you're interested to learn more about how to investigate vision transformers, check


out the repository and this article.


Hugging face launches the deep reinforcement learning class.


So this is a class about deep reinforcement learning is fairly applied, but there's also


theory and the cool thing is you will actually be using modern code.


So libraries such as stable baselines three, which is not only for people trying to learn


reinforcement learning, but this is a serious library that is used in practice.


Now in conjunction with the hugging face hub, you can just publish the agents you train


and many people have already done so.


Now the course has just started.


So there's still ample time to join if you want to do so.


Obviously you can still go and read older stuff, but the next class will appear on May


And it's going to be a surprise.


Oh wow, a surprise.


Alright, a few helpful things for this week, squirrel is a library to load, transform, share


and generally interact with data sets.


So this unifies a number of ways on how to interact with data sets such as how to load


data sets either from disk or from distributed sources, then import them, transform them


in some way and then feed them into your machine learning pipeline.


And as you can see from their benchmarks on various data sets such as CIFAR 100, which


is images, Wikitext 103, which is text data set, they outperform other data ingestion


pipelines by quite a bit.


So check out squirrel core on GitHub.


PyScript is not necessarily a machine learning thing, but it is Python inside of HTML, which


is pretty crazy.


And this isn't just some gimmicky thing.


No, you can seriously pack your modules and then ship them inside of the browser run Python


in the browser.


There's even a two way interaction between JavaScript and Python.


So this makes for some exciting new applications that are now possible.


If you're interested, check out PyScript.net.


Big Vision is an open source version of the code base of a line of work starting with


vision transformers over MLP mixer all the way to locked image text tuning.


So all of this code is by the same or similar groups out of Google.


And this code base is the home for that line of research.


So check it out if you are interested.


It's always cool to be just a bit closer to the source of research than the finished polished


repositories we usually see out of papers.


Do you like sports?


Do you want to make some money and also get to publish a paper at a workshop?


These competitions here might be for you.


The fifth international ACM workshop on multimedia content analysis in sports hosts these four


challenges.


So there's ball 3d localization, camera calibration, instant segmentation and player re identification.


All of them have associated data sets and you can get started right away.


There's even some starter code available on GitHub for each of the challenges for you


to get into it.


The challenges are structured in two phases.


In the first phase, the winners go on and get to publish their papers in the workshop.


And in the second phase, there's actual money involved.


So the best team is going to win 500 bucks and the most innovative solution also wins


500 bucks.


And these two things can be the same team.


So that's a cool incentive to propose some innovative solution that is also very good.


Alexey Korshuk releases Hugging NFT.


This is a code base to train GANs on NFTs.


Now where have I seen this before?


This was literally released like one week after I got done filming for my GAN-FT video.


Now I went through the painstaking process of actually getting the data, getting the


code, training all of it myself, looking at the hyper parameters, yada, yada, yada.


Alexey releases a code base that makes all of this much, much easier because it's specifically


designed to interact with NFT collections.


So if you want to reproduce what took me multiple weeks to perform in a few hours, check out


this repository.


All right, here's our last article for the day.


John Deere is slowly becoming one of the world's most important AI companies.


This is by The Next Web and is an article about an interview with John Deere, not the


person John Deere, a person from the company, John Deere, about their advances into AI.


And I have to say it's pretty cool, whereas we still lack full self driving in cars on


the roads.


For tractors, this has long been a reality.


Not only can these tractors drive themselves, the farmer can just control them via an app.


It's really crazy.


Now obviously, this is promotional material right here, but I'm not really doubting that


they are already doing this.


What's crazy here is that the tractors are not only used for things like tilling, but


they can also remove weeds with very high precision as they do the tilling.


So pretty crazy what's possible.


And we've gone from a world where almost everyone was a farmer to where almost no one is a farmer.


And pretty soon, actually, no one's going to be a farmer.


Now I'm not sure we should probably not lose the last you know, one or 2% of humanity that


can actually produce food.


But I have to admit, it does look pretty sweet to have a driverless tractor.


Now wherever there is technology, there are hackers.


So this is tractorhacking.github.io, which is not a malicious hacking.


But apparently they say John Deere has overly strict security on the electrical component


of its tractor.


Sure, overly strict security on the electrical components of your tractor.


That's certainly a bad thing.


Oh, no security.


But they do have a point.


Obviously these vendors lock down all the electronics so that only they and their technician


can update them.


So this project is investigating how to bypass those things in order to repair those tractors


themselves.


So this already sounds a lot more reasonable than just the name tractor hacking, but I


still think it's pretty cool.


So if you want to take part there is a form right here.


I don't know what happens if you fill out the form, but you know, give it a shot.


And that was already it for ML news.


Thank you so much for being here.


Stay tuned for part two, which is going to come in a few days time.


See you around.


Bye.


The hidden dangers of loading open-source AI models (ARBITRARY CODE EXPLOIT!)

Well, what do we have here? Totally harmless model. I kind of wonder what it is seems to


be kind of a distilbert recent version of transformers flow 32. I like this model. The


Hugging Face hub makes it very easy to try machine learning models. So let's let's give


that a go.


Python shell. Import auto model model equals from pre trained. And let's go.


And what's happening? Oh, wow, it loaded the model, but it also opened a random website.


I don't know what this website is, but it seems very interesting. So if you actually


look at that model, then you'll see this is a normal model, it actually works. So this


is a model to distilbert model with all the weights, you can forward pass data through


it. So this would pass any test of being a machine learning model. But every time you


load it, it also does something else in the background. And that's what we're going to


talk about today, the dangers of loading untrusted models, how does this work and how you may


protect yourself against this? Just a quick aside, look at these binary number over here,


I want you to take the first four of each and just kind of go like small circle and


big circle in relation to zeros or one. So like small, big, small, big, small, small,


big, small, small, big, small. And that's the logo of weights and biases. Look at this.


It's actually pretty, pretty cool. So small, big, small, big. If you look at actually what


the number translates to an ASCII, it's w and b. I did not figure this out on my own


Scott pointed it out on Twitter, but he's been working at weights and biases for over


a year before he even realized it's just attention to detail. So I just think this is this is


very cool. You're in the middle of a sponsor spot. By the way, if you didn't notice, the


weights and biases is not just a product that I advertise, it's actually a product that


I use personally on a daily basis. And so should you weights and biases is a total solution


for ml ops from experimentation all the way to deployment and monitoring and it is for


everyone. academics are using it hobbyists are using it personal accounts are completely


free and academic teams as well. But it's not just for individuals, very, very large


companies are using weights and biases. Now if you happen to be a company small or large,


then there's great offerings from weights and biases for you. The weights and biases


cloud gives you an all in one solution. But if you're worried about where your data is,


you can also go with a self managed instance. And now there is an even better solution.


There is a weights and biases dedicated cloud. So what they'll do is they'll pull up an isolated


environment on a cloud provider and a region of your choice. And that's just yours. It's


managed by the weights and biases team, but it's fully yours. And if like most businesses


today you're on some cloud already, then this is an absolutely great balance between security,


privacy and flexibility. Head over to the link Wannabe.me slash Janek this lets them


know that I sent you and I promise you won't be disappointed. Again, thanks to weights and


biases for sponsoring this video. Really awesome to have them on board. And now let's get into


it. So how does loading a model from the hugging face hub legit hugging face hub model open


a random website on your browser as you load the model for that we have to dive a little


bit into how the mechanics of saving and loading models work. So the hugging face hub is super


popular obviously for sharing models getting models out there. And recently I've been trying


out a bunch of models on the hub for a problem that I had. So I just went through here. I


was like, Okay, I'm looking for image segmentation filtering down the models and it occurred


to me Wait, I'm just kind of downloading stuff and executing it is this safe? And it turns


out no, no, it's not safe at all. And the gist is there is absolutely nothing that can


be done about it. But with more awareness, I hope the situation is going to improve.


Alright, so how do models even get to the hub? And how do you download what happens


when you download them? See, if you create a model, if you make a model in hugging face,


and you want to save it either locally or on the hub to share it out, you use this function


save pre trained now save pre trained is a method on a model. And it takes just one mandatory


argument the directory you want to save it to now, how could that possibly go wrong?


Well, you can also see a little bit of the mechanics of how this works already from the


function signature. So optionally, it asks you for a state dict. If you don't provide


a state dict, it simply takes that state dict from the model that you want to save. So essentially


this saved pre trained function takes the state dict and then saves that now how does


it save it? It doesn't use JSON or NumPy or anything like this because well JSON is text


and is not accurate and NumPy is very limiting. In fact, since the framework wants to support


any kind of models that you might possibly think of, it needs a general protocol of saving


and restoring stuff. Now hugging face makes it pretty easy right here, it simply calls


this thing called the save function. And the save function by default is just torch dot


save. So hugging face takes the state dict and then simply delegates to pytorch to save


that and load it again, save pre trained calls torch dot save and from pre trained calls torch


dot load. Alright, we're halfway down the rabbit hole. Let's dig into torch dot save.


What does it do? Here's the pytorch documentation torch dot saves saves an object to a disk


file. Easy enough, you can see here, it takes an object to save no conditions on what that


object is, it takes a file like object, something that comes out of a Python open call. And


interestingly, it takes a pickle module. And again, you can already see a little bit of


how this actually works internally in pytorch documentation of serialization semantics,


it says they use Python's pickle file by default. So you can also save multiple tensors or objects


like tuples lists and dicts. And yes, if we look at the internals of the save function,


then we can see right here, here is that implementation, here is that pickle module. And as we scroll


down, we clearly see the pickle module creates a pickler and that pickler simply dumps the


object. So what you might say pickle is a standard module of the Python library, it


saves stuff to disk, and then it loads that stuff up again. Well, let me introduce you


to that last level of the rabbit hole. How does pickle work? Now you might think pickle


might be something like saving a file to to a JSON or a CSV or something like this, something


where you take the data and put it on a file that seems pretty straightforward. However,


pickle, as I said, is used to save and load arbitrary things in Python. And since arbitrary


things can be well arbitrary, you need an arbitrarily powerful protocol to save and


load things. So by necessity, that means this is Turing complete code. But let me show you


what I mean. See here, I have a little Python file, it has a dict. So there's a name and


a company entry. And then I simply dump that dict to a file using pickle. Alright, executed.


Now here's the code to load that very easy, open the file, pickle dot load, I should get


my dict back. And I do. But what is actually in that file, we can look at that file. But


that's pretty strange. As you can see, right here, there's a bunch of signs and then name


young company meta. So there seems to be a semblance of the data we put in there's stuff


around it. Now, Python has an internal module that you can use to actually dissect pickle


files. It's called pickle tools. So we use it to look at that file. And we see a little


bit more what's going on. You don't have to understand all of this. But essentially, here,


you can see that we first create an empty dictionary, then we load all of the data into


memory to here is name, young company, meta. And at the end, we call this set items function.


And we can already estimate that what happens here is first an empty dictionary is made,


and then it's filled up by that data. It seems to be very specific. And you probably can


only do that with dicts and not with arbitrary objects. So let's dig in a little bit deeper.


All right, let's get a little bit more complicated. Here I have a class, the class is essentially


the same as before, it takes a name and a company in its initializer saves that to the


local dict of the instance, and we'll try to save that class to a pickle file. All right,


done. And let's now inspect that file. But this is slightly more interesting. So again,


we'll have this closed curly bracket from before followed by the data that we gave it.


But now we also have this prefix right here, the class name. Interestingly, there's nowhere


really a definition of our class. And if we look at the pickle file using pickle tools,


you can see the ending is very much the same, there is a build call instead of a set items


call. But at the beginning, we also kind of have a main my class stuff in the code right


here, indicating that it tries to somehow create or construct or load that class. But


you see the general principle first, we'll try to kind of create the object itself. And


then we try to fill it in with the data. Now over here, I have the code to load from that


file. And watch what happens when I do that, there's an error, it says it can't find my


class. So actually, Python doesn't really store the definitions of classes you write


into the pickle file. However, at runtime, it tries to automatically get those classes


from somewhere and slowly it dawns on you, hey, pickle isn't just saving data to a file


and loading that data again, pickle is saving executable code. And when you on pickle something,


it actually executes that executable code, whatever that is, and you can nicely demonstrate


that. Alright, we'll go a couple of steps back, we'll have the original class here again.


So this is a class and it has an init method. But I've also defined this method right here


called reduce reduces in fact what pickle calls in Python, lots of things they will


call these Dunder methods on objects that hook into a protocol and reduce is the hook


to hook into pickling. So if I want to modify the pickling behavior of any class, then I


have to implement the reduce method. What does the reduce method return? Well, the Python


documentation says that the reduce method takes no argument and shall return either


a string or preferably a tuple. When a tuple is returned, it must be between two and six


items long. The first item is a callable object that will be called to create the initial


version of the object. So that means whatever you return from the reduce method, that's


the code that will be executed whenever you load the file back up. So the code that you


return here is stored as executable code in the file, which will then be executed. So


I have my class right here, it has a bunch of data. However, the reduce method simply


returns a list actually returns the constructor for a list needs to return a callable. And


the first argument to that constructor is the list 123. Now I'm going to make that object


as before filling it with data. However, if I save that object, watch what happens. So


I've done that. And just for giggles, I've also simply dumped the list 123. So my object


here should have like Jan and meta in it. But if we look at the pickle files, built


ins list, yeah, none of that. And pickle tools tells us yes, it's importing built ins, it


gets the function list, it fills it up with 123. And it depends that to the list very


good. Now the pickle file for the second thing where I actually just dumped the list is a


tiny bit different as it just constructs an empty list from the beginning and then it


pushes 123. But it's just a more efficient implementation of doing exactly the same thing.


And when I load the two objects up again, and I'm also emitting their type right here,


and I'm even checking if they're equal, then yes, in fact, I just have twice that same


list, even though the first one was a pickle of an object that had a name and a company


compute. So again, pickle stores objects by calling their reduce method, whatever that


reduce method returns is then executed upon loading. And it's essentially up to the goodwill


of people who make these objects or mostly to the default behavior of Python to give


you the correct result. However, this is fully executable code, and it can do whatever any


Python program can do. So why don't we just write a function that opens a web browser.


And in our reduce function, we simply return that as a callable, nothing easier than that.


Now we actually save it and load it back up, what happens? browser opens, there you go.


But you see, there is a little problem right here. As I told you before, we cannot simply


do this and then load it up in some other file because we've defined a class right here.


Most importantly, we've defined this open browser function that is not going to be available


if we upload to the Hugging Face Hub and then someone else downloads it, they're not going


to have that open browser function. However, according to the pickle file, that's what's


going to be called and it should be in the main module. So we'll need to get a bit more


creative to make sure that whatever we want to do is going to be available on any computer


that loads up our model. And secondly, you also see that the return type here is none.


So we've substituted saving our data and we can now open a browser. However, the user


is going to notice something is wrong because they're loading a file and is not actually


giving them the thing they want. Now we can solve both of those things with some neat


tools of Python called eval and exec. Python as you might know is quite dynamic. In fact,


it's so dynamic, you can just load up code at runtime and have Python parse the string


of code and execute it. Two methods here are eval and exec. However, eval only works on


expressions. So two plus two is an expression because there's a return value, it's four.


However, if we try to eval something like import web browser, it's not going to work


because that's not an expression. Import web browser is a statement, we need something


that executes statements. And that is exec. exec is another function that takes in an


argument and simply executes that thing import web browser, good. And now web browser is


available. However, exec is not exactly as eval. So if we exec two plus two, it does


it but there's no return value. But with a little clever combination of the two, we can


achieve anything that we want. So I've written a small library patch torch safe, very small


library you can install directly from GitHub, what you do is you provide a function that


you want to execute before any model loads. In this case, opening a web browser, it can


be arbitrary Python codes with import statements with whatever you want, you then call my module


with that function, which will return a patched version of torch dot save. And now you can


provide that patched version to hugging face in the safe pre trend. Remember, it takes


as an argument the save function that's usually torch dot save, now you simply provide that


patched function. And that's that if anyone loads your model from the local folder from


the hub from wherever it is, it will act like a normal model, it will in fact be that model.


However, as you load it, that side effect up here will happen. The whole library is


just these 21 lines of code, it's actually very small. So here's what I do, I get the


source code of that function you provide as a string, I strip away the top, so the def,


whatever I just want the body of the function I indented by one because I want this to be


executable Python code in sort of the top level. And I construct this thing called bad


dict. And I replace your dictionary that you want to save that you would give to torch


dot save with a bad dict version of it. And then I call torch dot save. So my function


is simply a proxy for torch dot save that wraps whatever you want to save into this


bad dict class. The bad dict itself has the reduce method implemented, it simply calls


eval as a function. The argument to eval is a string with source code, the string with


source code does two things. First, it uses exec to execute whatever the body of the function


you provided was, and then it simply returns an empty dict, which it later fills with the


items of your original dictionary. So line 10 really does most of the work right here.


And as you can see, it's astonishingly simple and allows again for arbitrary execution of


code. So whatever you could do in Python, any of these models could do as soon as you


call from pre trained and you wouldn't even know anything, they could be running some


crypto miner in the background, they could be running a key logger, anything that you


can think of. So what can be done about it? Pretty sad outlook if you ask me. Now if you


look into the documentation of the Python pickle module, it very prominently says the


pickle module is not secure only on pickle data you trust this will execute arbitrary


code during on pickling. So they're very clear what's happening right here. Hi torch itself


in torch dot load, they say warning torch dot load uses the pickle module which is known


to be insecure. It is possible to construct malicious pickle data which will execute arbitrary


code during on pickling never load data that comes from an untrusted source only load data


you trust. So both Python and Pied Parch are adamant about warning you of only loading


trusted code. However, on hugging face, I was so far unable to find any of these warnings,


not that they would matter much, I guess most people wouldn't read them anyway, but it's


simply nowhere. Okay, quick addendum to this video for releasing it, I've actually contacted


hugging face and make them aware of the problem. And now there is a nice banner, nice warning


in the hugging face documentation, I feel at some point, hugging face just going to


be full of features they implemented because I did something stupid, but very appreciated.


So there's now a warning and I'm going to be working with them to make things more secure,


at least to share the little bit I know all the while my model is being marked safe by


their malware scanner, but their malware scanner is only just starting to ramp up and it actually


looks kind of promising that some of these things can be mitigated. So I'm looking forward


to that. If you want to try out totally harmless model feel absolutely free. It's available


on the hugging face hub. You're also free to use this library here to create your own


funny models that do funny things on loading up and in the spirit of responsible disclosure,


I've actually contacted hugging face ahead of time here and weren't them and ask them


to maybe implement one of the suggestions. Again, there is very little that can be done


Can Wikipedia Help Offline Reinforcement Learning? (Paper Explained)

Can Wikipedia help offline reinforcement learning?


This is the title of the paper that we're going to look at today.


This paper is borderline preposterous in the results that it presents.


Language model pre-training helps reinforcement learning, which is crazy.


The two domains have almost nothing in common with each other,


and yet there seems to be some transfer from language to reinforcement learning.


And this is not just about pre-training on any old task.


The authors here have tried various things,


and there seems to be something special about language.


So here's how the video looks.


This video right here is a paper review.


It presents me going through the paper together with you,


explaining the paper, explaining what I think about the paper,


what kind of questions I have, and so on.


After this video, you'll have a good understanding of what the paper contains,


what its main claims are, maybe also what I think its weaknesses are.


In the next video, which will be released tomorrow,


I will interview the authors of this paper, which is very cool.


The authors will have seen my review and are directly able to respond to criticisms,


to any questions that are raised there, and this is so valuable.


We're able to directly dive in and get you the best possible insight


into the behind-the-scenes stuff and into the research process about this paper.


I invite you to watch both videos,


although feel free to choose whichever one you like most.


As always, let me know what you think in the comments,


leave a like if you do, and I'll see you around. Bye.


Hello there. Today we're going to look at


Can Wikipedia Help Offline Reinforcement Learning?


by Shal Reed, Yutaro Yamada, and Shixiang Shenggu.


This paper is a special paper because it very counter-intuitively trains a language model.


So it pre-trains a transformer to do language modeling,


for example, Wikipedia text modeling.


As you can see right here, language goes in, it does next word prediction,


like you're used to from a language model like GPT-2, GPT-3, and so on.


And then it takes that transformer and fine-tunes it to trajectory modeling.


This is a special subfield of offline reinforcement learning,


where decision transformers have recently been introduced.


So in offline reinforcement learning, you have some data set of trajectories,


and then you try to do reinforcement learning just given on that data set.


It turns out that if you pre-train something on language,


and then fine-tune it on these trajectories,


that will turn out to be a much better model, like a much more performant model


for getting you good reward at the end,


than if you just train this trajectory model here from scratch,


which is very counter-intuitive because it means that somehow,


the language modeling task, like the language model pre-training,


has a beneficial effect on the reinforcement learning tasks that comes later.


To note that the reinforcement learning task has nothing to do with language.


And even more special, they also try a bunch of other things,


most notably, they try to pre-train the image GPT model,


and that does not result in good performance.


So it's not just the fact that you have pre-trained on something,


and it is really very special result.


So we're going to dive into the paper right here.


The setup is fairly simple,


and then there is a series of experiments that try to investigate this phenomenon.


So they say that the offline reinforcement learning, as I said,


has been seen as a sequence-to-sequence model.


And I've already pre-annotated some stuff right here.


Let me know how you like that.


I thought I'd do it in this way.


So I have the green, that is the current one,


and the yellow is from my previous escapades on this paper.


So they go into offline reinforcement learning,


and that is being framed as simply supervised learning


to fit return-augmented trajectories in an offline data set.


What do they mean?


They mean the setup of the decision transformer.


I've made a video on the decision transformer.


If you want to look at that, you can go after you watch this video.


But so the decision transformer says,


well, see, you are an agent somehow.


There is an environment,


there is some interaction between the agent and the environment,


and in offline reinforcement learning, we usually have a data set of this.


So someone else has performed this,


and they've distilled all the episodes into this data set.


And their goal is to learn just from the data set,


we can't actually interact with the environment.


So in the data set, there are a number of trajectories,


trajectories of the agent interacting with the environment.


There's always some sort of a state coming back from the environment


or an observation, if you will.


The agent always gives some sort of an action back,


and then there is a reward and then next state


coming from the environment and so on.


So that is naturally a sequence.


And the sequence is there is a state, then there is an action,


then there is a reward and a new state, then there is an action again,


and then there is a reward and a new state.


So this is a sequence.


And since I have a data set of these sequences,


I might as well throw that into a big transformer to do sequence modeling.


Now, this has its own problems,


which I've all discussed in the decision transformer video.


For example, if the transformer has a context length of four,


it cannot conceivably like look back further than that,


which is a classic problem in reinforcement learning


how to look back and forward infinite times.


The decision transformer has the limited context window.


It has sort of the caveats of language modeling.


However, we understand language modeling very well,


and therefore, we are quite able to do that.


There is one modification that they do.


What they do is they transform the rewards right here.


They don't let the model model the rewards,


they let it model the rewards to go.


We're going to see that in just a bit.


This here is interesting.


What they say is that we look at whether transformer based


pre-trained language models are able to be adapted


to standard offline reinforcement learning tasks


that have no relations to language.


And I've already told you that this is going to work out fairly well.


And that's the special message of this paper.


Yeah, so they show consistent performance gains


and significantly faster convergence.


By faster convergence, they mean that a convergence point


like a non-improving the loss anymore


is reached after much many fewer steps


than if you were to train from scratch,


which makes sense for pre-training


if it's in the same domain.


But given that the pre-training is a completely different domain


than the fine tuning, that is still just a special thing.


So here is how we're going to frame the problem.


And if you've watched the decision transformer video,


this should be familiar to you.


We model a episode as a sequence in the following manner.


This is almost as we've seen it, except


the reward right here.


They are not individual rewards, but they are this thing right here.


The sum of all the rewards at this end, the next steps,


which they call the returns to go.


So this, for example, says from here until the end of the episode,


I'm going to gather 50 reward.


Now maybe you're in this state and you made an action that gave you


a reward of one, so then this here would be 49.


So you'd say, well, from here on out,


I'm going to make 49 reward and so on.


So the benefit of this is that at inference time,


you can just put like a really high reward right here.


So at inference time, you would always


you would model these things you would get from the environment.


So you'd start out with like just a big reward right here,


just whatever the maximum you've observed plus 10% or something


to just encourage your model to go very high.


And you plug the state in here that the environment has given you


and you let the model produce this one.


So it's important that at training time we do sequence modeling,


really model the sequence of returns and state and action


as a GPT like next token prediction.


However, at inference time, we obviously only predict the action


and the environment is going to give us these two things


or the environment is going to give us the reward.


And then we simply subtract the reward


from the previous returns to go and we plug that in here.


And then we plug in the state we got from the environment.


We let the model predict the next action right here and so on.


So this is very cool because much like something


like upside down reinforcement learning,


this is conditioned on like a desired reward.


This also has advantages and disadvantages,


but the advantage is we can control the reward


we want at inference time.


So we don't always have to go for a high, super high reward,


but we can.


Yeah, so this is the setup.


You don't actually need to understand much more.


But what we're going to do is we're


going to model this as a sequence in our data set


and then at inference time, we just put


like some high returns to go.


And that's it.


We're going to use a transformer for that for the sequence


model and they're going to use a bunch of different models


right here.


For example, GPT-2-small, which is a pre-trained model.


They also pre-trained their own that they


call Chibi-T, which is the same size.


So that is the same parameter count as the original decision


transformer to make it comparable to them.


So the decision transformer is the one


that introduced this transformer as sequence model


for reinforcement learning.


And they are going to see this Chibi-T model has


the exact same amount of parameters


as the decision transformer.


So they can directly compare what


the language pre-training is going


to gain them in the same model.


They also use CLIP.


However, they only, as far as I am aware,


they only use the text encoder part of CLIP


because that's an autoregressive model which


can do the sequence modeling.


And they use Image GPT, which is an autoregressive model that


goes via image tokens.


So an Image GPT, it would split up the image into pixel,


no not pixels, but chunks.


I believe either chunks or pixels.


I don't even remember.


And it would do a sequence model,


essentially go through the image like this, and then like this,


and then like this.


So it framed the image as a sequence


of either patches or pixels and go through it


as a sequence model.


So that's a sequence model too.


We can pre-train it.


And then we can apply it to this space.


They do various things right here,


other than just language modeling,


other than just language or sequence prediction.


Let's call that sequence prediction right here.


Other than just sequence prediction


for the reinforcement learning data, they do two more things.


First of all, they want to align the input representations.


So they have a set of language embeddings,


which comes from the pre-training data set.


Now obviously, the pre-training data set has a tokenizer.


That tokenizer generates tokens from the text.


And every one of these tokens will


have one of these embeddings associated with it.


So V is the vocabulary size.


However, obviously, in the reinforcement learning settings


there, we don't have the same tokens.


We don't have the same input modality even.


And therefore, we need a new, we don't need a tokenizer


because it's already tokenized.


Each of these things right here is a token.


However, what we do need is now a new vocabulary,


not a new vocabulary, but a new embedding matrix, so to say.


So we have a different amount of tokens,


so from one to the 3n tokens.


And what we're going to want to do is, what they say at least,


we want to have a set of linear projections


that will map the return embeddings, the action


embeddings, and the state embeddings


to be very close in their cosine similarity


to some embedding vector in the original setting.


So that means they want to force, not force,


they want to encourage the model to reuse the embeddings


that it used during the language model training.


So for each of the input embeddings,


they're going to find the maximum close,


the closest nearest neighbor in cosine space


of the embeddings of the original vocabulary.


And then they're going to encourage the input embedding,


the new input embedding, to be closer to that.


So that is just a loss that they add during training.


So you can see right here, this is


the loss for the language or the sequence modeling decision


transformer objective.


This is the loss that encourages the embeddings


to be close to the original language embeddings


or to one of the original language embeddings.


And this loss right here is the continuation


of language modeling.


So during training of the sequence prediction


for reinforcement learning, they additionally also do,


that's what they call language model co-training,


continuing to train jointly on language modeling


and trajectory modeling.


This allows us to encourage, this allows us to encouraging,


this probably should be encourage,


the model's transformer backbone to be


able to handle both language and trajectory simultaneously.


OK, maybe it helps.


This seems either like an idea that


had been had at some point or something


they had to put in after the fact just


to make it even a bit better or because maybe it didn't work,


though they ablated at some point.


And it also works without.


So that's almost it.


Yeah, they describe a little bit their baselines


and their setup.


I was a bit confused here.


It says it's a batch size of 65,000 tokens, which I don't,


like, I don't, is that, I don't, batch size is usually not


in tokens, like the sequence length would be in tokens.


But in any case, they say for our additional objectives,


we decay lambda 1 and lambda 2 to reach 0 after 5,000 steps.


We tune the initial values of lambda 1 and lambda 2.


And these seem, they seem reasonable.


But the fact that you have to, like,


decay the additional losses after x many steps and so on,


it points to a little bit of brittleness in them.


And I'm not sure always how brittle these things are


because reinforcement learning is traditionally


kind of a very brittle field.


So the main results we have right here, the top one


is four games in Atari.


The bottom one is, I believe, three environments


in the OpenAI gym that are, oh, sorry,


the, this is a data set, the D4RL data set.


All of this is offline reinforcement learning.


On top, you also have the 1% DQN replay Atari data set.


So as you can see, in many cases,


the, both the Chibit and the GPT-2, by the way,


GPT-2 is a lot larger than, so this


is a lot larger in parameters than the Chibit model.


And therefore, also than the decision transformer model.


So just saying that.


So here, the pre-trained models outperform the other ones


in quite a few tasks.


However, there is also Qbert, where they still


do outperform the decision transformer, as you can see.


But the, one of the baselines is just a lot stronger.


The other baselines are just useless.


That's kind of what I mean when I complain about,


when I complain about reinforcement learning


is that it is just weird.


Like a bit of a different environment


can make a large difference.


But as you can see, the pre-language pre-trained models


consistently outperform the decision transformer models.


Also, something to note right here,


this is mean and variance across three seeds.


So this is variance, I'm going to guess


they mean standard deviation.


And that is like a large number.


So if that's the standard deviation,


then the differences to the decision transformer,


they are well, well within that.


And that means, I mean, it is visible


that across experiments, we see the same trend, right?


That gives it credence.


But also, this just seems extremely noisy.


And yeah, I'm not going to say, I'm


going to sound like reviewer too when I say, well,


you should make more experiments to estimate


or to get smaller error bars.


But it just seems like, I don't know,


it seems like results that you can't really put a lot of weight


on because they're very noisy.


However, a bit like a little bit less noisy


are the experiments here on the bottom.


You can see that the standard deviations here


are quite a bit smaller than on top.


That's also three seeds.


I like how they wrote the number three here and the word three


right here.


That is just something that you never


see until someone points it out.


You can also see right here that the decision transformer,


for example, is rather consistently outperformed.


What's also interesting is that image GPT just sucks.


You can see right here, it just doesn't get anywhere


on any of these tasks.


Also, clip very often underperforms.


You can see, for example, here, clip underperforms.


And they do have some hypotheses on that.


That being said, there are still a lot of times


where the baselines here are quite a bit better,


or just better, than all of these transformer-based models.


So just pointing that out.


Yeah.


They do also analyze, and this I find really interesting,


the attention pattern between the GPT-2 pre-trained model,


the image GPT pre-trained model, and what I understand


is a randomly initialized model that has just been fine-tuned.


Yeah, randomly initialized model that has just been fine-tuned.


So there's no pre-training.


So all of these models are fine-tuned,


but the random one hasn't been pre-trained.


Interestingly, if you look at GPT-2,


you can see these bands right here.


And the bands are always in the distance of three.


So there's always three distance.


Now, three should be an interesting number


if you remember the sequence, how the sequence is made,


right here.


So there is always going to be one, two, three.


These tokens come in packets of three, right?


Their next return would be here.


The next state would be here.


The next action would be here.


So every token in this attention pattern


is most focused on multiples of three behind it


in order to predict the next token.


So there's always a lag or a attention


to multiples of three, which means that essentially,


if I want to predict the next return,


probably the last returns are the most important.


If I want to predict the next action,


maybe the last actions are important.


This might also be a property of the environment.


This is on Hopper.


So on these continuous control tasks,


I guess it's very often the case that I'm just


going to repeat an action for a while


if I want to achieve some goal.


I don't know the frame rate exactly of these things.


However, that seems to be something that is rather maybe


viable to do.


And therefore, looking at the last action


can give me a lot of clues about the next action.


Looking at the last state can give me a lot of clues


about the next state.


I would wonder how this changes if it's something like,


well, I don't even know, anywhere where I don't naturally


repeat my last action often.


You can see this is the early layer.


Then in the middle layer, the GPT-2,


it seems to sort of focus on particular states.


That seemed to be important, as you can see right here.


So this is where the attention comes from.


This is where it goes to.


And you can see that it kind of decides


that particular states are important.


And it kind of remains at that.


So it selects a few states that, or a few tokens


that it chooses to attend particularly to.


In contrast to that, image GPT seems


to have a large recency bias.


So if you see this right here, there's


really this band right here, which essentially means


that every token attends to kind of the few tokens behind it


in order to predict it.


Then, well, the question is, is it even worth


looking at stuff further down?


Because this model clearly doesn't learn at all.


So I would consider this and this just


to be kind of random noise.


The early layers might be interesting, though,


because there is kind of a pattern.


And maybe that is influenced by the pre-training.


So in image GPT, since you have your image,


and maybe it's in chunks, maybe it's in pixels,


but I can imagine that if I want to predict a particular chunk,


that maybe the last few that I've predicted,


unless I cross a boundary right here and go one line down,


the last few that I predicted are or might be particularly


worth looking at.


And rather distant chunks might be not worth


looking at very much, other than in language modeling,


where I often have to go a little bit more across the distance


and the exact neighboring words might not be as important.


So that might explain why image GPT


has this particular recency bias pattern in its attention.


What's also interesting is that the randomly initialized model,


look at that.


This is another interesting pattern.


And you can see that it's very much the same as in the GPT


example happens, except much more extreme.


So you have these rows.


For example, this row right here,


you can see there is a hard attention for three back.


Like there is really hard attention.


Then there are rows where you can see right here,


there is always these two.


And then these two.


With particular attention on the first one


and then also slight attention on the second one.


And that's kind of, it's a special pattern.


So no, I'm one off, sorry, in the one above.


So this is the hard three.


Then the one below is the, I'm going to call it the soft three.


So there is one strong one and one weak one.


And then the one even below that there


is like one semi strong, one weak and one really weak.


So what's happening?


I'm not exactly, so what I don't know here


is which of these tokens is returns, which ones is state


and which one is action.


But I'm going to just guess and I might be totally wrong


right here that the very strong bias here,


that is going to be the returns to go, which would only focus


on the last returns to go.


And then after that would be the state tokens.


So what the state tokens would do is, and you can see this,


I'm going to, I'm just going to.


So let's say this is the returns to go, the bright ones.


And you can see that in the state tokens, there is,


actually there is one missing here on the diagonal.


So this diagonal one here is just completely blank,


which means that it just kind of ignores the token behind it,


which is the reward, right?


So what it cares about is the last state.


And it also cares about the last action maybe.


I don't know how to interpret that very much otherwise.


So if I want to predict the next state,


I'm going to care about the last state and the action after that.


Maybe that makes sense.


If I want to predict the next action,


then I might be able to care about all of the stuff


beforehand a little bit.


Again, I don't know if I'm interpreting this correctly.


However, what I am able to say is


that there is a very, very structured attention right here.


There is this pattern of three is very prevalent.


And it is in general very, very structured.


So this seems to be actually the best kind of attention, right?


It is very structured in the way it looks at the information.


It learns exactly, aha, there is a structure to it.


I'm going to attend to the different parts


in this different structure.


However, my hypothesis is, and that is not super duper


discussed in the paper.


I mean, it is discussed.


But my hypothesis is that this bias here,


it might be almost like too strong.


It might learn the exact structure of this stuff.


But it might be too strong.


And it might miss information because it, for example,


says, well, I don't actually need


to know anything in between here because the most relevant thing


for predicting the return is the last return.


And therefore, I'm not even going to look at other stuff.


Whereas the language model pre-training just kind of acts


as a regularizer that says, well, you should maybe


look at all of the stuff, even though you don't find it super


useful in this particular data.


Now, one thing that I didn't point out in the video


that I wanted to point out right now


is that if you look at GPT-2 at the very left column, what


it does is it focuses particularly


on the returns to go steps.


It doesn't matter which step it is at.


It always kind of looks back at the very first token, which


is the returns to go of the whole episode,


and among other things also at the second and the third


returns to go token.


And this is important because the returns to go


is kind of an indicator of how the episode's


going to go along.


If the returns to go are low, it means


that entirely different episode paths


should be chosen in order to achieve that reward.


Whereas if the returns to go is high,


then I would have to do different actions


to get that returns to go.


So it makes a lot of sense to look at the returns


to go tokens, and rather than whereas you


can see in the right hand column,


the randomly initialized thing, it only really


focuses on the returns to go in these middle layers


whenever it needs to predict the next return.


And so it's much more diffuse, and it doesn't condition all


of what it does a lot on these returns,


where it makes total sense to do that.


Because in one instance, the language modeling


is just sampling any sort of high likelihood trajectory.


However, additionally in the GPT-2 case,


it is almost like conditioning that sampling


on the most relevant information that distinguishes


between the different futures.


I hope that makes sense.


It makes sense why a model that would


learn to focus in particular on this information


would be better at sampling appropriate trajectories


for the current episode.


All right, back to my comments in the past.


We know that language models retain


large parts of their pre-training


even during fine tuning.


So the language modeling thing might just


be like a very good prior.


And I wonder if we could build these types of priors


into the decision transformers if we didn't do language model


pre-training, but just as sort of like a bias or a regularizer


or something like this.


Yeah, you can see that through the random attention


at the end, you do not get this focus


as you get with the language model thing,


that it focuses on particularly interesting last states.


But you'd rather you do get like an attention


matrix in the last layer that is kind of diffuse


and sort of similar to the image GPT that


just doesn't work at all.


So yeah, that would be my maybe postulation


that maybe it is possible to achieve the same effect


by introducing the correct regularizers.


However, I don't know.


So they look at a few other things, which I just


quickly want to go through.


Because they have pre-trained, they can demonstrate


that their model converges much more quickly.


So instead of like three hours, their models of the same size


needs 43 minutes.


And their model that is a lot larger, I believe,


GPT-2 is 144 times larger.


It only uses an hour and 27 minutes,


so still half of the time than the decision transformer.


Now, I also wonder whether they have based their code base


on the decision transformer or whether some


of this difference is also due to just kind


of like a better implementation.


So yeah, that is that.


They have some analysis right here.


For example, they say they hypothesize


that a generative training objective is useful.


That's how they explain why CLIP might not be as effective,


because CLIP is ultimately a discriminative objective


or a contrastive objective.


They also say that there are underlying similarities


between language modeling and trajectory modeling,


where there is a large difference between image


modeling and trajectory modeling, which is a hypothesis.


They say, yeah, there is the language modeling


has a natural sequential nature.


Versus image modeling is kind of a forced autoregressive task.


I agree with that, but I'm not sure


if this is really due to like language being particularly


similar or whether, as I said, it might just be a good prior.


This would be an interesting question to investigate.


And it might ultimately turn out to be the same thing.


So interestingly, the context size


doesn't really matter.


You can see right here, if they increase the context size,


they do get worse, actually.


So yeah, that's worse.


It's just more noisy, which is special,


which actually means that these models aren't appropriate yet,


or we haven't really figured out how to appropriately use them


yet.


More information shouldn't necessarily


give you less of a reward, unless I


guess maybe you have a fixed size data set,


and therefore you have less training data points.


So maybe that's an effect of that.


Interestingly, the pre-trained models,


they do scale better, which I guess


you might have expected if you've been in deep learning


the last few years.


But if you just take a decision transformer,


it will overfit after a while if you scale it up.


So these are millions of parameters.


You scale it up, it actually gets worse.


Actually, I'm not sure if that's overfitting or just it


gets too big, and then the average reward decreases.


However, if you pre-train first, then it can handle,


and it will actually increase with more data.


Interesting would be to see if that at some point


actually declines again, or if that sort of holds up.


If the language model pre-training,


for which there is infinite data.


In language model pre-training, you


can get infinite data, and therefore it


could be that this just kind of gets you diminishing returns,


but not ever come down again.


Yeah.


They also experiment with freezing parameters,


and they say that this drastically reduces


performance.


So if they only train, if they only train, what do you say?


Only action state and return projections being trained.


So only this alignment of this projection of the projection


of the token embeddings are being trained.


That doesn't work much, which is also surprising,


because there is a lot of work that kind of shows


that you don't have to train many parameters


of these transformer models to effectively transform


or transfer them from one task to the other.


They say that this might be due to the task


of generative modeling being harder,


as opposed to discriminative classification,


where this was previously applied.


They have a lot of, yeah, they pose a lot of hypotheses


here of why things might be, and I feel each one of them


could be its own research paper.


Yeah, I'm going to leave it at that for the paper explanation.


Hope you got a little bit an intuition.


I still find it very, very special and very cool


that this even works.


And I think it's like a sign of the times of our models


just becoming the same models for all modalities.


This would not even have been possible a few years ago,


where every modality would use very different models,


like CNN for images and RNNs for language and so on,


although RNNs were used for RL already.


But given that our models converge


and we're learning so much more, this type of research


is really cool.


Yeah, let me know what you think.


Have we overlooked something right here,


like something that could easily explain why this works


and gives good results that just no one sees?


Or are there more applications for this?


Full Self-Driving is HARD! Analyzing Elon Musk re: Tesla Autopilot on Lex Fridman's Podcast

Hey, how's everyone doing today, we're going to analyze Elon Musk's appearance on the Lex


Friedman podcast. Specifically, we're going to look at the part where Elon talks about the


Tesla autopilot and to a certain degree also the Tesla bot. We've previously analyzed the talk by


Andre Carpati about what kind of architectures and so on goes into the Tesla self driving system.


And this naturally progresses over time. So Elon is going to drop some more hints here,


what exactly is going on under the hood. We're going to dive right in. Let me know if you enjoy


talk analysis or not. Who knows? All I know is that whenever you put Elon Musk on something,


you get insanely many clicks. So thank you for that. Autopilot.


I love how to go like autopilot and then both are like, yeah, as if they're saying like, yeah,


like, like, like that's ever gonna work. As you might know, autopilot is a bit behind schedule.


It's been promised again and again and again, especially the full self driving sort of autopilot.


But there also has been insanely much progress like no one is pushing that people have told me,


you know, other car companies are doing it as well. Yeah, but no one's kind of pushing it


quite like that. And sure, there are some risks to to go along with rolling out alpha and beta


versions just to users. But I mean, come on. So there was a natural skepticism when I first


drove a Tesla with the initial system based on Mobileye. Yeah, I thought there's no way.


So first when I got in, I thought there's no way this car could maintain


like stay in the lane and create a comfortable experience. So okay, so I didn't know that the


first system was based on on Mobileye, which is interesting because at one point during my PhD,


we got visit from a researcher who also worked on Mobileye. I won't name the researcher here,


because I might be about to tell some stuff that would get them into trouble. But they showed us a


video of themselves in a car. I remember this vividly and the car was just kind of opened,


the whole dashboard was opened, all the cables were like hanging out then going into some laptop


that was just kind of dangling on sort of the middle of the car, you know, where the stick,


I don't know what what you call that stuff in in English, it was like a super unstable setup. And


you know, cables flying around there and you everywhere. And then the camera kind of pans up


and you can see that car is on the highway, like middle of the highway cars here cars here, and just


driving itself, you see the steering wheel, no hands on it. And it was insane. Like when I when


I saw this, I never expected technology to be this far already. And yes, I know, in the 70s and 80s,


people have done self driving on highways, but still for someone to trust the system enough to


essentially sit there and let the system steer the car based on nothing but cameras was insane. This


system is just the beginning like the baseline for the Tesla system. I didn't know that. And I thought


it was an interesting story to tell. I was already super impressed by the mobilized system yet,


as you will see that this has been surpassed a lot. What are some insights you've gained over those


five, six years of autopilot about the problem of autonomous driving? So you leaped in


having some sort of first principles kinds of intuitions, but nobody knows how difficult the


the problem like I thought the self driving problem would be hard, but it was harder than I thought.


So like, I thought it'd be easy. I thought it'd be very hard, but it was actually way harder than


than even that. So what it comes down to at the end of the day is to solve self driving, you have to solve


you basically need to recreate what you what humans do to drive, which is humans drive with


optical sensors, eyes and biological neural nets. And so in order to that that's how the entire


road system is designed to work with with basically passive optical and neural nets


biologically. And now that we need to so far actually for full self driving to work, we have to


recreate that in digital form. So we have to


the argument here is I guess, if you if you want to solve the self driving problem, you need to


essentially do what humans do. And I'm not exactly buying this argument, like just because humans


only drive with vision, especially just because humans have, you know, neural networks, we also


must use neural networks that seems a bit shady. But there is a point to it, right, that the whole


road system and cars and whatnot are designed around human capabilities and and the way that


humans drive is designed around human capabilities and vision and audio and stuff like this. And


therefore, yes, it's good to drive if you have like a radar and a lidar and whatnot, that's


additional sensors, but you're not going to get around building in the human sensors as well. So


a car that just drives mainly on radar or lidar is probably good at you know, avoiding obstacles


that are just on the road somewhere, but it's not going to be able to see any signs, it's not going


to be able to sense of the of the world visually understand what's going on and things like this,


which you know, if something's speeding along coming along, and you can anticipate it by vision,


it's probably a lot a lot better than you having to somehow detect it on the radar. So I think


that's a fair point right here. But, you know, humans having neural network, therefore, we must


have neural network. I'm not super sure that's that's valid. How much game theoretic kind of


needs to be involved, you know, at a four way stop sign, you know, are as humans when we drive,


our actions affect the world, like, it changes how others behave. Most of the time was driving if you


you're usually just responding to the scene, as opposed to like really asserting yourself in the


scene. Do you think?


I think these, I think I think these could be sort of control logic conundrums are not are not the


hard part. The, you know, let's see. What do you think is the hard part of this whole


beautiful, complex problem? So it's a lot of friggin software, man. A lot of smart lines of code.


For sure, in order to have create an accurate vector space. So like, you're coming from image


space, which is


so I think Elon's gonna make the point here that what Lex is concerned is that, you know, there's


a lot of game theoretic stuff. And he mentions the four way four way crossroads, and then you


sort of have to communicate who goes first, who goes last, and so on. And I think that's a really


good point. And I think that's a really good point. And I think that's a really good point.


And I think that's a really good point. And I think that's a really good point. And I think that's


not that's not the big problem in self driving is going to make the point that once you do have an


accurate representation of the world, once you know where every car is, and so on, what every sign


means that you can figure this stuff out easily. And I think I agree, at least the number of


situations you can broadly cover with programming heuristics is sort of countable. And I would guess


that that would work though, I'm not super sure if that goes all the way because there is game


theoretic stuff like you can, you know, change a lane based on the fact that you know, kind of game


theoretically that other people won't sort of cut you off while you do it because they'd crash their


car and so on, which you can't just know by looking at their speeds and the positions of the cars sort


of the anticipation of how everyone else is going to react in certain situations is I think a big


part of driving and also a big part of sort of predicting dangers. So I'm not super sure if you


can just hard code all of that. But I think saying that, you know, the perception problem is


conceptually the harder problem, because for the perception problem, there isn't even an approach


with regular programming, right, you have to sort of learn it. And yes, if you if you make a mistake


in the perception problem, that's going to have vast downstream effects. So I do agree here that


probably the self driving problem might at least at this time, largely be a computer vision, or


let's say, not only vision, but sort of world understanding perception problem after that,


it becomes sort of easier. Once you have an accurate vector space, the control problem is


so much that of a video game, like a Grand Theft Auto of cyberpunk.


Yeah, yes, I want my I want my traffic management says I want myself traffic system to be the one


from cyberpunk, please. Lord help us, please. Yeah, I mean, point taken right? What Elon calls


vector space right here, I guess you you'd sort of call a scene understanding a scene graph,


you know, any anything like this, essentially, where are the objects in the scene sort of what's


their position, their momentum, I guess, you know, where are the signs? What do they mean?


Where are the traffic lights, all of this kind of stuff. Once you have that, the problem of sort of


planning ahead, what you should do becomes probably relatively easy, at least compared to that


perception problem, like when's the last time you looked right and left, and you know, or and


rearward, or even diagonally, you know, forward to actually refresh your vector space. So you're


glancing around and what your mind is doing is, is trying to distill the relevant vectors,


basically objects with a position and motion. And, and then and then editing that down to the


least amount that that's necessary for you to drive. It does seem to be able to


edit it down or compress even further into things like concepts. It's not, it's like it goes beyond


the human mind seems to go sometimes beyond vector space, to sort of space of concepts to where


you'll see a thing, it's no longer represented spatially somehow, it's almost like a concept


that you should be aware of. Like if this is a school zone, you'll remember that as a concept,


which is a really good, good point. So Elon made the point essentially that


what your brain is doing, and therefore what you know, the AI should be doing is take all


that information and build what Elon calls this this vector space, which is, as he said, sort of


objects and their motions. But Lex goes a step further and says, Well, you also know sort of


that this is a school zone. And in a school zone, not only should I be driving slower, but there


might be children around. So I need to be sort of careful, I in fact, adapt my attention and my


vision on different things than if something like benefits a highway. And I think that is, as of


yet, probably not considered by these, these AI systems, I'm pretty sure they that the input feed


is all the same, no matter whether it's a school zone, or whether it is a highway. Of course,


there's different things, as humans have limited amounts of attention. And Elon just pointed out


sort of all the ways in which your system is screwed up, like blind spots, and yada, yada,


yada. And that might be the reason why we have to sort of focus our attention on different things.


And you know, depending on where we are, so it could be that the machines are just, you know,


they don't care, they can always pay attention to everything. And therefore, this is not a concern


to them. I'm not entirely convinced by this, the, the sort of guiding of attention and sort of the


top down feedback loop to the lower systems, I think is as of yet completely missing from the AI


systems. I'm not sure actually, maybe, maybe they do sort of feed, let's say they know they're in a


school zone, they know, you know, the speed limit is such and such and, or there's a construction


site, maybe they feed sort of embeddings of this stuff into sort of the vision networks, and the


vision networks might be able to adjust sort of their, their attention patterns, not the


probably use attention, they probably use con nets or so, but it would be interesting to see if that


was happening, I would be very surprised if it was though. So not sure, this might be a fundamental


limitation, it might be that without this, the driving problem is essentially unsolvable or,


or there's, there's major hurdles that can't be overcome. It could also be that just, you know,


the machines can always pay attention to everything. And therefore, it just doesn't matter.


You saw that there were some kids about to cross the road in front of the truck. Now you can no


longer see the kids, but you need to be able to, but you would now know, okay, those kids are


probably going to pass by the truck and cross the road, even though you cannot see them. So you have


to have memory, you have to need to remember that there were kids there and you need to have some


forward prediction of what their position will be.


It's a really hard problem.


I mean, yeah, exactly. They're going to talk about occlusions here, occlusions, detecting occluded


objects and so on. But I think Elon's point is bigger than that. You need to have a forward


predicting model in order to do the self driving, you know, solve the self driving problem to a


realistic degree. And here I would, you know, challenge his earlier statement that once you have


the vector space, the problem is sort of, you know, not that hard. I think this particular part


of the remaining problem is actually quite hard in itself, because it's not like you can just


calculate the Nash equilibrium of self driving and then assume that everyone's acting rationally,


you have to sort of take into account all the human factors right here and how you expect other


humans to act be that pedestrians or other drivers or anything like this. Yeah, I think this is a


another area, this sort of forward prediction where neural net or in general, machine learning


is going to make a big difference. And then as I said, I'd be wondering if there is sort of a top


down feedback loop that as you're predicting forward, you're going to change sort of the


perception pipeline on the fly or not. But like, let's say you you're parked at a light,


and you and you saw, you use a pedestrian example that people were waiting to go to the


cross the road. And you can't you can't quite see them because of an occlusion. But they might wait


for a minute before the light changes for them to cross the road, you still need to remember that


that's where they were. And that they're probably going to cross road type of thing. So even if that


exceeds your time based memory should not exceed your space memory. And I just think the data


engine side of that. So getting the data to learn all the concepts that you're saying now,


is an incredible process. It's this iterative process of just when I just think so. So what,


so what he said right there, I think is is quite important as well. You know, you can probably


understand it in the concept if you do reinforcement learning, let's say you did reinforcement


learning in this thing, typically in reinforcement learning, we have a finite amount of time where


you can go back over time and still be able to do back propagation, especially if you're at like a


high frame rate, like these, these systems operate right here, that's not going to be a long time,


it's not going to be a minute of real time. And therefore, yes, if you need to learn to remember


something like their pedestrians right there, and they're still there a minute later, because all


the lights were red, that is going to be quite a bit of a problem and a challenge in itself. So


learning to remember things is a long standing challenge in reinforcement learning. And you


probably be better off sort of coding all the objects in this what Elon calls the vector space.


So so understand the scene and then explicitly representing each object that's there, rather than


having the neural networks learn everything from perception, I think the data engine side of that,


so getting the data to learn all the concepts that you're saying now, is a really important process.


It's this iterative process of just this this hydranet many.


Isonet.


We're changing the name to something else.


Okay, I'm sure it'll be equally as Rick and Morty like.


There's a lot of, yeah, we've re-architected the neural net, neural nets in the cars so many times,


it's crazy.