objects in images that you could stream those images like on a video so like to detect things


Intro - TensorFlow Object Detection API Tutorial p.1

at like 15-20 frames per second especially if you want decent accuracy is was a big challenge.


So and to write the code that's going to do this for you yourself would just take forever.


So this is a huge offering from TensorFlow. So thank you for that. So anyways let's go


ahead and grab this stuff. So I'm going to kind of assume that you do have TensorFlow


already installed. Again if you don't I've got tutorials on that. Also make sure you


basically have all of this. You can I mean you can install it this way but I would just


install these dependencies this way like pillow, LXML, Jupyter, Matplotlib if you're on Windows


obviously without the pseudo. If you do need admin just open up CMD as an admin. So the


only thing that might hang you up on your installation on Ubuntu is if you're not on


16.04. So they say 16.04 and it's true but if you're on like 14.04 for example like if


you're using paper space the deep learning in a box that's on Ubuntu 14.04. The only


thing that will hang you up there is basically your protobuf compiler and I will cover what


you'll need to do to fix that. So anyways make sure you have all these including Jupyter,


pip install Jupyter and then once you have those I think I'll take it from here. So the


first thing we want to do is get the models directory. I'm going to just clone as a zip.


If you're on Ubuntu just git clone github.com.central-models.git super simple. While we wait on that download


I'm going to come over here it's just github.com slash google slash protobuf slash releases


and these are all the releases for the protobuf compiler. I'm going to because I'm on windows


grab this one so for win32 save and if you need it on Mac for whatever reason you could


download that here. If you need it on Ubuntu you can download that here and the process


will pretty much be the same at least on on Ubuntu you would just extract that zip folder


just do sudo dot slash configure sudo make or sudo make check and then sudo make install.


On Windows it's even simpler you don't really have to do any of that it's a standalone.


So anyway I'm going to move this over here. So let's go ahead and extract here the models


and then while that's running whoops I don't know what have I done let's see. While that's


running I'm also going to extract here protoc which is just going to slap it into bin here.


Now I'm going to leave it there just because for this tutorial I would suggest you probably


take it and put it in like your C program files or something more useful than in downloads


at least if you're like me I clear out downloads relatively frequently and so eventually I


would just delete this by accident. But I'm going to leave it here and then we just specify


the full path so protoc is right in bin. So yeah. So moving over here once you have those


two things we're going to come back over here and pop into object detection. So the next


steps are to set up that you know point that compiler basically so let me just come back


over here installation. So it wants you to do the protobuf compilation so from within


that models directory it wants you to run this. Now again if you're on Ubuntu and you


run this and I forget what the error is but if you get an error you probably have an outdated


protoc you can check it by doing just in the command line or terminal protoc dash dash


version if it's like 2.4 or something that's the outdated version I believe the most recent


is 3.4 yeah so if it's not 3.4 and you get an error go ahead and update it that's probably


your problem. So anyway so we need to do this step as well on Windows so I'm going to pop


this over here inside models dot master I'm going to go ahead and rename this just to


models that's what it is and inside just straight up models you want to run the following


command so let's just go ahead and copy that come over here cmd right click now protoc


isn't going to you're going to get this protoc is not a recognizable blah blah so a lot of


people ask me about like this is like something people are like I'm getting this unrecognized


command well it's because it's not in your path you could add it to our path if we wanted


or I'm just going to specify the full path to protoc where did we go there we are I hope


that's big enough for you guys to see I think it I think it is anyway c colon slash users


is that right I think it goes users H and it was in downloads downloads slash bin slash


protoc I think that's right oh it was right there I'm blind anyway yeah so we run that


no errors great we're good to go then if you're on a boom to or probably Mac as well you're


going to want to export here to be honest I don't even know what that line is doing


someone comment below and tell me what the hell that line does I'm going to wager that's


not necessary on Windows and then finally you could test the installation but we're


going to risk and we're going to run the the Jupiter notebook so now we want to do is let's


go and change directory into object detection and then I'm going to do Jupiter notebook and


then I'll pop up the Jupiter notebook for us come down where's I pick there you go I


pine notebook and this is just a very quick demo basically what's going to do is it's


going to download and in fact let's just run all and I'll kind of explain what it's going


to do so run all what's going to do is it downloads a pre-trained model for you and


the pre-trained model is this one here which is cocoa which is stands for common objects


in context if I recall right I think it's 300 objects but I'm not positive but it's


just a bunch of common objects in context so anyway it's going to download that it's


a pre-trained model now later you could train it on your own you can basically train your


own model if you wanted or you could also train like you could continue to train their


already pre-trained models so depending on what your goals are one might be easier than


the other especially if you're like trying to detect similar like things it's probably


it's pretty much always better to go off a pre-trained model so it's done so this is


runs a couple quick examples that it downloads of images you got a dog and a dog and then


you got that image that was on there on their homepage basically so running this whole I


Python notebook takes a little bit of time because you're kind of setting everything


up but the actual detection process like this here like this loop basically like once you


go into the session image can be anything this is iterating through just images in the


test paths which just downloaded two images for you but this here this process this loop


is from what I found to be somewhere between like maybe 15 and 20 ish frames a second so


you can stream really well so anyway in the in the next tutorial I'm going to show you


guys basically how we can take this you know super simple demo basically and begin to modify


this loop here to do what we want on our own kind of images or video from either a webcam


or I'm going to be using just video from like grant of thought oh but you can use anything


you want it's already a pretty good object detector but if you want to detect custom


objects you'd want to train your own so I'd also like to cover doing that as well so anyways


that's all for now if you have questions comments concerns whatever feel free to leave them


Feature Visualization & The OpenAI microscope

Hi there, today we're going to take a look at the open AI microscope and this


article on Distill called feature visualization. So the feature


visualization article is by Chris Ola, Alexander Mortvinceff and Ludwig


Schubert of the Google Brain team, while the open AI microscope is by open AI. So


keep that in mind. These tools are tools for visualizing what neural networks


learn and specifically here we're dealing with image classifiers and even


more specifically image net classifiers. So you know image net is this big data


set with a thousand classes of images and the images are somewhat like 200 by


200 pixels and you're just supposed to put them into one of these 1000 classes


and networks have become really good at these kinds of things. So our question is


what do these networks learn and there are a number of very cool works that


have started to investigate what these networks learn. So this started with work


like Deep Dream or even before that but this is a very good summary and also


and also overview of this. So in this article they first showcase these


patterns here where you can see that as you go through the network in the layer


this is layer CONV2D0 which is a very low layer in the network you have


things what looks like pattern just pattern detectors so these are the


things that the network is excited and we'll just get we'll get in a second at


how you create these things. Just be sure these are the things that the these


particular layers in the networks are most excited by. So this layer is


super excited by these textures right then but as you go higher the network


gets excited by these kind of textures then it gets excited by more complex


things right. So the pattern is the higher you go in the layers and this


people have been seeing and claiming this and measuring this since RBF


networks and whatnot that that the higher you go in these networks the more


complex features that they build. So they always build hypotheses is they


build very complex features from the lower layers and the lower layers they


have less and less complex features and until you're the very bottom layer they


simply extract edges and patterns of texture like this whereas in the top


layers all of these are hierarchically assembled to give you very very intricate


features. So these usually look pretty funky that's why I like to investigate.


So this article focuses on how is this done and the answer is by optimization.


Now what do we mean? I actually have the article here somewhat printed out but


the graphics they don't really print very well to the notebook here. So


imagine this here. So what you want to do is you want to see how much


activation is in the network for a particular input and you can do this in


many many different ways. The easiest form let's actually start over here if


you have a neural network here and this is the softmax classifier and these are


the classes in this case let's say this is dog this is cat and this is car house


let's go with house. You can think of okay I want to know


when the network sees a cat what does the network think of cats so what I


would do is I would take an image that is just noise right just random noise


like this one on the left here and I would start to optimize using back


propagation. I would start to optimize this image. Now you usually know back


propagation as the thing that will optimize these weights if we have given


an image and a label right but right now you have to so and we optimize the


weights here sorry I should theta right but right now you have to rethink what


we have given is the label and the weights we keep them constant but we


optimize the input image to maximize this label as much as possible so we ask


of xx please please update yourself such that the output is as much cat as


possible right and then we just optimize for that and so we hope that this


picture would turn out to be as much cat as we like right and usually here I


don't know exactly which class they have here but you won't get a cat sorry over


here you won't get a cat or here if you optimize the log it's it's the same just


you do the same thing before the softmax what you will get is some weird trippy


thing so in our classifier you might get something like there's a cat here but


there's also one here right because two cats are more captain one cat and there


is like a giant cat head here and inside the cat head there is another cat head


and there is like a cat tail somewhere here and again this there's a cat eye


right here so you get like some sense something super trippy that is as much


cat as possible all right now so this is somewhat somewhat interesting because


you can find out what what does the network think is the most cat like thing


there is and most dog like thing there is but what you can also do is you can


see what do the intermediate layers of the networks what do they get excited by


so what you can do is for example you can take an individual neuron in one of


the layers right so this here might be a convolutional layer with its


convolutional filters right and these are the different channels of the


convolutional layer and then this thing here is just a single neuron there and


what you can do is you can you can you can say okay X we have now again we


input the image X and we optimize the image X such that with a given weights


this particular neuron let's call it n and n is and is maybe this neuron right


here right this particular neuron is as much activated as possible right so we


no longer optimize for a label but we optimize for a particular neuron to be


activated as much as possible and then we can sort of see what is an image that


activates this neuron as much as possible in this case that thing you can


actually do the same thing with an entire channel if you simply say when is


the channel as a whole activated as much as possible and something like a deep


dream did the same thing but with an entire layer in the neural network like


what is this layer activated by they can imagine there's not only one image that


activates it the most but probably depending on your start here it's it's


it's a you'll get different different results for depending on where you start


and we'll go into that later as well so these are the kinds of things you can do


to investigate these neural networks and see what they what they pay attention to


in each of the layers right so let's go on here and so they say they say okay if


we optimize by optimization you get something like this here on the bottom


row looks pretty funky right but what you could also do is you can visualize


by what is called data set examples and date with data set examples you don't do


this thing you don't do the optimization procedure but you go into your into your


database into your data set and you find the images that activate so the XI from


your data set that activate a particular neuron the most so you simply sort all


the images in your database and you just pick the ten or so that activate that


particular neuron the most and that would also give you an understanding


it's actually a valid thing and the AI microscope combines both of them so on


the bottom you see what particular neurons are most excited by and at the


top you see that that the the data set examples they're most excited by so you


can see that there is a there's a healthy diversity in the data set


examples but they also all kind of map to the to the what it's most excited by


it's pretty cool the last point they make in this article is that of


diversity with the data set examples you do get naturally sort of a diverse set


of images of course where you can guess you can also say okay whether the


maximum activations and only slightly positive you can even give negative


examples but with the positive you can only either maximize fully or you can


you know take the negative and you can say okay what is this neuron not excited


by at all but you you won't get kind of a spectrum of what the neuron is excited


by or the unit or the layer and what they're doing is simply they add a


diversity term they say that works best so what does that mean it means that


here if we optimize X and Y right let's go up here if we optimize X in that


right and try to maximize the activation of n we don't want to do X by itself but


what we want to do is we want to do an entire set of X I right so we feed an


entire mini batch in there and we want to maximize n but also maximize a


diversity term let's call that D between the X 1 to X B right so you want to


maximize the activation of the neuron but at the same time in your loss


function that you optimize you also have this diversity term where you say the


the images that you produce should be far apart from each other or kind of


apart from each other and thus you do get diverse samples okay the printing


again doesn't work so here you see that if you just simply optimize you get the


thing on the left but if you have a batch of things and you optimize them to


be diverse from each other but also activate the layer or the unit you get a


variety of high activations and you can see here that this is some sort of curve


this could be a beak of a bird but on the right here this could also be kind


of a snout of a monkey or something okay so it's just curvy curvy things that is


activated by here they give another example you can clearly see that there


is like some sort of eye in the picture but then if you optimize with diversity


you can see that some of them do not have this eye thing and in fact also


some of the data set examples do not have this eye thing so it might be


interesting to to to optimize here with diversity and even in they say in higher


layers it gets even more more diverse with what you achieve with this ball


detector they also say they research interactions between neurons where they


can interpolate between them right so you can here have two different near


units let's say this top left one here is the thing that we've just seen with


the curvature activator and then on the right we can select this thing here that


is appears to be activated by these bird like bird like things and if you


optimize an image that activates both of them you get the thing on the bottom


left and this is very good for understanding how neural networks work


because what a neural network will do is exactly it will take the thing on the


top left and the top right from lower layers and it will combine them to form


features of higher layers so while the top right thing looks like generic birds


the bottom left thing looks much more like birds with let's say long necks and


then kind of curved necks so more stork ish birds right because we've added in


this curvature thing so this is very very cool to play around you can also


hear interpolate between in two neurons like you would interpolate in a in a GAN


right and yeah so they do make a point of regularization I don't want to go


into that particularly but you have to be careful you can't just apply the


optimization procedure as I said right now you have to actually have to do some


regularization and to get to get rid of what are essentially adversarial


examples in this process because if you just straight up optimize you will get


pretty high frequency crappy results I actually want to jump over now to this


open AI microscope so this is a tool that lets you explore these visualizations


so at the beginning you can pick one of these models and I'll pick inception v1


just because some of the other ones they don't have everything done quite yet


the all the all the all the visualizations so on the right here you


can actually see the architecture of the network if you know what an inception


network is this what it looks like and you you would be able from here to


select one of these units straight away but I'm gonna sorry we're gonna go to


the left here so you have deep dream activated which means the entire layer


is optimized for so per layer on the right side you have an image here and


you can already see that if we go from the bottom what we saw before we get


patterns that become more and more complex as you go up the layers and then


more and more until you finally have what the network appears to be most


activated by is mostly dogs which is okay because ImageNet is dominated by


dogs so you can click on any of these right here like this one and now you'll


be able to in to inspect the individual nodes in this layer so before we had the


whole layer right the whole layer was activated by something but these layers


they have different channels and also different neurons within the channel so


you can select this here you can go neuron activation or channel activation


and these are the images that these channels are excited by the most you see


you get pretty funky pattern if we select one interesting one maybe this


this one right here you can see on the left this is the channel optimizing


optimization on the right this is the neuron optimization and here you get the


data set examples that are most activated that mostly activate this


particular channel or neuron so sorry this particular yeah channel so you can


see this is pretty similar to the thing I drew where except for it being a cat


it's some sort of a fox dog thing right and and you can explore the neural


network in this fashion so you can go through the layer here and look at that


good units this seems to be whiskers classifier and lo and behold things with


whiskers will activate the the neuron and as you go up the layer and this is


the the cool thing right so we're right now we're here in this layer for as you


go up you will see more and more intricate patterns of activations I I


could play around this for very very long time but I won't I won't waste your


time too much they there is a disk sorry there is a a slack workspace where


people discuss interesting patterns oh what is this okay yes this is a some


sort of temple temple constructor very cool there is a slack workspace where


people discuss interesting things for example they discuss how the car


detector that you see right here is one of the units there is literally endless


units to look at that detects cars can be clearly seen to be built from lower


level features such as this wheel detector you see this wheel detector


here is the unit 337 in the mixed for layer and this car hood detector right


here is unit 237 also in the in one of the layer force right so these are both


from layer 4 and then the car detector I've actually haven't looked this up ah


isn't layer 4 as well but let's check it out this isn't layer 4b this is in


layer 4b and this isn't layer 4c so you see I this was a risk the car detector


is built from lower level features of car hood and car wheel right the car


wheel right here detects wheels and the car hood detector detects hoods and then


the car detector detects cars so there are very like I really invite you to go


look at it check out what people find and explore these models all of this is


based on this lucid library right here also invite you to check that out where


you can perform such optimizations yourself I will link to that and with


Adding a tutorial option - Tkinter tutorial Python 3.4 part 19

Hello everybody and welcome to the 19th part of our Python with Tkinter programming tutorial


video where we're actually building a full-sized application.


So where we left off is we were basically adding all the menu items, kind of creating


the backend structure to how those will work.


Some of them are actually doing what their actual job is, some of them still need some


building up to actually realize what has been done.


So now we're going to add the last thing which is the tutorial option.


We're just going to kind of show when you go to help tutorial.


We're not going to actually write the tutorial because the program isn't complete yet so


there's not much point to write a full tutorial when that might not be the best tutorial for


the end result.


But anyway we're just going to show how we might do a tutorial.


So let's go ahead and get started.


So we're going to actually define tutorial.


Let me just cut that and we'll put that up here.


So tutorial no longer will pass.


We've got a couple of things.


We are going to go ahead and define page two as...


Actually I think on this I'm trying to think here.


Maybe...


Okay because we're going to start Tut...


So I didn't really quite think this through.


Because this one in my actual code I use that leave mini.


Well we'll change that maybe at the very end.


So for now we'll say leave mini what and then what.destroy.


So then define page two will be leave mini...


So in theory I think what I could do is just like tut.destroy and then we could do tut2


equals tk.TK and in fact let me just comment this out.


We might not need that after all.


So the load up tutorial will load a window and then it will progress into page two which


will redefine itself tk.tk.


And then we'll do define page three like this and then again we'll do tut2.destroy and basically


you would continue doing this all the way to as deep as you wanted it to go but for


now we'll just go to three pages.


And this will basically force a specific succession.


You might not want a specific succession.


You might actually want your tutorial to have like a navigation bar to it and people can


pick various options.


I've always been a huge fan of like clicking on a tutorial and having them walk me through


the application one step at a time and that'd be it.


I've always found this to be the most beneficial to me.


You might change your mind and if you do feel free to write your own little tutorial that


you think is superior and if it's good enough share the code with me and we'll let the user


decide if they want to walk through or more of a navigation.


You know you can always have two tutorials.


So page three tut2.destroy and then we define tutorial three.


So tut3 for example would equal tk.tk and basically what happens is around so like we


haven't actually defined the tutorial page yet and we will.


It will be like down here.


I pointed to the screen.


You can't see the screen.


It'll be down here and then page two's definition will have page three and whenever


we're ready for page three and because like basically what we'll have is we'll have page


two code here and then the button that you click next the command for next will be page


three which destroys page two loads up page three with whatever data is in page three.


That's how it works.


So what we'll do is we'll put some code here now.


So tut3 equals tk.tk tut3.wm underscore title we'll call this part three and we'll say label


equals ttk.capital label tut3 text will equal part three and then the font that we use we're


going to use just the norm underscore font like that.


Good enough.


And then we're going to do label.pack as usual with our labels we do side top fill x and


patty 10.


Why not one nine D ten.


Then we're going to have b1 equals ttk.button with a capital B that button goes to tut3


text is since this is the last one we're going to say the text is done and then the command


itself will just be tut3.destroy.


Easy enough.


Then we're going to do b1.pack and then tut3.main loop.


Okay so that's page three but now we have to define page two.


So this is what will run when the user clicks the same kind of structured button as here


and it'll run tut instead of running tut3.destroy it runs page three which just so happens to


be the first thing that page three does is destroy page two and then it runs page three.


So now that we've defined page three we need to define page two and page two will basically


be exactly the same as page three so I'm going to go ahead and just copy this and then put


it here paste and highlight this section hold control open brackets that should shift it


over if that doesn't work for you sorry and now instead of tut3 this is tut2 it's not


part three it's part two it's not tut3 it's two it's part two not done next and then not


tut3.destroy it's actually tut actually no it's not even any destroy it would be the


command is page three page three packs and then not tut2 main loop tut I mean not tut3


main loop tut2 main loop okay and now we define the main tutorial and so so for example


we'll just we'll write this one ourselves so let's say tut and then that equals tk.tk


and then tut.wm underscore title title equals tutorial and then we're gonna have label equals


cdk.label and as that's going to be to tut the text will be what do you need help with


and the font will be norm underscore all caps font okay now label dot pack as usual our


packs are always the same so let's go ahead and just copy this right here I'm not gonna


write that again paste good and now it's where we're gonna put in some buttons so and this


is what I was talking about before you don't actually have to have a walkthrough necessarily


like the user might just have a simple question so like for example we could have a b1 equals


tdk.button the button goes to tut and the text of that button equals overview of the


application so this is the one that's gonna walk them through step by step in a very linear


progression methodology that is what we basically coded up in this window here that I'm pointing


to that you can't see overview of the application and then command if they do that would be


page 2 okay and that's this function which immediately destroys the tutorial starts up


a new window loads up page 3 just in case we call it and then runs page 2 and you know


in theory gives a tutorial so or the second part of our tutorial so that's for button


one then we'll just say b1.pack easy enough so we don't have to type everything all again


let's just copy that actually that needs to be empty parameters there so copy this and


then paste twice paste paste now we're gonna talk about b2 b2 and this will be b3 b3 so


for b2 what do we want button 2 to do well button 2 maybe the user wants just to know


how to trade so how do I trade with this client okay the command here we're gonna say for


now it's just lambda colon pop up message not yet completed and then this other one


will be the same thing with the command not yet completed so copy that paste so they're


both you know commands not yet completed and then so someone might ask how do I trade with


this client and then someone else might ask a popular question that people want to know


is is with indicator questions help you know how to indicators work that sort of thing


another one is how do I get an API you know how do I acquire API information because to


trade on the various exchanges you have to connect to the API on that exchange so people


might ask that question and then finally when everything's all said and done however we're


gonna do touch dot main loop empty parms and that's that that is our tutorial function


so now well you could save and run that hopefully without error we'll see now we have help tutorial


they click on a tutorial this little window comes up what do you need help with they might


click how do I trade with this client that's not yet completed that's not yet completed


overview the application however they click that new window pops up part two we didn't


pack the button good job okay one second let's go pack that button oh actually we got a traceback


ah let's see if I can figure it out before we get there but I'm not seeing it oh here


it is here's our traceback right here figured it out guys so we packed that's the reason


the button didn't even show up either the button was packed to tuts three we do tut


two but we still might let's see if I can figure out if we have another traceback somewhere


I'm not seeing why we would or another error another traceback anyway we'll have to watch


for traceback help tutorial overview part two next part three awesome done okay that


worked no errors besides the syntax that counter nonsense awesome so um but this wording is


up here by the way is just that we have this these default values there's a couple of things


that we could do to like get away from this but we're just gonna leave them there for


now it's not like a it's not a world-stopping error as you can see the application still


runs it's really just more of a syntactical problem guido would be very unhappy with me


so anyways the tutorial is done now I mean it's not this is the back end so this would


be simple questions that people might have this would be like a walkthrough tutorial


so they click on that now they're learning about one part there it says you know click


on top indicator add RSI click 14 awesome you've added a top indicator okay so that's


part two then they hit next because they're done now you can add bottom indicators click


on bottom indicator back and so on okay so you can do that all the way through really


walk the client through how the walk the client how the client works anyway so that's that


so now we've got all of the menu options that I initially want us to cover and now we need


to start building those menu options so a lot of them are fairly easy but well some


of them are fairly easy the rest of most of these are going to require some pretty fancy


code and at this point this is where this program is going to heavily veer off into


weighing more so into the actual purpose of this client so up at this point we've really


built a backbone that really could be supported by a huge array of clients but now we're actually


going to be really building this client up for this very specific purpose of working


with Bitcoin trading and all that the main thing that I would say is still pertinent


to anybody else is what we'll end up doing with trading especially manual trading what


manual trading is going to do is it's going to open up another window so it'll be another


window but it's still part of the application itself so you'll have two windows mostly because


people may want to put you know their trading window over here the main graph window over


here you should let them kind of orient where everything is so and that was actually something


that people kind of voted on on reddit so that's the way we're going to do it so anyways


so that's that that's what you guys have to look forward to we're going to start working


on getting this graph to connect or this client to connect to the sea of BTC for the daily


data then working on the open ILO close actually adding the indicators that kind of stuff and


then after that trading that kind of stuff also pause resume doesn't actually work at


the moment anyway that kind of stuff so we're really going to actually start pulling this


application together and making something interesting out of it now so anyways got a


nice back end hopefully you guys are enjoying I was I've really enjoyed doing this really


the series and actually working on this application I didn't really know if you could do something


like this in Python successfully so I'm pretty confident we can major hurdle I think now


in the future is actually threading this application but I know with 100% certainty it's possible


I just have not done it so that'll be a hurdle anyway if you guys have any questions or comments


please feel free to leave them below otherwise you know the deal as always thanks for watching


thanks for all the support subscriptions and actually thank you for donations as well I


have received I've started receiving donations someone actually mailed me a check the other


day for this specific series so thank you so much for those so anyways stay tuned the


next video as always thanks for watching these are all the sports subscriptions till next


IT ARRIVED! YouTube sent me a package. (also: Limited Time Merch Deal)

໒ຌຈໄຈ່຀ໍ່໇ໄ໇໛໒ໆ່່໇ເ້໇ໄ໇່໇່໇່ໍ໇໇ໄ່່໇໇໇ເໍ້ໍ໇ໆໍ໇໇ໍ່໇່໇່໇໇ໍ່໇໇່໇ໄ່໇ໄໍ່໇່ເໍ່ໍ່ໍ


ប្ន្រារារាន្ន្ន្ន្ន្ន្ន្ន្ន្ន្ន្ន្ន្នន្ន្ន្ន្នន្ន្នន្ន្នន្ន្ននាធ្រាដងហ្ន្ន្រាញំល្ន្ន្ប្រាំលាញ�


very shiny and this part in the middle is like a mirror. You see this? Amazing. It is super cool. I'm very, very excited. This arrived, I would not ever have believed when I started that that this would at some point be in the mail. It's incredible that 100,000 of you are interested in very lengthy and long winded explanations about ML research or news in this space or anything like this. So giant thank you to every single one of you.


That is subscribed. If you're not subscribed, what are you doing? The button is right there somewhere. No, but really giant thanks to the people who watch you come back to watch content, all the people who leave a comment, I do still read, try to read like every single comment. I don't always respond, but I do take your suggestions very seriously. And very big thanks to the discord community, especially to the moderators there. They, they have a huge


job, there are spam bots and whatnot. So giant thanks to the moderators. They also organize paper discussion. Every Saturday, we have paper discussions. And these are like some of the most valuable times because I end up learning so much myself. And it really helps me sometimes for new videos that I put out where I directly take well, you know, people's opinions from there and try to integrate it into the video. So you know, giant thanks to all of that community. Anyone who's


ever helped me out all the authors who've come on this it's it's been extremely rewarding. And I hope I can I can keep up the content. I hope I can continue to deliver good content. It's not so easy on YouTube, because you have to change in order to stay interesting and relevant. You have to go with the times but still you have to keep the essence of what makes channel great. And this is a challenge. And I'm relying also on you a little bit to tell me what's good, what's bad.


What works. I'm also going to try some new stuff. I hope you've enjoyed so far the more inclusion of the original authors of papers. I think that's super valuable. The ML news, it looks like it's more click baity, but and it's less work. But it's actually also I really enjoy making the ML news. But also it is more more time goes in there scheduling the authors. By nature, I'm not an organized person. So scheduling people and keeping up and sending them to the next page. So I'm really looking forward to seeing what happens.


They've been patient like, let me totally work on it. Well, you can definitely spend a lot of time with the people. But when I speak withiająji's people whose?'re watching this and also from in our own numbers, then going天 kara appear man hotspot a lot of sill


So this is the rating 핑 s on all the stories on the comments that this has been tabled that. Before that, and then some cost points that I would definitely link on the description because I find it a great idea to Essentia II for find that赶 B


go forward together. With that being said, you might have noticed something else which


the people over the years have asked me again and again for merch. This honestly, it's more


for myself because I just think it's fun to walk around with the channel logo on like


a hoodie or something. But if you want to support the channel and want something a little


bit in return, merch could be an option for you if you enjoy these things. Alright, so


I'm going to show you some of the merch right here and we should talk about prices. I just


came to this website, it's called Teespring, I think now it's called Spring and I just


left all the default prices at their setting. Now the idea is obviously that it isn't just


a markup for a regular clothes retailer, it is a bit more because the idea is that you'd


support the creator. However, that makes the merch kind of pricey. So for people who can't


afford this, I've decided that for five days after this video goes up, all the markup will


be set to zero. Like I will not make a single dollar off of this merch if you buy it five


days after this video goes up. Now if you have already bought merch like this, I've


activated the merch shelf a while ago and you would like to make use of that, you know,


contact me, we can for sure work something out. If you do want to support the channel,


you can become a Patreon. I have several ways of supporting me, all the links are in the


description or you just wait for a week and you get the merch then. But I just thought,


you know, if you want to run around advertising my channel then and you don't have much money


or you'd like three t-shirts instead of one or instead of two, you know, knock yourselves


out. Yeah, so just five days and we'll do things like this in the future again. This


won't be the only time where the merch is reduced and there will be other merch coming.


I'm looking for like sunglasses merch which is hard to find, I can tell you. And I'm also


working together with a bit more, let's say professional designers to get more, just kind


of more extra extravagant merch out there. Again, five days, markup zero. After that


I'll set it back to the default values. Look at this, the ice is so thin but still the


birds they're just insane. So I'm wearing one right now. I had to do how to have a


few iterations people who follow me on stream saw the first iterations. This is kind of


the second iteration. I wanted to make sure everything's placed nicely before I shouted


out so he has the logo in small right here. This is a hoodie. It is a small European and


extra small US. I don't know why they differ these sizes. I'm not too tall of a person


and it fits kind of kind of snuggly. I'm like 175 for you Americans. That's like a num


some number of feet. The same design also exists in black as you can see now this was


the first iteration so the logo is too far out here. So in the new iteration, the logo


will be a little bit more inside. It's almost like under the arm right here. But I do quite


like the the white logo on the black background makes it pop a little bit more. There is one


person one of you has actually bought a first iteration hoodie after seeing the store on


stream if you would like that replaced with a newer iteration hoodie or just if you would


like one. I'm very happy to send you an additional one please contact me because I feel kind


of bad because yeah it is it is a bit out of place but rest assured if you get the black


hoodie now the logo will be in the correct place and it looks poppin by the way there's


nothing on the back of any of these. I've opted for kind of smaller logos so that it


doesn't look like traditional merch. However, as you can see we also have the large logo


available. Again, this is an S. I am a small person but I have a bunch of shoulders. This


fits kind of snuggly here. It is it is okay. If you're taller than me I definitely suggest


like an M. We also have t-shirts with the smaller logo design if that's your favorite.


These are also available in dark and we also have this design right here. Now this is


the channel motto which you might have never actually seen directly. It is not something


that I've shouted out in particular but I think the design looks cool and there is a


little story behind it. When we were at the end of high school we used to play a lot of


online poker which was sort of at its peak back then and we used to play online and also


circulate in poker forums where people discussed strategy and things like this. We always took


sort of a statistical approach because essentially you're playing towards an expected value and


you're trying to be as mentally robust as possible against the variance that inevitably


comes. So at one point there was this one player who just let off like steam in one


of these forum posts essentially saying that the world is against them they always get


the bad cards and if they have good cards the opponent always gets lucky and it's just


every time it's happening just kind of the entire universe is conspiring against them.


That's why they lose right and it's unfair and they were just really really really ticked


off and one of the people who responded was this very high ranked players one of the highest


ranked players at the time he just responded with this one line skill greater than destiny


and I just thought that was really really cool. I'm not a deep philosophical person


or anything like this it resonated with me since then I've took it up as a little bit


of a motto a little bit of a mantra to live by and the meaning of it is obviously subjective


but to me I've interpreted as something like it doesn't matter how much the world is stacked


against you how much your destiny has chosen a path for you that is not good doesn't matter


if the system is rigged against you you can overcome it by working hard by putting in


all your effort in fact it doesn't matter how the world is you can't change that you


can change yourself and you can try to do the best you can yeah if you're smart work


hard and obviously a little bit of luck is always a of essence but independent of how


the world is structured you should do your best and that's just something that I think


is nice to have somewhere around every time you look at it it kind of reminds you that


oh wait I'm just gonna try to do my best today and not get mad at how unfair the world or


the system is to you and the absolute cool thing is if you get the zip up hoodie you


can like double represent look at that yeah we also have this beauty right here which


is actually it's a crop top you can't even see it so again the logo here will be placed


in the current iteration more inside more on top a little bit smaller but I think looks


pretty cool so if you're interested check out the store it's available at store.whykilture.com


there's a link in the description there's also a tab directly next to this video we


also have other stuff other than just clothes for example there is the beaker right here


now the logo again is a bit tall here a bit large so we're gonna make this a little bit


smaller but in essence this is a cool beaker it holds half a liter that's like some gallon


for Americans it really keeps stuff warm on the inside and the lid is a it kind of pops


off like this and it has an it has a seal on the outside so it's not screw on press


on there's also other stuff such as cups and these right here pillows so I have these


two in different sizes so they go together they go together nicely on it on a couch but


I don't I don't know who wants these but I find them I find them hilarious and with that


being said thank you so much for being here for continue to watch continue to enjoy and


most of all you know I really appreciate all the people who helped me who gave me feedback


I still try to read every single comment what you people post is really valuable and shapes


the future of the channel and I hope we can continue doing that indefinitely with that


being said I wish you an absolute pleasant rest of the day and I'll see you bye


have I told you that I quite like hoods I don't know what it is but something about


hoods it just it's snuggly and if you have very short hair the hook kind of turns with


Can Wikipedia Help Offline Reinforcement Learning? (Author Interview)

Hey, this is the interview part of the video, Can Wikipedia Help Offline Reinforcement Learning?


If you haven't seen it, I've made a comprehensive review of this research paper in the previous


video. So be sure to check that out. The authors that I speak to today are the authors of this


paper, they've seen my review, and they're ready to dive in and tackle all of my criticisms. It's


a big privilege to have the authors on and to be able to ask them any questions. So please let me


know how I'm doing. Let me know how I can improve these videos for you. And as always, if you like,


leave a like and I'll see you around. Bye. Hi, everyone. Today I'm here with Michelle


Reid and Yutaro Yamada, who are the authors of the paper Can Wikipedia Help Offline Reinforcement


Learning? First of all, both of you welcome and thank you very much for being here and discussing


the paper with me. Thank you. So I obviously, you know, the basic ideas of the paper I've mentioned,


what would interest me is just how would you pitch the paper if you had to pitch the paper?


Let's say someone comes up to you at a poster presentation or something like this. What would


be your initial pitch, like whatever, 30 seconds or a minute, the basics of what you do?


Well, I'll give it a shot. Let's see. So here in our paper, we look at seeing whether, say,


Wikipedia or language retraining can help other sequence modeling tasks. And in this case, we


focus on offline reinforcement learning. And I found this to be personally like a pretty cool


project because essentially the reasons are not completely clear, to be honest. But we see that


with this language retraining, we can actually see quite substantial gains in certain areas


over like regular, like random initialization. And I think even more interesting is that


these models managed to converge faster, which shows that there is some sort of information


there that is helpful. And personally, I'm pretty interested in this line of research because


it really begs the question, like, how are these seemingly unrelated tasks similar? Is there a way


to see how similar they are and maybe even encourage like a new paradigm for transfer learning


where you don't even need like conventionally related data?


How did you, you mentioned it a little bit why it's interesting, and I completely agree,


and the results are astounding, I would say, how did you get the idea to do this? Because initially,


if someone told me, you know, you just pre-train something on language and then use it for


reinforcement learning or something like this, you'd dismiss it quite quickly, let's say,


of all the ideas that you could choose from. So how did you like, did you have some indication


that this could work or a hunch? Or did you just try it at some Saturday morning? Like,


how did it come about? Sort of a mix of all three. So like, I guess as a background,


we have that, like, say, in multilingual learning, it's been demonstrated by a couple of papers now


that say you can transfer like an English BERT to a Spanish BERT, for example, or you can like add


new languages to, like, say a model where it wasn't pre-trained on those languages. Or even there's


like an experiment in the M BERT paper, I think, where they have like this ablation where they


pre-train on like six languages, and then they test on like some unseen languages, if I remember


correctly, and that works too. So like in the multilingual settings, this sort of intuition


has been demonstrated, though you could argue like, oh, it's language to language. And then


I was talking with the other author in this paper, Shane, one day we were just chatting and we ended


up talking about like pre-training for RL, and I was like, huh, there's no pre-training for RL.


Like, they haven't had like their BERT moment or their GPT moment yet. And we were discussing,


he was like discussing the limitations. And then I was like, why don't we try doing a language model?


And then, yeah, and then it became sort of like the Saturday morning experimentation session.


Which you like alluded to, which is like, that day I was like, okay, let me just try putting


the language model there and see what happens. And the initial results were actually quite


surprising in a good way. So we decided to continue doing that.


Oh, I was going to just add on to like, I remember you, Marshall was saying that when Shane's first


reaction was like, there's no way that's going to work, like that sort of, you know, like,


I don't think he was really excited about the idea. But like, when Marshall actually did


experiments and showed the results, he was like, really, yeah, excited.


And you, the basic concept here is, I think it is very simple. And therefore, the sort of the


setup of the paper is very simple, right? You pre-train on this language modeling objective,


and you make a point that it is the autoregressivity might be somewhat important


right here in what you do. And then there is this decision transformer on the right hand side.


Now, did I, I don't know how much you've seen of my introductory video, but did I get anything


wrong in the setup here? Or did you, did you want to highlight a specific part of this? Like, why


could language models be particularly useful for this kind of reinforcement learning offline,


offline reinforcement learning with decision transformers?


Right. Yeah, I think you captured it pretty well. I guess like we'll go deeper into like,


sort of maybe the reasons why this would work as we go deeper into the questions, but like,


as like a high level idea. Yeah, I think you captured it pretty well.


I was always just maybe as a side note, I was always a bit astounded by these decision


transformers by the whole approach of doing this as kind of this sequence modeling with this fixed


context size and these returns to go and then I just essentially say, well, I just want like a


really high return, like just get me there. It seems very special, but it seems to work. I don't


know if you have any thoughts on this, not necessarily related to your paper, but I do


find it a very special model for reinforcement learning specifically.


Yeah, like for sure. Like, actually, I was experimenting with like,


trying some higher returns. I don't think we included in the paper. But sometimes,


like, especially during early stages of training, you could like get free returns almost by just


using like an artificially large returns to go value. And then suddenly, like the model would


get better at test time, for example. So yeah, this, I think it's pretty amazing, honestly.


Maybe shows something about the power of transformers to sort of like gather sort of


ideas or like states together and combine them in interesting ways.


I think we can directly go a little like into the results. Because as I said, the setup is quite


simple. Now you test on two different on two different data sets. So just to remind people,


we have the decision transformer, which kind of serves as the baseline for what we're trying to do.


That's sort of a same model with the same technique and the same inputs just not pre trained on


language. And then there is this, if I pronounce this correctly, Chiby T model that is the same


size, but has been pre trained on language. And then there's GPT two, which is a lot larger,


and obviously has been pre trained on language. And then you have some some baselines over here


that are just for offline reinforcement learning. Now, you can you mentioned that your models


consistently outperform or the language pre trained models consistently outperform the


decision transformer. But one of my worries here was that the standard deviations, especially in


this experiment, they seem they seem ginormous. This is there, like, how can we be sure we're


not just measuring like it's better in the in the bottom table right here, but on this DQN benchmark,


how can we be sure we're not just measuring noise in these cases?


I would say, well, we can't be sure. But I would I could say I would say that like the trends across


experiments do tend to like point towards a certain direction. And also, like this was like,


I'm generally like a language person. So when I was coming to RL and I was saying, oh, well,


we just changed the random seed. And it changed by this much. It was quite surprising to me.


But after like running experiments many times, it seems the trends were towards one direction. But


I guess we could clarify that with some like significance tests. Yeah, I was I was I think


I was mentioning that that the trend is in one direction. I think that's that's much more


convincing than, you know, anything being inside or outside of some standard deviation. What


surprised me also is that they're just I think that's just a property of reinforcement learning


as such. For example, this the Qbert environment, all of a sudden, you see, for example, there are


baselines that just fail, like they just nothing right. And then all but all of a sudden, these


models also aren't as good. But then this model is really good. Like how do you and also in the


bottom table, I think a lot of times sort of which model is better than which other model


is all over the place. Sometimes these are better. Sometimes these are better. Do you have an


explanation of what's going on here? What Why is there such a let's say I a diversity of which


approach wins in which circumstance? No, but I would say this is what is pretty interesting.


Like I feel now again, I'm coming from like a language perspective. And I'm sure an RL person


could give you like a much better explanation. But even when I was experimenting, I noticed like for


some environment, the transformer tended to do like, like even early on, like the language


pre training tended to like significantly better than the say, like the not language pre trained


models or even like the other models we have here. And this is just honestly, it's my intuition. But


I feel like some of these techniques are like very specialized, or maybe like very specialized


to the sense that maybe we don't know exactly what it is. But there are some properties of


the environments that really go nicely with certain techniques, but then don't go nicely


with certain others. And it's sort of like this random sort of puzzle game that's being played


here. That was my intuition when I was like playing with a I was like, Oh, wow, this is this


is pretty weird, actually. But yeah, that's that's my intuition.


Yeah, it was the even when like, if you look at like a GPT to a GPT columns, I think it sort of


varies across the environment as well. So I think that sort of speaks to it.


I also feel in reinforcement learning, a lot of times these algorithms are almost like designed


with a problem in mind. They are formulated as these general algorithms. But I think a lot of


times people, people go and they see what's the problem. I felt like this, you know, like go


explore that the first algorithm that solved Montezuma's revenge, right? I looked at it. And


I was like, you just you just like essentially hard coded the game into the algorithm, even


even with their they had two versions, even with their non non human designed feature space. I'm


I was just like, you you looked at, you know, you looked at what fails and you just hard coded a


solution. And you just I'm trying to tell me that this is a general, maybe something like this is


happening here to where people they analyze what goes wrong in particular environments, and then


they make an algorithm that would specifically address those problems. I find this to be,


I find reinforcement learning to be an interesting field because of because it seems like it's so not


solved yet. When we just look at your models, there is a there is a discrepancy. First of all,


I've noticed that a lot of times the GPT-2 here doesn't significantly, sometimes it outperforms,


but oftentimes, it doesn't significantly outperform the much smaller model. Do you have an


intuition as to maybe what's, you know, why don't we see a bigger benefit of large models here?


It's you say somewhere, it's over 100 times larger.


My intuition is this. So like, I think with like the certain papers, we've shown that like,


larger models can fit, like larger amounts of data better, maybe can even extrapolate from those


larger amounts of data better. But if we think about what we're transferring here, and it's not


again, it's not completely clear as of yet. But if we assume that it's a maybe a smaller set of


features or properties rather than like language as a whole, but maybe like some properties of


language, then we can maybe say that, okay, if GPT and GPT-2, despite their like very different sizes


have learned sort of the same sort of maybe some element of the structure, some notion of hierarchy,


or something like that, and they're both learned like relatively equally, so to say,


then maybe size doesn't matter as much here, given that we're fine tuning on the same like


relatively small amount of like trajectory data. So that's what I think.


Is it called GBT because it sounds like GPT?


No. Because, well, it was sort of related. But GBT like is means like sort of small mini type


of thing in Japanese. So it's like a joke. Because initially, so initially, I was calling it GBLM


actually, like when I was just referring to it, because I needed a name, I couldn't write like


the small pre trained language model every time. And then Shane was like, yeah, I'm going to write


GBT. And then Shane was like, you know, let's make it GBT. So then that's that's what


And you mentioned that clip often it performs a little bit worse. And to note you only you only


use the text encoder or sorry, the text model from from clip, which is in an a sequence model like


the other ones. And also there's IGPT image GPT that performs a lot worse. We can see it in this


image GPT. It just gets nowhere, right? And you had some hypotheses. Do you want to maybe,


especially for for the image GPT? You what what is your hypotheses on why that is just


kind of a failure case?


I think Yutaro can answer this one because he was like, master running these experiments.


Yeah, so well, I think the image, like the structure that's in the image, like image


GPT is trained on this quick angle pixels from like from images. And I think the structure that's


there in the image is like a really different from the structure that you've seen in language.


And in a way that like, like if you only have a static image, and if you only have like a pixels


out there, it's really hard to even like group, you know, which pixels group together into a


discrete like up like unit of objects like in a discrete, I guess, discrete objects. So that's


that. So first of all, like a GPT, IGP image GPT sort of like has to figure out that sort of like


discreteness like before, we can actually has ability to transfer to these RL settings where


it has more discrete structures. Yeah. And so yeah, that's that's, I think, one of the main


reasons why the current version of image GPT that are trained on static images are not really


good at transferring from from their domain to RL tasks. And I think if we can actually train


the sequential modeling or sequential models for like a video data, where it'll be much easier to


extract these like discreteness because if you only look at images or static images, it's really


it's and if you don't have any prior information about objects, like it's really hard to extract,


you know, objects only from static images, but if you have a temporal dimension, if you have a video


information, then it becomes much easier to extract those disc these objects because you know,


if you look at like a frame T and frame T plus one, you look at like pixels, the transform from


T and T plus one, you know, there's a difference in terms of perspectives. So that sort of gives


you a strong hint or strong cue regarding like which which pixels grouped together.


And that's a really difference, I think that will make eventually I think if we have if we invest


more into video research, and if the sequential modeling in the video domain, I think it'll be


really a big difference. So that I think I'm really excited about like the the future of


like a structural modeling that uses the video. And I'm excited to see how that


pre-training model on the video will be transferred to like a different domains like RL in the future.


And possibly the sort of the direction into vector quantized models might also help a little bit


because not working on as you say, it's really hard to even get what pixels belong together. But


if we had more of token based approaches, maybe, you know, that could that could help decouple


from the pixel level just just a bit. But that's, I guess that's just speculation by me. And one


speculation I also had was with respect to your alignment modules right here. So you have these,


you have these linear projections that try to make the token embeddings of the RL problem


as close as possible to the token embeddings that were seen during language pre-training, which


makes sense because you kind of get to reuse, let's say the paths that are already there for


the language models. In your ablations, you show that these it also works without them, which was


good for me to see because sometimes it's little things like this that only make stuff work. But


in, you know, there is a difference between the distribution of language tokens, which is usually


like a zip distribution or some sort of very heavy tailed, but but you know, sharp distribution


and image tokens, which by construction tend to be more uniform, especially, you know, if you think


like pixels, but also the vector quantized models, they are by design, uniform. And with the RL


problem, could it be that it's, it's also a matter of how the tokens are distributed? Maybe the RL


tokens are again, more more zipfian distributed, and that's why it might fit a lot better? Or did


you investigate the appropriateness of this, how the embeddings look like?


So, no, we didn't actually look into how the embeddings look like that was like, we actually


planned to do this, because I think, like, personally, I think it would be like really cool,


for example, if we found out that it actually like, you know, if we found out that the embeddings


like these embeddings turned into like a sentence, or something like that. But I do


agree with your hypothesis about maybe like how the tokens are distributed, or how frequent things


are. And I think this also sort of relates to sort of the structure in language or like this natural


tendency to express things in a certain way. And you may want to express certain concepts more


often than others. And then there's also like sort of this conditional nature, like maybe only if


this concept appears, which is represented by a certain set of tokens, then you want to talk about


this. Which in a sense, you could say mirrors RL, or like just any like sort of activities that you


would do. Versus image modeling, personally, I feel it's cool, like as a topic, but I also do


feel it's very forced in a sense. It doesn't feel very natural to me, if that makes sense.


Do you feel that there are other disciplines that would transfer well to reinforcement learning? I


don't know if you've thought about this, you do include language and images. So maybe you thought


of even other things. There are, I don't know, protein modeling, genetic sequences, there's


sound and so on. Do you have any hypotheses or any plans to try out other modalities?


Yes, that we do want to try other things. I think like some interesting things like in addition to


what you mentioned could even be like, you could, this isn't natural language, but it's usually


grouped in together with like the NLP community, but like code, for example, or even like testing


out different languages, simpler languages, controlling for complexity, really, maybe even


music. I definitely think speech could be something else to try as well. As you, Tara,


alluded to video, I think there's so many things in sort of our, I don't know about saying like


daily life, but there are a lot of things around us which sort of have like a natural sequential


nature of things. And it would be interesting to see if somehow, especially in like a low data


regime, if these things are able to transfer to each other well, and if they're like some maybe


underlying principles, or maybe like some like biases that are learned that correspond to like


a large majority of sequential data, or maybe certain types of sequential data, and might also


help us like group sequential data types, maybe learn more about how they relate to each other.


And I think if we're able to do that, then I think we'd be able to study this even more in depth and


maybe build models based on those findings. It's a pretty special world, right? That all our models


converge from all the different modalities that even allow us to do things like this. I find it


to be a very special time, because it would not have been possible if all the image models are


ConvNet, right? And all the speech models are somehow Fourier transformed some things.


Everything sort of converging to transformers. Some people might not like it, but it does enable


sort of a bigger picture on what even what it means to process data or, you know, if you want


to look at it like this. So these attention plots right here, I found to be very interesting. Now,


to be clear, this you say this is on Hopper. So this is one of these gym tests, one of these


continuous control tasks. Is this one particular sample? Or is this like an aggregate over the


data set? Or how do we what is displayed here? So this is an attention map, given a single


single one. Okay. Yeah. But we can we can assume it's kind of representative of kind of what


happens in general. So I have made a bunch of observations here in my video, which some of which


you also state in the paper, for example, this structure of three, like the models often looking


back three steps back, which makes total sense, because the decision transformer input comes in


these tuples of three, right? And I'm gonna guess if I want to predict the next return to go,


it's probably very related to the last one, especially if the reward is more sparse, I can


just predict like the same number again, I'm going to be correct most of the time. And maybe the same


with actions. Given that in the continuous control, frame by frame, I don't want to switch my action


around too much, maybe right. But it pays to look mostly at these things. What I found interesting


is the image GPT had a sort of just a recency bias, like it just seemed to look just two or


three tokens back in time, which I think supports very well what you claimed that image modeling


might be different from language modeling in that. Yeah, it might be that the image transformer just


sort of looks at a local neighborhood, and then just goes on, doesn't care too much about the


big structure. And it's just hypotheses. And then I think the most shady thing I said was with


respect to the randomly initialized decision transformer. So this would be the baseline model,


a transformer that from scratch is trained on this RL data. And I claimed what we can also see this


sort of pattern of three, but much more strongly than in something like GPT. So I think that's


a really good model, but much more strongly than in something like GPT too, which does have have a


more diffuse attention. So here, it's really super duper hard attention. And I claimed that might


that might hinder the model from learning proper connections between things in the future, because


it already kind of discards in the early layers, everything that would connect sort of a state and


this, does this come close to what you concluded? Or do you have like different insights into these


attention maps or what's happening here? It's actually very, very close to what we're thinking


after looking at these attention maps, I think, one thing actually, after watching your video that I


didn't really notice until you pointed it out was like, those yellow blocks of two, I didn't actually


notice that they were actually two, which I think is actually pretty cool to see like maybe,


like for those ones that weights, like two of them together, maybe with different weightings,


but overall, I think the interesting thing is that it's pretty consistent. Like it doesn't


necessarily change, like the patterns don't change significantly, which is sort of unlike language,


for example, where you can see things like generally, there is a recency bias to some degree,


but you can see things like depending on the token, go like, like pretty far if it's like


attending to similar tokens from far back. But then again, if you do think about it that way,


you could argue like action representations would probably be similar to action representations,


state to state representations and so on. So maybe actually the language models and even


the randomly initialized model are mirroring that. Yeah, it's I found it to be very special,


how hard the attention patterns are is right here. But also, there is always in distance of


three rows, there is one that is just only looking at three steps back and six and nine and so on.


And then the ones in between, there is one that has, as you say, that has two, and one that even


has like, it seems like almost it has three, but just one is a bit stronger. It'd be interesting


to figure out which one is which I don't think I can tell from this thing. But yeah, so I think


the one that's only looking at like three, but if I remember correctly, is the returns to go.


And then the ones between that are, let's say the state representations and then the action.


Yeah, so the order is basically words, yeah, that makes makes a bit of sense. And we I think


the sort of the result right here, I think in the middle layer, it's really nicely shown that


something like GPT, it will start to focus on maybe kind of the important things in the past,


it will select some of them to focus on. And so no matter which time step, it will kind of look


back at maybe what it determines to be important states, where as the randomly initialized one,


it will almost be like stuck in this mode of how it looks back. And my so my question here,


and you can clearly see it in the last layer, in that in GPT two, there's still this sort of


focus and attention on maybe what what it determines to be important things in the episode,


and the other ones, they just have like a diffuse attention matrix. And my question would be,


might it be possible that we could achieve the effect between let's say GPT two and the random


one, like this, this benefit through a much simpler procedure of just kind of regularizing,


just saying like, you know, don't make your attention so hard, like make, you know, just kind


of keep your options open. Try to look back a bit further, don't don't try to be so sure yet. Is that


you know, is that something that's reasonable? Or do you think there's reason to to discard that


idea? I think it's I think it's reasonable to try. But I still do feel that I think the if we do


something like this, then maybe we again fall into the trap of what we were like talking about


earlier is like this essentially like putting a band aid on like a very specific problem, per se.


But I think like the cool thing about transformers is they can learn a lot of different things. So I


think if you say like with a language model, for example, it's it's an initialization, you can find


your however you'd like to. And I think it's more like flexible in that sense. Unless like some


we were trying to tackle like a very specific issue, then I think yeah, it would be for sure


something to try. Like, I think there's this recent paper for language modeling by like Ophria


Press from UW. And he they were looking at like, say how they can bias the like basically enforce


the recency bias towards a language model and that like improves like extrapolation towards


longer sequences and so on. So I think in the in this case, in language modeling, it's like one


specific task that they're trying to solve. But here, if we like just talk about like offline


reinforcement learning, it's very, very broad. And I think, for example, if you tried like Ophria's


trick in like say for pre training BERT or something like that, and again, this is just


conjecture, but I have a feeling it may not work as well. Given like there's, I would say a lesser


like, there was also another paper by, I don't know who it was, but I think from Danji Chen's


group at Princeton recently, about like, the masking rate in BERT models and things like that,


and perplexity doesn't necessarily correlate with downstream performance and so on. So yeah, if


we're tackling a specific task, I would say sure, but I think the one nice thing is that


I think the one nice thing about the language model pre training is how flexible it can be.


Yeah, I was, I mean, I was the same. I'm probably as you say, falling in the same trap that I


criticize the field of reinforcement learning, say, you know, looking at one thing and saying,


can I make up something that would that would just solve this one thing? Yeah, and I think,


you know, the differences also to clip show a little bit that it's not it's not just I can't


just do any architecture or anything, there might actually be something to to language modeling.


In this table, you specifically show that the the language model pre trained ones converge faster.


And I had one question here. And that was that is how different is this code base? Like how much how


much of the difference in convergence can I attribute to you just being better at implementing


stuff? And how much is really due to this these two things being pre trained? Is it the same code


base? Or did you reimplement or implement from scratch? I wish I could say I was like this


amazing programmer that can make things so much more efficient, but no, unfortunately,


okay. So yeah, so this is legit, legit speed up that that is due to the pre training. Nice.


I guess like one caveat that mentioned like about GPT two, is that the faster training speed is due


to like faster conversions. Even though like it's pretty even though it's pretty big, but like say,


when like you're doing like your rollouts and stuff like that inference time, it is definitely


like slower as to be expected by a larger model. Yeah, that makes makes sense. I was also surprised


because in reinforcement learning, usually the conventional wisdom is that it is not


the same as the traditional wisdom is that it needs a lot of resources. And here you mentioned


something like, you know, you have a single V 100. And the time here is mean, even for the decision


transformers, it's a couple of hours, it's not, it's not I have to train on eight GPUs for a


couple of days, I was just positively surprised by by just sort of the requirements and this makes


sense. Yeah, I think that's the cool thing about offline RL, you just, well, you just have to like


say fit a certain set of trajectories. And there have been like a lot of pretty efficient models


recently as well. So yeah, I think it's when you get into the online setting, then things get


pretty like computationally expensive. You also mentioned that context size doesn't really matter.


In fact, more context seems to make stuff worse a little bit, right? Like how significant this


really is. But do you have an idea here? Is that it's just because there's more noise?


Or is there something wrong with the objective of the decision transformer?


I think, partially more noise. And two, I think, because of like, say the tasks that are tested in


gym. It's like you see a cheetah running, for example, or you have like this opera,


which is literally just hopping. And that those, those emotions are relatively repetitive.


Like in Atari, for example, the context is, I think, quite a bit larger. I don't remember


exactly what the value was, but maybe like 50, or maybe even a bit bigger than that.


But it's like, okay, for Atari, maybe you need more information because I guess like the actions


that are being performed are more diverse. And like, sort of what can happen is more diverse.


But then for these tasks, then maybe that much context is not as necessary.


This is just my intuition. Maybe an RL person would be able to give a better idea of why.


So the last thing that was here very special is just the scaling behavior of the data.


This of these models, namely the with the language model pre training, you could scale


too much larger models. Do you have a feeling of how that continues? Like does it does it continue


dropping off and just not giving you a returns anymore? Or would it eventually also say you have


like a model that's too large, and it would drop in performance again versus a smaller model?


Because my hypothesis was that language modeling, you have infinite data essentially. So you can


never overfit on the pre training. And therefore, you know, the there might never be really an


opportunity to overfit on a fine tuning data set. I don't do you have an intuition? I'm going to


guess, you know, maybe you didn't want to go up to too high parameter models.


Yeah, for like computational reasons, but for but I do generally agree with you. Like if we have


I think if we have a decent initialization, like from the like language modeling on say, like,


like, quote unquote, like infinite data, then I think we should be able to arguably at least


retain the same performance or get very close to it. And I think that's a good thing.


Arguably, at least retain the same performance or get very close to it. Perhaps there is a time


like a point where it just gets too big, that it starts overfitting. But I would say that


would probably happen when it like not not close to the parameters we tested.


Now you so so I think, oh, yeah, sorry. It's like one thing, one good thing about like offline


RL. So you can also collect a lot more trajectory data from just running agents and in training


offline data. So I think there's that perspective in this in this figure. Like, we can also train


like a larger model and larger trajectory data. And then if you have like a really good language


initialization, then you can also try that sort of direction of thinking.


Do you have an idea how that trades off? Like, would I rather would I rather invest


into pre training my model on language data? Or would I rather invest into gathering more


offline RL data?


Personally, I think if you're working with a fixed like say, okay, say if we fix the amount


of offline RL data, and say we're going to like use that versus like designing like a better


algorithm or something, I would say pre train your language model. But then again, as we see


with like GPT versus GPT experiment, making it that much bigger, like sure, it does help


like by advice, some margin, but it's not like that super significant. So based on that,


if we're going to assume that language transfers only like a certain set of maybe limited


properties to these RL tasks, then I would say yeah, collect more RL data, I would say


you said at the beginning, you tried it out, you thought about it, it kind of all it worked


out of, or initially, you got some promising results. Was there ever a thing that didn't


work? Like, like the something in this project you tried and just didn't work at all? Or


it didn't work at first? Any sort of avenues you got stuck in?


I would say that what was interesting was that the cosine, the cosine loss that we added,


especially like towards like later stages, everything sort of smooths out, but this more


has to do with how fast the model converges. So that's actually maybe we should have ablated this,


but the cosine loss actually allows them to converge much faster. And one thing that was


interesting was especially in the early stages that the cosine, so say we weren't using the


cosine embedding loss initially, and we just had like GPT and GBT, and GBT was like quite a bit


lower than GPT, but then like say GPT without this extra loss, and then GBT with the loss,


GBT managed to catch up to GPT, which is like pretty mind-blowing to me. So like something


like that was interesting. I wouldn't say like a hiccup, because it actually worked like pretty


well, like straight off the bat, but it was pretty interesting to see. And another thing was


without say like the positional embeddings, for example, I would do a general, like I think we


ablated this, but we would generally see like quite lower returns and things like that. So maybe even


like the position transferred from language is also quite important. Is there anything else you'd


like to get out about this paper? Can people get into this themselves? Your code, is it available?


Yeah. So actually it's in the footnote of the first page. So yeah, I think this stuff personally


is super interesting to see how we can transfer different sequence modeling tests to each other,


sort of unite. So like say one big model that handles all the sequences or something like that.


Another thing that was actually pretty cool is with like the language modeling co-training


that we did. When we did it, the language, like it was, we actually had a model that was able to


language model and was able to handle trajectories at the same time. And like the language modeling


performance didn't degrade significantly, which was also pretty cool, because it means that


we essentially have the capacity even at a small scale to do both of these tasks at once.


And if we have like these models that are able to handle these separately, then it begs the question,


okay, what can we do together? Like can we model everything all together? Like basically I think


with, what was it, like say like with multilingual pre-training that we have, it's sort of like until


I guess Ember or maybe like a few papers before that, we didn't really feed old languages just


together at once and see what happens. And then on top of that we see like, oh, we have like this


zero-shot transfer. Whether it's truly zero-shot is a different question, but still it's pretty cool.


And I think if we can sort of replicate that, say we have like, I don't know, a remotely related


language modeling, like a domain in language, and if we fine-tune on this domain in language, suddenly


we can do like trajectory modeling on this domain that say has to do with what was talked about in


language and things like that. Like it opens a new set of possibilities for maybe like generalization


and just like zero-shot. I don't like using that word, but like that sort of performance in


general, like these new behaviors and stuff. Cool. Excellent. Well, Michel and Yutaro,


thank you very much for being here and sharing the projects. I hope to see you again very soon


with more modalities and more, I think this is, I'm still amazed sort of by the results.


Building a Quadcopter p.2 - Parts

What is going on everybody welcome to the second how to build a quadcopter tutorial video in this video


Building a Quadcopter p.2 - Parts

We're going to be talking about all the supplies and decisions and choices and all that exciting stuff that you're going to have to make


Building a Quadcopter p.2 - Parts

In order to build a quad first things first


Building a Quadcopter p.2 - Parts

I'm going to be assuming you guys are building a 250, but it really doesn't matter from this point forward


Building a Quadcopter p.2 - Parts

Some of the power of the motor and the ESE and the battery and all that changes as you get bigger


Building a Quadcopter p.2 - Parts

But for the most part the types of supplies you need are all the same every quadcopter breaks down into


Building a Quadcopter p.2 - Parts

The same types of parts you need some motors you need ESEs which are kind of buried in here


Building a Quadcopter p.2 - Parts

But I'll show you some in a minute


Building a Quadcopter p.2 - Parts

You need motors ESEs obviously some propellers some sort of frame some sort of flight controller


Building a Quadcopter p.2 - Parts

Which basically accepts the transmitter's signals and also does balancing and stuff like that


Building a Quadcopter p.2 - Parts

Obviously a receiver to receive those signals and it feeds them into the control board


Building a Quadcopter p.2 - Parts

The control board then sends out those signals to the ESEs which then tell the motors how fast to spin basically


Building a Quadcopter p.2 - Parts

So then so those are like the main parts that you need okay from there you have a


Building a Quadcopter p.2 - Parts

You basically you require a battery


Building a Quadcopter p.2 - Parts

I put the battery on the bottom


Building a Quadcopter p.2 - Parts

But a lot of people put the battery actually in here if you do put a battery on the bottom just for the record


Building a Quadcopter p.2 - Parts

Make sure you can land and have like landing gear that is taller than the battery you don't want to be bouncing on your battery


Building a Quadcopter p.2 - Parts

the battery for the record a lipo battery is


Building a Quadcopter p.2 - Parts

Highly flammable and so what happens is first of all just another thing since we're talking about batteries if a lipo battery


Building a Quadcopter p.2 - Parts

Over-discharges it has the


Building a Quadcopter p.2 - Parts

propensity to


Building a Quadcopter p.2 - Parts

Expand and so for example


Building a Quadcopter p.2 - Parts

This is a an iPod that was left in my drawer for I don't know years


Building a Quadcopter p.2 - Parts

And you can see that it looks like an airbag deployed actually that's a lipo battery


Building a Quadcopter p.2 - Parts

And what's happened is it got over discharged and expanded if I then took a screw and like


Building a Quadcopter p.2 - Parts

Hammered the screw in there or nail or whatever this would likely erupt in very violent flames, okay?


Building a Quadcopter p.2 - Parts

So that's lipo batteries for you, so it's important that you don't over discharge your lipo battery


Building a Quadcopter p.2 - Parts

But also that you just don't go puncturing lipo batteries. That's a recipe for


Building a Quadcopter p.2 - Parts

problem


Building a Quadcopter p.2 - Parts

One thing I'll show you grab one of these so here's a battery. This is a lipo battery


Building a Quadcopter p.2 - Parts

This is a venom flight pack 2100 11.1 volts. I recommend the battery right now


Building a Quadcopter p.2 - Parts

I would not buy the 2100 this is the older model get the


Building a Quadcopter p.2 - Parts

2200 with the universal adapter well talk about that in a little bit here, but by that


Building a Quadcopter p.2 - Parts

But anyway, this is a low voltage warning some control boards will come with one


Building a Quadcopter p.2 - Parts

But if it doesn't I highly recommend you get one what it does is it plugs into the battery


Building a Quadcopter p.2 - Parts

And it tells you the voltage of the battery


Building a Quadcopter p.2 - Parts

And of its entire battery and then also the voltage per cell in the battery and then as it gets below a certain point


Building a Quadcopter p.2 - Parts

An alarm will go off and tell you that you need to bring in your quadcopter and charge it that way you don't over


Building a Quadcopter p.2 - Parts

Discharge the battery and create a fire hazard and all that so I recommend that on the topic of batteries, too


Building a Quadcopter p.2 - Parts

Basically when you go and buy let's say you buy a quadcopter kit there


Building a Quadcopter p.2 - Parts

It's going to come with a lot of times


Building a Quadcopter p.2 - Parts

It'll just be the frame the esc's and the motors


Building a Quadcopter p.2 - Parts

But then you can find some kits that come with the frame the esc's the motors the control board the power distribution board they come


Building a Quadcopter p.2 - Parts

basically everything you need


Building a Quadcopter p.2 - Parts

except a


Building a Quadcopter p.2 - Parts

battery


Building a Quadcopter p.2 - Parts

Some sort of battery charger. This is what I used to charge balance discharge my batteries


Building a Quadcopter p.2 - Parts

This is a tenor GTB 6B. It's a nice high quality charger. I recommend you do


Building a Quadcopter p.2 - Parts

Not cheap out on your charger, but it has like a fast charge


Building a Quadcopter p.2 - Parts

Balanced typical charge and a discharge and you can save


Building a Quadcopter p.2 - Parts

Various profiles in there for your batteries anyway highly recommended this charger


Building a Quadcopter p.2 - Parts

So you know battery quality is really important and taking care of your batteries is important so


Building a Quadcopter p.2 - Parts

That it won't come with the battery won't come with a charger


Building a Quadcopter p.2 - Parts

It won't come with a receiver, and it won't come with a transmitter, so this is my transmitter


Building a Quadcopter p.2 - Parts

this is a


Building a Quadcopter p.2 - Parts

Fs-th9s fly sky th9x controller or transmitter rather and


Building a Quadcopter p.2 - Parts

This one's a pretty high quality controller. It costs basically just as much as the entire quad


Building a Quadcopter p.2 - Parts

So you don't have to spend so much money on the controller. I like this one


Building a Quadcopter p.2 - Parts

It's got various profiles you can program stuff, and you can fly just about anything with it


Building a Quadcopter p.2 - Parts

But really for any quadcopter you just need these two sticks


Building a Quadcopter p.2 - Parts

That's it so find a controller that has those two sticks, and you're good to go, okay?


Building a Quadcopter p.2 - Parts

You don't need to spend too much money on this in all honesty, so


Building a Quadcopter p.2 - Parts

There's that so the that's kind of like all the essentials that you'll need now


Building a Quadcopter p.2 - Parts

You can you can buy the ESC's and the motors separate the frame separate the control board separate and all that


Building a Quadcopter p.2 - Parts

Or you can get them together. It just depends on the prices at the time so in the description


Building a Quadcopter p.2 - Parts

I'll put a list of supplies that you need and some suggestion some some suggested


Building a Quadcopter p.2 - Parts

Kits and all that but at any given time the prices are going to fluctuate and change and all that if you want to support


Building a Quadcopter p.2 - Parts

The channel you can use those links


Building a Quadcopter p.2 - Parts

Their Amazon affiliate links, but if you find like a better price somewhere else or whatever by all means go there and save yourself


Building a Quadcopter p.2 - Parts

some money


Building a Quadcopter p.2 - Parts

so


Building a Quadcopter p.2 - Parts

Generally at least for me what I found is see like the motors and the ESC's cost about


Building a Quadcopter p.2 - Parts

65 to 75 dollars


Building a Quadcopter p.2 - Parts

But you can find kits that are the motors ESC and frame and sometimes some other stuff for like 85 dollars or 80 dollars


Building a Quadcopter p.2 - Parts

So I usually buy at least those together, and if you can buy like the whole kit with everything else that you need go for it


Building a Quadcopter p.2 - Parts

So now we're going to talk about is the control board so this is one example of a controller


Building a Quadcopter p.2 - Parts

And this is another flight controller


Building a Quadcopter p.2 - Parts

These are two totally different ones as you can see this one is mounted like diagonal


Building a Quadcopter p.2 - Parts

It's shaped like a diamond almost because the control board what it has is a bunch of gyros on it


Building a Quadcopter p.2 - Parts

And it not only it take does it take signals from your receiver or yeah?


Building a Quadcopter p.2 - Parts

Basically so your receiver takes signal from the transmitter the receiver sends the signal into here


Building a Quadcopter p.2 - Parts

And then this says the signal out not only does it take the signal from your transmitter basically it also has gyros on it to


Building a Quadcopter p.2 - Parts

Help balance the board so like if a gust of wind


Building a Quadcopter p.2 - Parts

Pushes your quad this way real quick the quad will just automatically stabilize itself you won't actually have to do that


Building a Quadcopter p.2 - Parts

so that's one example of


Building a Quadcopter p.2 - Parts

What the control board does for you?


Building a Quadcopter p.2 - Parts

But as you can see this one so it wants to go this way which works on some quads


Building a Quadcopter p.2 - Parts

But not all other quads and what I found is most quads actually want


Building a Quadcopter p.2 - Parts

They have holes punched for the the flight board to actually be a square flight board like this one is


Building a Quadcopter p.2 - Parts

So this is a KK. Multicopter board. I like the board. It's super cheap. It's like five dollars if you find it at the right place


Building a Quadcopter p.2 - Parts

But as you can see I've kind of had to


Building a Quadcopter p.2 - Parts

Do my own work here. This is just a piece of wood literally


Building a Quadcopter p.2 - Parts

I cut the piece of wood and I drilled some holes in the piece of wood so this could mount and stay stable


Building a Quadcopter p.2 - Parts

It's really important that this is lined up very nicely, and it's stable otherwise you're going to have unstable flight, okay?


Building a Quadcopter p.2 - Parts

So anyways that's that I bought this one this one's a lot more expensive. It's more like I think 15 20 dollars


Building a Quadcopter p.2 - Parts

But it's a lot


Building a Quadcopter p.2 - Parts

It's square and so that's why I was really just hoping to find a square one so do what you want if you want to save


Building a Quadcopter p.2 - Parts

Money this is more than fine. I've flown this I've crashed the crap out of this one. It does fine


Building a Quadcopter p.2 - Parts

But anyway, that's that so that's one difference between the one I'm going to build and


Building a Quadcopter p.2 - Parts

Ones I've built also let me pull up the distribution boards


Building a Quadcopter p.2 - Parts

So these are two distribution boards


Building a Quadcopter p.2 - Parts

This is one here. This is a this one basically allows you to


Building a Quadcopter p.2 - Parts

They've got bullet connectors, and this is just a simple distribution board where basically you're going to connect to the wires


Building a Quadcopter p.2 - Parts

And then on the bottom side you'll solder the wires to the distribution board


Building a Quadcopter p.2 - Parts

This distribution board is actually the distribution board


Building a Quadcopter p.2 - Parts

I learned a valuable lesson on and that is that carbon fiber conducts electricity. I had no idea


Building a Quadcopter p.2 - Parts

I didn't even think that the frames. I was buying were actually like real carbon fiber


Building a Quadcopter p.2 - Parts

I thought they were just like fake carbon fiber made to look like carbon fiber


Building a Quadcopter p.2 - Parts

But but yeah, if you have a real carbon fiber frame understand that carbon fiber has resistance and that means it conducts electricity