and anybody who tries to tell you they knew what model neural network was going to work is a dirty liar.


Optimizing Neural Network Structures with Keras-Tuner

It's trial and error. Now of course there are some tasks like MNIST for example,


a paper bag could solve MNIST to 90% accuracy or more.


So obviously there are some problems that are just so simple any network will do


and then there are some neural networks that will solve most problems, especially like image problems, stuff like that.


But for real world problems that aren't solved yet, the solution is trial and error.


And this has historically involved me writing for loops to solve it and then in that for loop I just tweak things


and then I run that overnight and then I save validation accuracy or loss or both


and then in the morning I just see, okay these are the attributes, like three layers at 64 nodes per layer,


it seems to be the thing and then now I'll test with batch norm, because every time you change one little thing like dropout or not,


50%, 20%, 10%, batch norm or not and all those things you got to keep testing.


So anyway it's historically just been ugly for loops to be honest with you.


But recently I came across the Keras tuner package which does everything I was doing but better


and it does a few other things that I wasn't even doing which is pretty cool.


So I thought I would share it with you guys, basically the crux of it is you got a model and then you can define little hyper parameter objects


inside that model and then you create this tuner object and then it changes those hyper parameters


and the hyper parameters you can specify, you don't have to, not everything has to be a tunable parameter


but the ones that you do are, you can do anything like an int, float, you can do a choice, a boolean


and the ints and floats are like a range with a step size.


So anyways that's pretty cool, to get it you just pip install keras-tuner.


Also I'm using keras tuner 1.0.0, now as a tutorial person this throws up tons of red flags for me but I'm still going to do it anyways.


But 1.0.0 means one of two things, it either will never get updated again which would be sad or two,


it is going to be updated a ton in the next year and this tutorial could be rendered out of date very quickly.


If you're hitting errors, check the comment section, google the errors or you can pip install keras tuner


the exact version that I'm using.


So once you have that you're ready to get started but first a quick shout out to the sponsor and supporter of this video


Kite which is an ML and AI based auto complete engine for Python and it works in like all the major editors


sublime text, vs code, pycharm, atom, spider, vim, but why?


Anyway all the major editors and it's actually a good auto complete, honestly I hate auto complete so when they reached out


and were interested in becoming a partner I was like we'll see but it's actually pretty good.


It took me a little bit to realize how good it was and in fact I used it for about a hundred hours now


and the real test was when I removed it, like I just disabled it, just to see what the difference was


and I was like oh wow there's so many big differences, the biggest thing is like the auto complete, it's not just variables


it's like methods and even like snippets of code and like the imports are so much nicer, huge differences


I'll show some of them here but I highly encourage you guys, I'll put a link in the description for you guys to check it out


it's super cool and it comes with kite copilot as well, just kind of like a stand alone app if you want it anyways


and basically it's like live updating documentation for whatever you're working on which again is super useful


so yeah it's a really exciting sponsorship, I don't take like any sponsorships if you haven't noticed


like I don't do VPNs, mobile games, all that stuff because it just doesn't really make any sense


but this is actually a really cool product so I'm excited for them to be supporting the free education


and I definitely suggest you check them out, the auto complete is the best


so okay cool, so let's go ahead and get started, so first we need a data set and we need a model


there's like so many things that we need just to even get started with Keras tuner


so I'm going to try to like truncate this as much as possible but first we need a data set


the data set that we're going to use is fashion MNIST, so it's like MNIST only like I said MNIST is like too easy


so it's kind of no good for showing this, so we're going to use fashion MNIST


so I'm going to say from tensorflow.keras.datasets we're going to import again just as an example


here are all the data sets, fashion MNIST, so then fashion MNIST.loadData is what we need to load the data


and then just to shill out completely kite copilot comes down here, it returns this right here


so I'm actually just going to copy and paste this, since most people probably aren't running kite copilot right now


you can go to the text based version of this tutorial or download kite real quick


anyway so here's all your data and I think I'm just going to show a quick visual example of the data


and then we'll probably just copy and paste a model honestly, I don't want to waste too much time


so first let's just quickly display one of the data, so we're going to say import matplotlib.pyplot as plt


lovely auto complete thank you very much, plt.mshow, oh no, it tried to help me


mshow and then we'll do xtrain, we'll go with the zero-th and then plt.show


and then I'm also going to do a cmap, cmap equals gray here only because it's going to be all colorful if I don't


and people are going to be, wow that's loud, people are going to be kind of confused


so I'm going to say python 37 ktuner tutorial, so this is kind of the data


so again it's a 28 by 28 like mnist, it's black and white, it's got 10 classifications


it's just articles of clothing rather than handwritten digits


so in this case it's like a boot or a shoe of some kind, I don't really know what that is


so let's do a different one, let's see hopefully we can get something a little more recognizable


so it's some sort of short sweater thing


so that's what we're dealing with, so it's just a little more challenging to get like 98% accuracy


as compared to like mnist, so this is a thing that we can practice tuning on


mnist just doesn't work, so once we have this we actually need a model


now again I don't really see any benefit in this tutorial for me to like write out this entire model for you guys


it would just be a waste of time, so I'm actually just going to copy and paste


I'll either copy and paste this into the description or I'll put a gist or you can just go to the text based version of the tutorial


but this should all be, except for this hp here, this should all be totally understandable by you guys


there should be nothing here that is like whoa what's that


so hp stands for hyperparameters which is what we're going to use to go through


this is a comment that I made for the actual tutorial


so this just creates our model object, returns the model, pretty simple


so from here you can define the model and do a.fit for example


so the other thing that we need to do is import, so let's go ahead and from tensorflow


or actually from tensorflow we're going to import keras


then we need to import all that layer information, so from tensorflow.keras.layers


import all of these basically, we're going to need every single one


so dense, actually we're not using dropout but conv2d yes


what else we got, we got activation, we've got a flatten


and we've got some max pooling, cool done


ran over my face a little bit, again there is the text based version of the tutorial


and again I'm really just trying to run through this, there shouldn't really be anything confusing to anybody here


so once we have this, real simply we can test this, I just want to make sure it works


so I'm going to say model equals, what do we call that, build model


so then from here we should be able to say model.fit


yet again cool thing from kite is this entire snippet, boom, done for you


and then you can kind of tab through it, so for example batch size, let's go with 64


epochs, so then again I'm just tabbing through that which is pretty cool


validation data is going to be a tuple of information


in this case, what are we going to do, we're going to say x train y train


so x train, or x test rather


oh it tabbed me over, x test


y test, and then actually in this case I don't care about verbose, I had it there because I was building this tutorial in a notebook first


so ok, so then we can do that, oh we need to have our x data


so x train, and then this will be our y train


and we need to reshape the data before it complains to us


x train equals x train.reshape


and it's negative one, twenty eight, twenty one, again if this is confusing to you I strongly encourage you to go back


to one of the deep learning playlists, ok so that should all work


I do want to test it real quick to make sure it works before we get into the actual


tuning aspect of things, so I'm going to save, run that again


please work, oh


it's because we have that hp there, let me just move it real quick


put this back, please work


whoa, ok so it's training, we can actually see it's already working


we could just go to one epoch, so for example after one epoch it's actually not that accurate, like 77 percent, not much


but whatever you got, I can just tell you it's not above 90 percent, ok


so the question would be ok what do we do from here, can we actually do better than this model


so the fashion MNIST dataset is still easily learnable


what's not easy is being 99 percent accurate as opposed to like MNIST


so what you would do from this point, like if you're me and you're trying to compete in a Kaggle competition


now you're just going to start changing stuff, but what we can do with the Keras tuner package is we can like automate that entire process


so what I'd like to do now is, I'm going to move that over


and we're going to import a couple of things, again I'm going to try to keep this super short


so I'm going to, as short as possible anyway, I'm going to just copy and paste that in


so from Keras tuner.tuners, I'm going to import random search and then from Keras tuner.engine.hyperparameters


we're just importing hyperparameters, and again these are the things that allow us to do


like the int, the float in ranges, choice, boolean, and I want to say there's another one but I forget it


but anyway, again text-based version has links to all the docs, I'm just trying to show you a general idea of how all this works


the next thing I want to do is I'm going to import time, and then we're going to specify logdir equals


and for now, this is kind of a silly one, but anyway this is a stupid F string to use, but whatever


time time time, for now the logdir will just be a timestamp, but you could add other things to this if you wanted


but again, trying to keep it short and simple


so, ok, so build model, now we will pass the hyperparameters, and so far nothing is actually unique here


so the next thing that we're going to do is I'm going to come down to the very bottom here


and I'm going to comment out these two things, because this no longer will even work


and in fact I'm just going to get rid of them because that build model is no longer functional in the way that we have


as long as we're going to pass hyperparameter there


so, first you're going to specify the actual tuner that we intend to use, and that's going to be a random search


and here we're going to pass a bunch of things, first is the actual function that we intend to use, build model


no parms there, the random search object here will automatically do the hyperparameter stuff for us


so, you just pass the function name here for your model, then we've got objective


this is the thing that we're interested in tracking, so in our case, val accuracy


that's what I'm interested in, then we're going to say max trials


I'm going to set this to, for now, I don't know, for now 1, we have no dynamism here


so it doesn't matter, I'm going to set that to 1, I'll explain that in a moment, and then we're going to say executions


executions underscore per trial equals 1


and then we just give it the directory, so directory equals logdir


so max trials and execution per trial, so trials is how


so like, things can get, options can explode very quickly when we're allowing ourselves to have a dynamic number of layers


a dynamic number of nodes per layer or features per layer


and then a bunch of boolean operations like, do we want dropout? and then if we do, what range of dropout?


this can explode to like thousands or even millions of combinations pretty quick


like with a two layer convnet, pretty hard to get to millions, but it can get to a huge number


like a thousand different combinations of models is not going to be very hard


so here, when we say max trials, this is just how many random pickings do we want to have


and then executions per trial, this is what you would, like if you're just trying to search for a model that learns at all


then I would keep this to 1


if you're trying to eek out the best performance, I would set this to like 3 or 5 or maybe even more


as long as you're like shooting in the dark, just trying to find something that works or just a general idea of what seems to work


I would keep this low, but the point of this is, is each dynamic version, you're going to train it this many times


so if you've got some sort of, you know, when it randomly says, hey let's do 4 layers at 64 nodes per layer


it will train that one time, but you might want to say 2 or 3 times


because if you're trying to eek out like 1% accuracy, given 2 different runs from the exact same model


you might find that you get more than a difference that's higher than 1%


so if you're really trying to eek out performance, one you might be seeking a model that has a smaller standard deviation of validation accuracy


but also, you might want to run it a few times to figure out what the true average is


because you might have just gotten lucky, or maybe you got unlucky or whatever


so hopefully that makes sense, if that doesn't make sense, feel free to ask in a comment below


so that's our tuner object, and then now what we're going to do is actually do a tuner search


so tuner.search, and in this case we're going to specify our x data which will be x train


and then our y data will be y train, and I think I'll go ahead and tab that over


actually, I suppose if we want to be all, I think it's correct, pep8 to remove those


x, y, and then we need epochs, so how many epochs do we want to train every single time


again, this really depends on your needs, I'm going to set it to one just because to iterate over this stuff


it still is going to take a while, and in fact we're probably not even going to do it, but locally I'm just suggesting maybe start with one


batch size, again this is going to vary by what you're running this on, I'm going to say 64


and then validation data, did I do, I did, dang it, I've got to type it


validation data equals, I've been so spoiled, I haven't really had to type too much


x test and then y test, ok


so now what this will do is search given these parameters, right, and then


what kind of stuff is it going to search, it's going to based on this information here, we're only going to run one trial


again I'm going to just quickly run this to do a bug check and then we'll actually start making the model truly dynamic


so, cool, runs, good, so now what we're going to do is come to our actual model


and start adding things that make it dynamic, so first of all, so for our input layer


let's say we take 32, and we want 32 to, yeah sure maybe be 32, but maybe we want it to be anything from 32 to 256


but we don't want to say like 32, 33, 34, right, we want to have a step size there, so the way that we do that is


hp. capital I int, and then we're going to give it a name, and so I'm going to say input


units, not shape, and then it's the starting value, max value, so 32 to 256


and then the step size, I'm going to say 32, so, and in fact I'll just add the actual names to this one


I'm not going to do it every time, but just to make it super clear, min value, max value, max value


and then this is our step


so, I'm just making sure it's on screen, I'm not covering it, so now the input units is a dynamic value that will be randomly chosen


but somewhere in that range, okay, so, yeah looks good


so again, I'm going to save that really quickly and just make sure we run no bugs


cool, so in theory that is with a different, you know, who knows, I don't know which one it was in that case


the tuner does save all that information, I'm just again bug checking at this stage


so, random search, and so okay, we've got a unique or a dynamic number of


for the input to this, this convnet, and then we can come down here


and then, so the input layer is a little unique because it has an input shape that we need to make sure we retain


but then the subsequent model, or the subsequent layers can, are all basically the same


and in fact I'm going to remove max pooling because that's going to cause trouble for us


we're going to run out of things to pool if we, in some cases


so I'm going to remove that, and now we're going to say is, we're going to 4i in range


and again we're going to pick hpint, and then the name in this case will be n underscore layers


and we will go from 1 to 4, and since we want the step size to be 1 we don't need to input anything more


so from here we'll do that


so now, model add conv2d, and then again, what if we want this 32 again to be unique


so I'm going to take hpint here, copy that, paste that over that 32


rather than input units, I'm actually going to say underscore i, we'll make this an f string


and I'm going to say conv i units


and again, 32 to 256, cool


so, looks good to me


model add relu, ok, I think that's good, we can go ahead and save that


let's come down to here, go ahead and run that, make sure it runs


it does, ok, so I could continue to let that run, and in fact it might, let's see, I think we said one epoch


so hopefully at the end of at least one epoch it should give us a brief summary, I would hope


oh that's right, we said only, so at the end of it, ok so we can see, fine


in this case we had a neural network so that input convnet was, so the input units was 32


then we had two conv layers, so it's a little bit out of order actually, but the input unit was here, right


so we actually had a 32, so it was 32 by 96 by 32


not by, but each layer, so the input layer is 32, like the convnet has 32 features


and then it feeds into 96 features, then the 32 features, so it says n layers is 2 but it's actually 3


it's plus one more, it's just n layers is the answer to this for loop here


here, right, so the actual number of layers is n layers plus one, I hope that makes sense


coming back down here, so then this tells us what the score is, it also tells us what the best step for that score was


which is useful but we only did one epoch anyways, ok so we get the information


but of course this was relatively meaningless because we


we only tested one thing, so the next thing to do would be to make this a much much bigger test


but each test takes, this says 19 seconds, so and it's gonna vary depending on how many layers you do and all that


so it's just going to vary, what I've done is I went ahead and created


I just went ahead and saved it as a pickle object, so I took tuner at the end, you get this tuner object


so at this stage all I did was imported pickle, I saved the tuner object to a pickle


thought I had saved it but I made a mistake so I ended up just retraining everything


so here I am like an hour later, I've got my pickle, I've got my directory and just to really quickly show you


inside this directory and then inside of untitled project we have all of the trials that ran


and then for each of the trials we have a checkpoint which obviously is the model


and then the trial.json file contains obviously your ID but then the hyperparameters more importantly


and the score that that model got, so even if you didn't have the pickle you could still go back in time


and get this stuff, but anyways it would be kind of challenging and tedious and annoying


especially if you've got many iterations per trial, you'd have to pair up I guess by exact hyperparameters


I don't really know, anyway it would be annoying, so it's nice to have the pickle saved instead


just to come back later, so yeah, so cool, and so you have that and then also I want to say here


you've got, is it the oracle? I think it's inside the oracle that has the best information


it's definitely not tuner 0, but you can also interact with the tuner object which is why I saved it as a pickle


so first of all we can get the best hyperparameters right out of the gate just by running that


and in fact before I do that let me comment out this, this, and then we don't want to do the search again


so I'm going to comment that out, cool, ok so let's just run that real quick


and we get here the best hyperparameters, now I don't really find that very easy to read


but then the next thing that is a little more useful to me is the result summary


so this should give us the top, I think by default, top 10 models, so here they are


and as we can see here the top 10 got, basically the best was 87 but it's anything from 86 to 87


it's within what, 5% accuracy it looks like, or 6, so pretty close there and also 87 is the very best


interestingly enough in the other one that I trained that I lost, the best was 89.6


so I do think we could for sure find something more than 90 and if you run this it's totally possible that you'll find some that are better than 90


and the reason for this difference is even though it seems like we made very minimal things to make dynamic


we actually did a lot and so to actually run through all of those I don't actually totally even know


so you could have up to 4 and then 4 layers of each unique convolutional layers


and then of all of those combinations times all the possible combinations here


I mean it would just be a huge number, it's got to be hundreds or even in the thousands, I don't know


someone go ahead and do the math if you want, anyways it's a lot, so there's a lot of combinations


but that just goes to show that I already know that somehow with this exact code you can get actually 89%


it's just a function of running through all the combinations, so again it's very nice if something can do it just automatically for you


so finally the last thing I will show is that you can actually from here get the actual model


so a tensorflow model, so print tuner.get best models and then I'm doing a dot summary here


but this is an actual model so you could actually do a dot predict as well, I'm just going to do a dot summary just so you can see it


I think this is the easiest way, the hyperparameters is still kind of hard to be like ok but how would I build this model again


whereas a dot summary really just makes a lot of sense to me, so anyway we'll just do that really quick


but mostly it's just to show you it's an actual tensorflow model


so as we can see you've got the input layer activation pooling right, so you just build this model exactly to these specs


so ok I think that's enough information, hopefully you guys have enjoyed, like I said this is one of the most common questions I get


the answer is probably not as intriguing as you maybe would have hoped


but hopefully Keras tuner can actually make your lives easier


if you have questions, comments, concerns, whatever feel free to leave them below and again shout out to Kite


if you want to try them out they are, it's a totally free plugin and like I said it's pretty awesome


like I'm really enjoying it so hopefully you guys will enjoy it as well and thanks to them for supporting the video


[ML News] Blind Chess AI Competition | Graph NNs for traffic | AI gift suggestions

We play some blind chess graph neural networks are used in Google Maps to predict traffic


and AI makes for thoughtful gifts. Welcome to ML news. It's Monday.


Hello and welcome friends of the Monday. Welcome to ML news. Now to be honest with you,


not a lot of stuff happened this week. I guess that's what they call a slow news day or something


like this. So I thought we'd just take a look at more lightweight things that I came across. So


the first one is reconnaissance blind chess, which is a chess variant that is now also a


NeurIPS 2021 competition. The rules are the same as in regular chess except you can't see what your


opponent does. So every move that you have is actually split in two, you can first use sort of


a oracle to sense the board or a piece of the board. And then after that you can make your move.


So now you have to be strategic about where you use this sensing. And when you make your moves,


you have to be strategic because you can count on making your regular chess moves. But you can also


make moves that you think your opponent won't scout, which makes for some nice surprise attacks,


the notion of check is removed, and the game ends when a king is captured. So on the website,


you can actually play ranked matchmaking or play a bot. So here on the white pieces,


and it's my turn first of all to sense now at the beginning, it doesn't make much sense. But you can


see you can sense a three by three square anywhere you want. So let's sense here, wow, what a


surprise. They're still in the initial configuration, and then make a move and now the opponent senses,


you won't see where they sense and you won't see their move. Now I'm not particularly good at chess,


but I'm just gonna scout about here. And you can see that it reveals their move that they've made.


Now had I scouted somewhere else, I would not have seen that move. So now I can react with a


bit of an attack. And not only do you have to pay attention to what your opponent does, but you sort


of have to model what your opponent might know about you. And maybe even from the moves that


your opponent makes, you can sort of parse out what they might or might not know about you and


your pieces. So here my opponent goes for a bit of an attack, and I just like horses horses are nice.


All right, so move has been made. Now you do get informed when a piece of yours is captured or when


you capture a piece. So none of that happened yet. So let's sense around here. And that did not


reveal anything. Oh, yes, you can pass as well in this game, which makes it even more complicated.


So I'm going to guess the opponent guarded this pawn back there, I'm going to try some attack


here. So now it's my turn to sense I'm going to sense about here to see if they countered any of


my things. So now is an interesting situation, right? I have no indication that anything is in


the way between me and the king. Now, if my opponent had sense that I move my bishop there,


they would have probably moved the king out of the way by now. So the king might be here in front,


yet if they hadn't scouted it, they have no motivation to move the king at all. Therefore,


I could now just capture the king. I want to create this chess pro Magnus Carlsen, bring it on,


bring it on. Alright, this is reconnaissance blind chess. If you're interested, I'll link it in the


description. Let's see if you can win to I played against an opponent level of trout here just for


reference. There are various settings and they instruct you how to build a bot give it a try.


Next news, there's some discussion on Reddit about colab Pro. Now we've reported previously


that colab now has a new tier called colab Pro plus, which gives you even more priority access


than colab Pro two GPUs. So now people are starting to notice that colab pro subscriptions


don't always give them very good GPUs anymore. Now the thread is filled with various comments and and


the general opinions of the different people are that yes, probably now that people have even more


priority access. If you are just a pro user, you might get less access be colab is still one of the


most cost efficient ways of running on a GPU on the planet. And see a lot of people still do get


good GPUs with colab Pro. So it could just have been a problem of some kind of usage spike. So


make of that as you will for what it's worth Google never promised to give you good GPUs,


they simply promised to give you priority access. And that's about that. It's just important to be


aware if you're considering colab Pro, if you really rely on getting good GPUs all the time,


then the colab Pro plus might be for you. In a big collaboration between DeepMind Waymo, Google,


Amazon, Facebook AI and CAI lab researchers have used graph neural networks to do better traffic


prediction. Specifically, they talk about ETA prediction estimated time of arrival, and that in


real time. So the way they do it is they segment roads or paths in general into these segments. And


then they use graph neural networks to integrate all live information to give you an accurate


estimate of when you'll arrive. The interesting thing is they don't do that much crazy stuff with


these graph neural networks, they have some tricks up their sleeves, like the use of meta gradients


in order to control hyper parameters. But in general, it just sounds like a really solid


engineering effort. And this is deployed in Google Maps. These statistics here show you by how much


the ETA prediction accuracies have improved. And sometimes this is really staggering. So you see


great improvements across the board, sometimes up to 50%. I'm not exactly sure what the metric here


is. But 50% is a big number. Can we all agree? Yeah, good job. Okay, let's look at some helpful


libraries and data sets. The first is ISAC gym, a high performance GPU based physics simulation


for robot learning. We saw something similar with a library called Brax. These physics simulations,


they now run directly on accelerators such that you can do end to end research on the accelerators,


you don't have to switch between devices all the time, which massively speeds up research in


control and reinforcement learning. So this one's called ISAC gym, you can get it from video, which


is a bit worrisome. But it looks very cool. In these demonstrations, they have an evaluation,


and they also do train some policies on it. Now that is disturbing. But in general, it seems like


if you are on GPUs, and you're trying to do reinforcement learning and control settings,


this might be a good option for you. Also in the domain of physics, nimble physics releases the


differentiable human body model. So this apparently is a gold standard human body model that was used


for simulation. And now this library made it end to end differentiable human body model isn't just


one body model, but it is a configurable body model where you can sort of control the size of


all the different parts and still get accurate simulations out of it. And now with it being


differentiable, there's a whole new range of applications in research that become possible


with this. If you're into biomechanics or differentiable simulations, I think you should


check this out. LV IS is data set for a large vocabulary instance segmentation. And the goal


here is to do instance segmentations on categories that are vast. So there are a lot of categories


in these instance segmentation problems. And a lot of them don't appear very often, which is what


they're referring to here as long tail. So some of these things you might have never seen before,


we've seen a couple of these data sets, this one is especially challenging, because not only do you


have to recognize what it is, you have to segment the instances. So here you can see examples of


donut, pineapple, teacup, wine glass, wrath, I don't even know what a wrath is. Wrath,


an arrangement of flowers, leaves or stems fastened in a ring and used for decoration,


or for laying on a grave. Wonderful and bird feeder. So there are even competitions and leader


boards to go along with that. If you're into this kind of stuff, check it out. Next is behavior by


Stanford University. The Aver stands for benchmark for everyday household activities and virtual


interactive and ecological environments had to bend a lot of stuff to come up with this acronym.


But now it's called behavior. This is a data set for doing robotics in what are supposed to be


relatively real life scenarios in virtual environments. What's interesting is the creation


of this data set, the data sets are modeled after real scenes. So people analyze what they call


everyday situations, and they try to recreate them with objects from WordNet, you can let


AI is run in this simulated environment, but you can even do it yourself by VR. And the data set


includes VR demonstrations of these things by humans. On top of that, it's not a fixed set of


environments, but the environments are sort of described by a little bit of a grammar. So


therefore, potentially infinite variations of these environments can be generated. Here you see a


bunch of examples of this grammar. So for example, fish can be burnt or cooked or frozen, the


microwave can be open or closed, the apples can be on top of the plate, and so on. The AI's are


supposed to fulfill tasks in these situations. And I guess the goal here is to come ever closer to


real life robots that actually help you in everyday life. The problem I have a little bit


with these things is that even though the simulations are modeled after real life,


they're still very, very far from it being limited to WordNet, I guess limits the amount of stuff you


can put into a scene, scenes are probably still kind of regular real life happens to be much more


messy. So it's a bit of a question how useful this is for the end goal. But still, it looks like an


interesting problem. And it's definitely a step into the direction of robots that interact with


real life in a more realistic and competent manner. Next news, wired writes a new chip cluster


will make massive AI models possible. Cerebras says that they've built a cluster that can run


a neural network with 120 trillion connections for reference, that's about 100 times more than


what's achievable today. So if you want to build a large scale neural network today, your options are


you can use TPUs, which are somewhat large if you use a cluster of them, or you can just stack GPUs


together and connect them with some sort of infinite band, both are not really optimal,


as the accelerators themselves are relatively small, and they have to communicate a lot.


Therefore, Cerebras strategy is to build giant chips here you can see one in comparison to the


largest GPU currently available. So these things are actually huge. Now the article details the


various engineering problems that you have when you want to create such a large chip. Notably,


the chip itself has to be much more error tolerant as you can't simply switch out one piece


whenever it breaks, like you could switch out a GPU GPUs by no means are cheap. But compared to


this thing, a GPU is certainly a bargain. Now they didn't stop at building single chips, they built


an entire cluster of those chips. Now at least as the article states it, they're just waiting for


someone to come around and actually train a model on it. Their CEO says, so we know we can, but we


haven't trained a model because we're infrastructure builders. And well, there is no model yet. If you


have an idea of how to use 120 trillion connections, maybe give Andrew Feldman a call. The bigger


question is a little bit of whether scaling individual chips is the correct approach. Or


if it's just better to stick with the smaller accelerators, but improve our abilities to


communicate and shard models, I guess only time will tell. Washington Post writes AI gave Val


Kilmer his voice back, but critics worry the technology could be misused. Of course, critics


always worry the technology could be misused. So the article details about this startup called


Sonatic that used recordings of Val Kilmer's voice in order to make an AI that can synthesize any text


in his voice. Val Kilmer lost his original voice due to surgery after throat cancer. And this model


essentially gives him back the ability to communicate in audio in the way that people


remember him speaking. Now this isn't a prosthetic, I think he still has to type the things he actually


wants to say. But with some good brain interface, this could be an actual technology for people who


lost their voice to be able to speak again in the future. The article also goes into a little bit


of the possible economy that could result from this, namely that as a voice actor, I don't


actually have to voice act for every project I do, I could simply sell my voice for other people to


use as a sort of a licensing deal. The article also voices skepticism with respect to that and


quotes Jay Britton, who is a voice actor that says, when I'm an actor, I get to decide whether


I support the content, it would be a devastating thing to drop on a voice actor that your voice is


out there saying things that you might not necessarily support. So the criticism is that


someone could buy your voice for a license fee, and then have it say something that you disagree


with. And rather than sounding the alarm bells about this, I think we should simply adjust to


the fact that yes, this is a new possibility we have, but it's not a new thing by any means. I


mean, stock photographs have existed for about as long as the internet has existed. And if you're a


stock photograph model, then it's absolutely expected that your picture can be used for


something you disagree with. That's just part of the deal. And no one faults these models if they


appear on such a picture. So I think what needs to shift is not people not using this for various


things, but simply our attitude towards what can be done with voice technology nowadays.


So the last article for today, Forbes writes, can artificial intelligence give thoughtful gifts and


exploration of the possibilities and limits of AI's humanity. This is a bit of a fluff piece for a


company that uses AI to sort of recommender system gifts for people, which is interesting,


because usually the media is rather critical of these recommender systems. However, in this case,


it's sort of framed as the AI really understands you and knows what the good gift is in a moment


and what a thoughtful gift is and so on. And you know, in my opinion, they're probably not wrong,


like most gift suggestions could be made by an AI much better than you just kind of sitting there


and coming up with something. So the startup is called GOSB for people who are interested. I just


want to show you how these things might look about. So this is one of these little plugins


that you can have as a YouTuber that does a little bit of analysis for you. It's not super useful,


but I always enjoyed this feature right here where it gives you ideas for your next videos.


And I'm not going to say that the quality is anywhere near or close to what GOSB is doing.


I have not tested them. I just want to show a little bit that you get the feeling of what this


might be like. So here are videos I could do. I've not looked at these yet I get three per day


because I'm cheap and I'm on the free version of this product. So we're going to look at them


together. Devlog tech demo interactive game. Well, I don't think that's exactly for my channel how to


enable CNBC news alerts. I think it just estimates my channel as sort of like a tech channel or


something like this. Maybe this is because I made how to bypass neural hash dismiss a revolutionary


product for Apple users. This is definitely because I made the videos on neural hash now.


And that was it. Now usually, usually I have to say they're a little bit better. They're a little


bit into the direction of what my channel is actually doing. I guess I've just confused it


with the recent videos about neural hash. But safe to say if you're searching for gifts for people


that you kind of know a system like this might actually be a good place to go. It will probably


suggest you a bit of generic gifts, maybe personalized a little bit to what you input about


the person you want to give to and that's all we need. Okay, this was already it for ML news. As you


can see, really nothing happened this week. If you're an ML researcher, if you're an industry,


or even if you're just interested, please make something happen for next week. Please, I need


Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Hi, today we're looking at batch normalization accelerating deep network training by reducing


internal covariate shift by Sergei Ioff and Christian Skedadish, not my, not the best


pronouncer, Segedi, close enough. Alright, so this is a bit of an older paper and it's,


I think it's still good to look at it, it's relevant and people just kind of throw batch


normalization into networks and maybe don't really know what it's doing. So let's look


at it. So what these people argue is that in a network usually you have structures like


this. So if something like that, it means that your loss kind of, this is a two layer


network, your loss is a composition of the first layer on the input u with parameters


theta one and the second layer with parameters theta two. So conceptually that would look


something like this. You have your input, maybe it's an image, right? And you put it


through the network and it becomes some intermediate representation, right? That's x zero, that's


x one and or maybe we'll call it even h, h one hidden representation, right? Then that


becomes then through the layer becomes h two and so on, right? So these, this stuff here,


these, this would be weight matrices, w one, w two, that transform the image into a new


image or whatever. So what they're arguing is that, well, if you only consider a single


layer like the first layer here, it's kind of the same if you only consider the second


layer with the h one now as the input, right? It's pretty natural to see each layer of the


neural network is kind of like its own transformation, taking inputs and producing some outputs.


So what people usually do with the very first input here with your data in machine learning


generally is so-called whitening the data, which means that they have this over here.


The data is whitened. I can't find it. But what it means is you basically want to, if


you have data, let's say here is a coordinated axis, you have 2D data and you want to, you


might want to do a kind of a linear regression on it and you have data that's kind of like


that, right? It suits you to transform this data into by, first of all, looking where


its mean is, mean is about here, then subtracting that, so here, here, and then kind of dividing


by its standard deviation in each direction. So there's a standard deviation here and there


is a standard deviation here. So you would transform this data into something like, maybe


something like this. So you see that the mean, the mean is now in the middle and it's not


so elongated anymore. So you have a much easier time to kind of learn something on this data


than on this data over here, simply because our classifiers usually tend to rely on like


inner products. And if you do an inner product here, you have one of these vectors here and


you do some inner product, it's always going to be far away from the mean and thereby the


inner products are going to be large no matter what, right? Whereas here, if you take a random


one and then another random, so if you take two random points here, they're two vectors


from the mean are almost the same. Whereas if you take two random points here, they tend


to look uniformly in the direction. So it's kind of the sense we know that machine learning


methods work better if you widen the data first. So these people ask, hey, why do we


only do this at the very beginning, right? Why don't we, if each layer is basically takes


its input and learns something, each layer is basically a machine learning method, why


don't we just widen the data to every single layer, or, you know, every single sub component


of a deep network. And that's the kind of basic step here. So they argue how this has


been kind of tried before or what kind of methods you would usually get, and why these


aren't so good. Mainly because you kind of need to intermingle this whitening with training


the network. And thereby, if you just go about this naively, then you would not you would


not you kind of produce artifacts from training. So that's that's this section, this section


here, where they argue that it's not really, you can't really go about this super naively.


But what they do isn't super complicated, but they just do it in a smart way. So we'll


jump directly to that. What they say is, okay, let's look at what they call normalization


via mini batch statistics. Right? Let's say we have some some D dimensional input x, right?


And we're just going to look at per dimension. So we only care about per per individual dimension,


normalization. Alright, so what do we get? What do we need to do? When it take the kth


dimension, we're going to subtract from it, the mean of the kth dimension within a mini


batch, right within a mini batch of data. So mini batch, maybe something like 32 examples,


or 100 examples or something like this, and then we'll divide by the variance of that


mini batch. So this is this is done over here, in basic, so you compute mu B mu of the mini


batch, which is simply the empirical mean of that of the data at that particular layer.


And then you compute sigma squared B, which is simply the the empirical estimate of the


variance of that of computed on that particular mini batch. And then you transform your data


by subtracting that, and by dividing it by this and this, this, this constant here is


simply to prevent from dividing to by, you know, two, two small values. So you get like


in numerical problems. So what does it do? It does basically what we what we did above.


But now, what they say is, okay, we want to make sure that this transformation can potentially,


you know, represent the identity, because sometimes, or like a natural, natural, if


you had to do something with your input, when giving it to the next layer, like the very


baseline is to do nothing to it, right to do the identity transform. But if you do,


if you do this, you probably won't end up with the identity transform, except if the


mean is exactly zero, and the variance is exactly one, right. So what they say is, okay,


we'll also introduce two new parameters to this here is this gamma, and this beta here.


And these are learned, like other parameters in the network, we learn the parameter gamma


and beta and gamma and beta are simply gamma is simply a scalar that this transformed x


is multiplied by, and beta is simply a scalar that is then added to it. So in each dimension


of your hidden representation, you basically learn how to scale it, and how to shift it


scale and shift after you've done the normalization. So first, first you do the normalization.


Where is it? Right? First, you go from this type of data to this type of data. And then


you say, well, but maybe it's actually more beneficial to, you know, have it not centered,


or whatever. So so that the network can actually learn them to transform this somewhere. This


might seem this might seem redundant, but it's really powerful. Because what you're


basically saying is that, okay, this probably isn't the best, you know, distribution, this


probably is better. But if the network kind of, if the backpropagation algorithm or the


training algorithm decides that this first representation was actually useful, it has


the option of going back. But it also has the option of going to any other kind of form


of distribution. So so it's, it's pretty powerful in terms of what it does. Okay, it's not really


correct here that it has the power to go to any distribution, because it's only kind of


per dimension scalar that it learns, but still, it the potential to transform the distribution


by these learned scalars is is pretty big. Alright. So basically, that's it. That's,


that's, that's the whole, that's the whole shebang. You normalize your inputs to each


layer by this formula. And then you introduce new parameters that you learn along with your


network parameters. So this kind of has some implications. First of all, one implication


is this here. If you build a batch norm into your network, it kind of learns this this


plus beta, which is basically a bias parameter. If you think of a traditional kind of fully


connected layer, this isn't the fully connected layer, because the scalar here is only per


dimension. But the bias in a fully connected layer is also just per dimension. So the beta


is equal to a bias in a fully connected layer. So if you have a batch normalization after


or after a after a fully connected or convolutional layer or anything that can or sometimes has


a bias parameter, it's almost not worth it to kind of learn both. So you would rather


just only have the one from the batch normalization and leave and use the convolution or fully


connected layer without a bias. So that's kind of one implication. Another implication


is we have just lost the kind of the the ability to have deterministic test time inference.


So much like dropout, which is kind of random dropping out of nodes here, we have quantities


that depend on the mini batch. So not only the individual sample, but they actually depend


on what other samples are randomly selected to be trained with that particular sample.


So that's, that's kind of awkward if you kind of want to have some deterministic reproducible


thing at test time. So what people do is and here, this is discussed, what people do is


while training, they use these quantities, the did the quantities we just discussed,


but they keep kind of a running average over them. So what I would do is in each mini batch,


I would compute this mini batch mean and this mini batch variance. And I would keep quantities,


I keep running averages of them, right? And at test time, I'm going to plug in these running


averages. So there's nothing dependent on the mini batch anymore. That's so that that's


that's pretty neat trick, I think. And you can you can even imagine like at the end of


your network training simply using these here to kind of fine tune the weights to these


exact parameters. So that's one thing that's, that's kind of, you have to pay attention


to. So in usually neural network libraries, there are there are parameters you can set


whether or not this network is in train mode or in test mode. And depending on that the


batch norm layer will use the mini batch statistics or will use the kind of all over data set


statistics. Alright, the second thing is training. So how do you actually train this thing? Because


now you can't you can't just write we we started with our with our multi layer network up here.


Right f2 f1, right? First, I'm going to put my things through f1. And then I'll put my


things through f2. Right and the the back propagation here is is quite easy. So let


me get rid of this. The back prop here is quite easy. You go to L and maybe you want


to drive it by theta one. Right, so you first going to drive it by the hidden representation


one and then the hidden representation one with respect to theta one. So the hidden representation


would be whatever comes out of here, or h1, sorry, not I, and so on. So you kind of chain


rule your way through here. But now in between these layers here, you have these batch norm


things. And so that the authors discuss how we now do back propagation in the face of


these things. So here's basically what they discuss. It actually pays to to have a graph


of what's going on. So here is x, this is the input to our layer, right? So what do


we compute from x, we compute mu, let's just call it mu, or mu B, it's called here, right?


This is the mean of all the x's. So this is x, x i until x, well, x one until x n, this


is the mini batch. We compute the mean, and then from this, and from this, we can compute


this estimate of the variance, right? We need both. Right, so we now have the mean and the


variance over the mini batch. So we're going to take one of these x's, just the i th one,


right? And we're going to use this and this to compute x, what compute x? Is it called


hat? Yeah, probably. It's called x hat, right? Yeah, we saw about x hat. So x hat is x hat


i is x i minus mu B divided by sigma squared B, the square root of it plus this kind of


little constant here. We're gonna leave away the little constant for clarity sake. Actually,


that's in the calculations here. But so then we have a new parameter gamma, right? And


we're going to use it and our x hat to compute and also this beta here to compute y hat.


y or y, just y. And of course, this is i, this is i. So and this here is our final output


of the layer. So you can see now the back propagation paths if you go through here.


So the back propagation path, if we have some loss coming in here, we back prop through


y i, right? So here is the L the loss to y i, that's here, right? So if we want the for


example, the back prop with respect to beta, what we do is we simply and this is this is


over the mini batch, of course, we simply back prop here through this path. So in our


in our formula for beta, there should be only mentioned y i. And that's what we see here,


right? In our formula for gamma, there should only be mentioned of y i. So because the path


leads only through y i. Oh, no, I'm sorry, actually, because the of the what I mean is


of the derivative with respect to y i. Of course, the we also have to pay take into


attention that this is multiplied here by this x hat i where of course, that's not the


case when we just add something because the derivative of two of an addition, like x plus


b with respect to b, disregards x, whereas if it's x times b, it doesn't disregard disregard


x. All right. So if we Yeah, so you can you can go back. So the interesting bit basically


comes when we want to find out, okay, how because here is here is another layer right


down here somewhere, there is another layer. And we basically want to know this input here


to the next layer, how do we compute it in the face of this mess here? Because we it's


not it's not so easy, right? So you have to see we have three paths here, we go back through


x and let me get rid of these blue lines. We go we go back through x hat directly to


x. We go one path is through here. And one path is through this this mu. So basically


have to compute derivatives with respect to sigma squared, and mu and for that we need


the derivative with with respect to x hat. So basically, the way back prop works is you


just find all paths from where you are, to where you want to go. And then you you kind


of iteratively compute this. So this one here is the easiest, the easiest, as you see here,


they did it on top. Well, first, they did this one, which is simply going from y to


x hat i start, then they go from x hat i to sigma squared, which simply involves kind


of the reverse operations of how you got it. This is simply a derivative formula here of


the of the division by square root. Then you can use this you can use this quantity here


to compute that. So basically, you just go in reverse of how you computed the operations


in the first place, we said we needed mu B to compute sigma squared B. Now we need the


derivative with respect to sigma squared B in order to compute the derivative to mu B.


And once you have that, and you see the addition here, the add here is the fact that whoops,


is the fact that two things contribute to mu B. So two paths lead to lead to mu B. One


path is from from here, and one path is through here. Right. So here, there should be a green.


Since two paths, you have two components to your derivative, and you add each of them.


So that's how that's to be. And then this here, with respect to this x here, we have


three paths, right, because we have three arrows going out of x i one here, one here,


and one here. So we have to take into account all of them. Right. So this one is pretty


easy. That's the first one. Then the second one, I'm sorry, this, the second one goes


through this mu B, which we've already computed. And the third one goes through the sigma,


which we've also already computed. Right. And these are added. Because all the paths,


you have to add all the paths in the back prop algorithm. Maybe we'll do a video on


back prop later to get to really dive into how this works. And finally, they they compute


these, these we've already discussed. So in essence, the whole thing is differentiable.


You just have to kind of pay attention how to do it. But the whole thing is differentiable.


And thereby, you can basically back prop through a network that has these batch norm layers


in a built in. So that's pretty cool. I just want to quickly jump over to the results.


And yeah, keep in mind, this paper is from 2015. So networks weren't that big. Back then.


We didn't know that much about training yet. But the interesting thing is they basically


discovered, look, we can we can have drastically fewer steps in order to reach the same accuracies.


And these are kind of the activations of the network over the course of training. So without


batch norm, you see, especially at the beginning, there's large fluctuations in the activations.


And because because they use batch norm, now, there's no such thing. So basically, the reason


for that is pretty simple, right? While you learn, and you learn your layer representation


here, let's say there's x and x is fed through layers, and there is hidden representations,


each in between, right? So you're trying to learn all these parameters, let's say this


one here, w three. But at the beginning of training, everything is kind of prone to shifting


around a lot. So when you change w one, that kind of changes the entire distribution of


your hidden representations after the fact. So basically, whatever you learn for w three


is now already almost obsolete, because you've changed w one, basically, and w three was


kind of assuming that its inputs would would remain the same, because that's what you assume


in machine learning, your input distribution is kind of the same. So that's why at the


beginning of training, you see these kind of large variances. And with batch norm, this


tends to go away. So that's pretty cool. They also kind of show, they mainly show that they


can reach the same accuracies as other as other training methods, but with much, much


fewer steps, and they can go much higher learning rates than others. So because because of that,


so that's pretty cool. I encourage you to you to check out the rest of the paper use


batch norm in your network sometimes works sometimes doesn't work strangely enough. But


GPIO Basics with LED light - Raspberry Pi and Python tutorials p.6

What's going on everybody and welcome to another Raspberry Pi tutorial. In this


tutorial what we're gonna be doing is introducing the GPIO pins on the


Raspberry Pi, how to use them and all that. So what we have here is one breadboard,


two male to female jumper wires, one resistor, the resistor should be


somewhere between like 301k ohms and then an LED light, this one's yellow, it


really doesn't matter what color you choose though. And basically what the GPIO


pins do for us is they allow us, GPIO stands for general purpose input output


and it allows us to either send out a signal either high or low so just a


binary signal out or receive a signal back in and there's all kinds of ways


that we can manipulate that to have a lot more dynamism to it. Now the GPIO


pins are labeled for us and I'm gonna pull up this image from Adafruit and


these are the GPIO pins above that kind of gray line that's the older pies and


if you have one of the more later versions you'll have all 40 pins and you


can go all the way down there towards the bottom of that image. Now the way


these are oriented if we go back to here it's as if it's the same way this pie is


oriented in this image or this video right here so that translates directly


to these pins here so the top leftmost pin is 3.3 volts and then the two top


rightmost pins are your 5 volt outputs. Now what we're gonna do is we're gonna


rig up a simple LED light that will just turn on when we ask it to via our


program. To do that and to create the circuit we're going to be using this


breadboard here so this is breadboard the way that it works is basically all


on these edges here sometimes you'll have like two rows and breadboards will


look a little different but they're generally gonna look something like this


and these are connected vertically basically so by column I suppose you


could say and then you've got these middle bits each of these are connected


by row so for example let's say this this spot here is connected to this one


is connected to this one and so on but these are not directly connected to


these you'll connect them yourselves and then like this one this closest one I


suppose this one is not connected to this one or this one but this one is


connected to this one so to help you understand that even further I've just


kind of circled or squared some examples of ones that are already connected but


of course you can connect them yourself with jumper wires or we'll use the


resistor to do it and so on and then even the LED bulb will use. So what we


want to do is wire this all together so here's a quick diagram of the actual


wiring that we're gonna do now this is not a real wiring diagram by any means


but I know whenever I was first starting out the last thing I wanted to see was a


wiring diagram I wanted something a little more obvious to me so basically


what I've done here is this green is the Raspberry Pi the called the filled in


boxes by color or the actual jumper wires so black to black and so on and


then the resistor is this like blue blob and the LEDs right here and it's labeled


and all that so it's just like super simple so coming back to our Raspberry


Pi now let's go ahead and connect everything and let me just pop up the


diagram there so it's actually kind of a little hard to see if your screen is not


very big at least that yellow so the yellow here is basically six pins down


and then three pins down for the the ground so if you want you can take a


screenshot of this also you can go to the text based versions tutorial and


save this image it's always good to have an extra one around so anyway yeah


ground here the third pin down on the right and then we'll also be using GPIO


pin 18 also just for the record these are the Broadcom names it's a little


unfortunate the Raspberry Pi has like the board names for the pins and that's


just like the physical location basically of the pin you can use the


board name pins or the Broadcom names and the Broadcom that's the processor


that we're using that's you know the actual chip that's how it recognizes


those pins that's the name of that pin according to the chip so we're gonna use


the Broadcom names okay so so let's go ahead and wire this up just going to


connect here this this blue cable will be my ground so we'll go ahead and


connect everything up nothing's plugged in so it doesn't really matter the order


that we do this but yeah so ground here and then the female end for the actual


LED here and then ground will connect right here and then we'll use the


resistor


here and there


and then next to the resistor will be the LED bulb and now one thing to take


note of let's see if you can see that you notice that there's one end that's


longer than the other not all LED bulbs are gonna have that but when you do have


that the positive end is the longer end so since this is the ground and we


actually want the negative end to be on that side so we'll go ahead and plug


that in if for whatever reason your circuit doesn't work flip them around


you probably just made a simple mistake otherwise the bulb could be burnt out


there's all kinds of reasons why it wouldn't work or your circuits bad or


whatever okay once we've done that now we can just plug in for the actual


resistors kind of touching we don't want them to be touching you don't want metal


to metal there now in reality you'd actually trim that resistor down so it


wasn't sticking up so much but yeah make sure the resistor isn't directly touching


say the positive end of the the GPIO cord anyway or the LED rather okay so


I've got everything hooked up and again it's according to this diagram if you


need more pictures I've got some better pictures of the actual physical wiring


of this thing in the text-based version so if you're still confused or whatever


definitely check those out but yeah so now let's go ahead I'm gonna plug in the


Raspberry Pi we're gonna fire it up and then we're gonna get to the actual code


to make this light turn on and off


right with the Raspberry Pi all connected turned on I'm in the terminal


via SSH let's go ahead and change directory to the desktop and also let's


just make sure we've got RPI GPIO chances are you already have it but


let's just make sure so sudo apt-get install Python dash RPI dot GPIO and we


already have it so it's no problem so now I'm gonna go ahead and do is we're


gonna nano LED example dot PI and now we're gonna go ahead news import RPI dot


all caps GPIO now mind the caps that's capital R capital P lowercase I dot all


caps GPIO import that as GPIO then we're gonna also import time and then now


we're gonna do is GPIO dot set mode GPIO GPIO dot BCM so BCM is short for broad


com the other option is if you wanted board names but I'm gonna use the BCM


broad com names because I just think it makes more sense and we just want to


make sure we're using the same names as as we think we are anyways you wouldn't


want to mess up the pins so now we've done that we're ready to set up the pin


so like like I was saying before our pin that we connected to is pin 18 for let


me pull it up here so we connected that third from the top right ground you


could have connected it to any of the grounds it's not a big deal which one


but our actual pin that's gonna send an output signal to that LED bulb to


actually turn it on is going to be GPIO pin number 18 again you could have


plugged it into any of the other GPIO ports it's not essential that you do the


exact same port or pin rather as I do anyway what I'm gonna do now is GPIO


dot setup and we're gonna set up pin number 18 and we're gonna call that a


GPIO dot out it can also take an input so it could be GPIO dot in now we're


gonna do is GPIO all caps GPIO dot output and we're gonna output to pin 18


and we're gonna output a high signal it could be GPIO dot high or low those are


the two signals that you can output from your GPIO so we're gonna do high and


then we'll go ahead and time dot sleep we'll do five seconds and then we'll do


GPIO dot output again to pin 18 GPIO dot low so a low signal and then when we're


all done let's go ahead and run GPIO dot cleanup and that should be everything


that we need so let's go ahead and exit that save yes cool let's go ahead and


run Python LED example dot PI and let me just pull my screen over here just so I


can make sure that shows up on camera no problem go ahead and run that hard to


tell with that yellow light but it is on and I think we use five seconds yeah and


now it's off so hopefully yours turned on and off minded there okay so that's


just like a really quick basic example of using the GPIO that's one way to send


out a value now what we're gonna do is use that distance sensor to actually


input bring back in some data from our from a sensor so that's what we're gonna


be doing in the next tutorial a little slightly more complex setup but it's not


too bad so that's what we're gonna be doing the next tutorial if you have


questions comments concerns whatever up to this point feel free to leave them


How far can we scale up? Deep Learning's Diminishing Returns (Article Review)

Hi there, I saw this article in IEEE spectrum called deep learnings diminishing returns,


the cost of improvement is becoming unsustainable. This is by Nielse Thompson, Christian Greenwald,


Qihong Li and Gabriel F. Manso. And I thought it was an interesting read, because it talks about


the computational limits that we're reaching with deep learning today. And I have it over here in


annotatable form, though it might not look as pretty. I think the article, it leads up to the


point where it shows just how much compute will be needed to make further improvements in deep


learning and what the consequences of that might be, and some of the ways that people are trying


to get around it. Now, I don't agree with everything the article says, but I think it's a


it's a pretty neat read, it's pretty short. So I thought we can talk about it a little bit. So the


article starts out with essentially praising deep learning for achieving so many things, for


example, translating between languages, predicting how proteins fold, and many other things playing


games as complex as go, they say it has risen relatively recently, but it has a long history.


They mentioned 1958. And Frank Rosenblatt at Cornell, they designed the first artificial


neural network, they say Rosenblatt's ambitions outpaced the capability of his era, and he knew


it. Apparently, he said, as the number of connections in the network increases, the burden


of a conventional digital computer soon becomes excessive. So why are deep neural networks working?


Because of course, computers have increased in power massively, just for computing power, there


has been whatever a 10 million fold increase according to Moore's law. And that's usually just


measured in something like CPU instructions. And now we went even beyond that building special


purpose hardware such as GPUs, which aren't actually special purpose for this, but also TPUs.


So they say these more powerful computers have made it possible to construct networks with vastly


more connections and neurons and hence greater ability to model complex phenomena. And of course,


these are the deep neural networks that power most of today's advances in AI. They draw a


comparison right here, they say, like Rosenblatt before them, today's deep learning researchers


are nearing the frontier of what their tools can achieve, essentially claiming that we are in a


similar situation today, we have the models that can achieve things. And we know pretty much that


scaling them up can increase performance. However, we're kind of at the limits of how much we can


scale. For example, I reported on this that Sam Altman apparently said GPT-4 will not be much


bigger than GPT-3, it will be trained more efficiently, will have some smartness in it on


how it's processed, it will use more compute, but it will not necessarily be that much bigger in


scale. So the first thing the article touches about deep learning is the fact that deep networks are


over parameterized. For example, the noisy student model has some 480 million parameters yet is


trained on only 1.2 million labeled images, which is the ImageNet data set. Now of course, the noisy


student model, if I understand correctly, also may leverage unlabeled data, but granted today's


neural networks are massively over parameterized, they have more parameters than data points


available. Therefore, they should horribly overfit, but they don't. They say classically,


this would lead to overfitting where the model not only learns general trends, but also the random


vagaries of the data was trained on deep learning avoids this trap by initializing the parameters


randomly, and then iteratively adjusting sets of them to better fit the data using a method called


stochastic gradient descent. Surprisingly, this procedure has been proven to ensure that the


learned model generalize as well. Now, I'm pretty sure that we are not yet sure why exactly deep


networks don't overfit or why they generalize as they get over parameterized. I know there are some


proofs around SGD and so on. But these proofs usually require assumptions that just make them


completely lose touch to reality. But the core message is true, deep networks are over parameterized,


and that is probably one of the reasons why they work so well. And being over parameterized, they


are quite flexible, they say at the good news is that deep learning provides enormous flexibility.


The bad news is that this flexibility comes at an enormous computational cost. This unfortunate


reality has two parts. They say the first part is true of all statistical models to improve


performance by factor of K, at least K squared more data points must be used to train the model.


Does this really hold for all statistical models? Is this from the same theory that says the


statistical models should overfit when they're over parameterized? I'm not sure. The second part,


they say of the computational cost comes explicitly from over parameterization. Once accounted for


this yields a total computational cost for improvement of at least K to the fourth power,


meaning for a tenfold improvement, you would need to increase the computation by 10,000. Now


regardless of whether you think the theoretical analysis is actually accurate here, again,


this is from the same area that says these models should overfit horribly, it doesn't matter,


because these people have actually collected data. And they say theory tells us that computing needs


to scale with at least the fourth power of the improvement in performance. In practice,


the actual requirements have scaled with at least the ninth power. So when you actually measure how


much people need to scale computation in order to achieve a given performance, then it's actually


it's much worse than the theory predicts. In fact, they have these neat graphs right here. So on the


left, you can see the percent error, I believe this is the ImageNet classification data set. And


on this axis, you can see the time. Now here, you can see that over time, as time progresses,


the error has come down and down and down again, as new state of the art models were proposed,


ever since the 2012 success of Alex net. And if you extrapolate that you can pretty clearly see


that around 2025, we should be at approximately 5% of error. See, I thought you'd had to actually do


something to reach a new state of the art on ImageNet. But as it turns out, we just need to


sit here and wait until 2025. Okay, jokes aside, they overlay this graph with another graph right


here. And that is the comparison of again, percent error on the y axis. But now it's not the year in


which the achievement was made. But it is number of computations in billions of flops. And notice


the log scale down here. Now I have to say this graph right here makes it pretty clear that there


might be something like a relationship, even maybe a linear relationship that you can extrapolate


right here, I'm not so sure like these models are up here and then goes like here and then it goes


here and then it goes here. And then it goes over here to 2020. And really without that, you probably


have a line that goes something like this. Now in any case, if they do actually the line that they're


doing, then you can see that if you extrapolate the same thing to this 5% error rate, you do end


up at something like 10 to the 18 flops. And they also compare this to the equivalent carbon dioxide


emissions. For example, right now we are somewhere between the co2 generated by the average US


resident in one year and the co2 generated by the average US resident in a lifetime, the current


models somewhere in between to train them once if you actually extrapolate this to the 5% error rate


to the 10 to the 18 flops, then it becomes suddenly co2 generated by New York City in one month. So


the entire city of New York City for one month is the same as GPUs go brrrr to train ImageNet. Now


that is pretty shocking. I have to say, you know, it checks out they have done the research, they


extrapolated correctly here and they come to this conclusion, the co2 equivalents, I'm sure they are


measured correctly and so on. I do have several problems with this though. The first one I already


said the zigzag in this graph right here doesn't really suggest that you can simply extrapolate


over these advances. Also the 2020 point seems to be quite out there. So if there was any


architecture search involved, if there was any giant free training involved or anything like


this, I'm sure like that that adds to the co2 emissions, but it doesn't say that you cannot


achieve the same thing with something else. So whether the slope of the line is really the black


one right here, or more like the blue one I drew, it makes quite a bit of a difference actually


makes a exponential difference. So I'm a bit doubtful that you can really pinpoint this 5%


error point to five years in advance. Okay, it's 2022 now so three years but still and speaking of


co2 equivalents, not all energy is equal. For example, Google prides itself in being zero


emission. Therefore, if Google trains a model, there is no co2 equivalent, presumably. Now I


think carbon neutrality and zero emissions and words like this are sometimes a bit of a scam,


but still not all energy is equal. And especially these large companies, they can distribute their


workload across the planet to where the energy is used most efficiently. And lastly, and this I


think should really the main point here is that we have made advances. None of these achievements


here that we've made over the past years are only scaling up the scaling up always came with some


sort of invention that made it more efficient or more viable to scale up residual networks,


all of a sudden could scale to many, many more layers because of the invention of the residual


connection or the addition depending on who you ask. So the residual networks became bigger and


deeper without having to waste more computation. In fact, they had less parameters than many


equivalent models of the time. So I don't think we should neglect the inventions we make along the


way in order to scale up. Now, of course, people are always going to put in whatever flops they


have in order to achieve the best possible number. But I think for most of these advances, it was


really new inventions that triggered the usage of these flops rather than the other way around. And


the authors of these articles actually agree a little bit they say, is it really reasonable to


extrapolate like this and extrapolating this way would be unreasonable if we assume that researchers


would follow this trajectory all the way to such an extreme outcome. We don't face with skyrocketing


costs, researchers will either have to come up with more efficient ways to solve these problems,


or they will abandon working on these problems and progress will languish, which is true. So


rather than being a warning cry about we're going to waste an entire city's co2 emissions for a month


for one model, it's more of a warning against we're going to have to come up with new methods


and different ways of training these models. And we can't rely on scale to bring us advances. They


also give some money numbers right here. They said for example, DeepMind traded system to play go,


it was about $35 million in cost when they trained AlphaStar, they purposefully didn't


try multiple ways of architecting an important component because the training cost would have


been too high. In GPT-3, they made a mistake, but they didn't fix it due to the cost of training,


it wasn't feasible to retrain the model and so on. And also mentioning that GPT-3 cost about 4


million to train. Now, yes, of course, researchers that train these giant models comes with


substantial costs. So you have to think twice if you really want to do your grid search and whatnot.


So the experimentation methodology has become a bit different. But also you have to keep in mind


these big numbers $35 million, $4 million, and so on. First of all, this isn't really that much in


comparison to what the people costs that worked on the model. And second of all, this is almost


necessary. All of the models that we see today have cost substantially more in the past to train,


but someone had to do it first. I can only train BERT today because Google has invested ginormous


amounts of resources trying out how to train it training the first one at considerable cost. And


only after that have other people jumped on prices have come down training got more efficient. And


now I can do it from the comfort of my home essentially on a collab or on my home GPU. And


isn't this the case with all inventions somehow at first, it's just a few, it's really expensive,


because it's custom because we haven't figured it all out yet. And then over time, cost will come


down, efficiency will go up and the easiness is just much better. So rather than saying, Oh, wow,


deep mind spend $35 million Oh, no, I'm like, cool. You know, since they're doing this, two, three,


four years, I will be able to do so for simply 2 million and pay, you know, so the article gives


some solutions to that different avenues, though they are mostly a little bit pessimistic about


most of them. So first of all, they said you can use specific processors designed specially for


deep learning. Now the newest generations of GPUs are actually a little bit tuned to deep learning,


but there are also tensor processing units. And there are a number of other hardware vendors that


try to get into the space of specifically building chips for deep learning. What they criticize here


is the fact that this hardware has to do trade offs, they have to increase specialization for


generality. And also with specialization, you face diminishing returns. And of course, the more


specialized you are, the less you can invent new things because you're essentially locked into what


the hardware can do. They also discuss training networks that are smaller, but they criticize that


often this increases the training costs because you essentially train a big network and then you


train again to make it smaller to distill it. And that's also not the solution to reducing training


cost. But it might be a good solution if a model needs to be trained once and then largely runs in


inference mode, such as GPT-3. They also discuss meta learning where you essentially train a good


initialization for a lot of problems. And then you transfer that initial solution to new problems. So


if you have a good meta learner, they will be at an excellent starting point for solving new


problems, therefore reducing the training cost in each of these new problems. But they also mentioned


that and I agree meta learning is yet at the stage where it doesn't really work. The training you put


into the initial meta learner doesn't often pay off to new problems. Yes, it works in papers,


but in papers, you already know which other problems you're going to measure it on. So


they say even small differences between the original data and where you want to use it can


severely degrade performance. Now they also mentioned this paper right here Benjamin Recht


of the University of California Berkeley and others have made this point even more starkly


showing that even with novel data sets purposely constructed to mimic the original training data,


performance drops by more than 10%. Now I want to highlight this a little bit because this talks


about a paper called do ImageNet classifiers generalized to ImageNet. This is also usually


called ImageNet v2. Because what these authors did is they tried to follow the protocol of the


original ImageNet data collection as closely as possible and come up with a new test set the so


called ImageNet v2. It's not a training set is just the test set. And they show pretty convincingly


that for any classifier that performs in any way on ImageNet v1, its performance on ImageNet v2


will be something like 10 points lower. It's a fairly straight line. So this is what the


article talks about. However, the article doesn't talk about this paper right here called Identifying


Statistical Bias in Data Set Replication by MIT and UC Berkeley, which shows pretty convincingly


that there is in fact a difference between the data collection mechanism of ImageNet v1 and v2.


It is a subtle difference, but there is a difference nonetheless. That difference makes


it such that there is a significant difference in what kind of images are chosen for the two data


sets. And when you correct for that difference, then this drop in accuracy for ImageNet v2 almost


entirely vanishes. Now, okay, the article is right in first instance, there is a small difference


between the original data and the new data. And that severely degrades performance. But this


particular difference in performance is due to the new data set having a different methodology,


and that directly makes the samples harder. It's not like the samples are different in some sort


of a there are different kinds of images is that very directly because of how they collected them,


they are more difficult to classify. It's the same data, but more difficult. So we shouldn't be


surprised that performance drops by 10%. In this particular instance, I just thought it was


interesting to mention since the article specifically focuses on this paper right here,


and I don't think this paper is a good example of what they're trying to say. Okay, so what's


the conclusion to all of this, here is the final recommendation that the article makes to evade the


computational limits of deep learning would be to move to other perhaps as yet undiscovered or


underappreciated types of machine learning. And of course, what they mean is that they want to


bring the insights of experts which can be much more computationally efficient, and that we should


maybe look at things like neuro symbolic methods and other techniques to combine the power of


expert knowledge and reasoning with the flexibility often found in neural networks. Now, why does


every discussion about the scaling of deep learning always end with well, we should use more


expert systems and reasoning and logic and the neural networks don't understand anything. Now,


granted, it is okay to suggest this, it's probably a good way forward. But as of yet, as of now,


the neuro symbolic systems are actually just the expert systems as well, they are so so


not good. And of course, that's the case with any young research topic. But just because something


is computationally efficient, it doesn't mean that we should switch to that because of it. Now,


I'd be super duper happy if symbolicism makes a comeback, if we could somehow combine algorithms


and deep learning, if we could combine reasoning and knowledge bases and input from domain experts


and all of this. But as of today, that is not really a benefit. It's more like a substitute.


So you can make machine learning more efficient by inputting lots and lots of priors from domain


experts. That's completely cool. But what we've seen over and over and over again, is that as soon


as you give the ML system enough data, it starts to outperform these experts. And I think what I'd


like to see from a neuro symbolic system or anything like this is that in fact, it does


outperform even the most data hungry machine learning methods that the symbolicism is not just


a substitute for more data, but an actual improvement over any data that I could find.


And that's just something that I personally haven't seen you might disagree, but I haven't


seen a convincing argument yet that that is the case for any of the symbolic systems we have today.


computational efficiency alone is simply not enough. But hey, tell me what you think. What do


you think about this article? Do you agree with them? Do you not agree with them? I'll link the


Jukebox: A Generative Model for Music (Paper Explained)

Alright, so what you're hearing is the OpenAI Jukebox.


This paper came out and it is a surprisingly good quality generative model for music including


lyrics, so including singing, which I believe is pretty novel and the fact that it works


so well and has musical consistency throughout entire songs is something that is very very


novel and cool.


Alright, so we're looking at the paper it's called Jukebox a Generative Model for Music


and it's by Profola Darival, He Wo Jun, Christine Pine, Jong Woo Kim, Alec Radford and Ilya


Sotskever of OpenAI.


So the model here is a is not very novel, but I think the the way they set it up makes


a lot of sense and we'll quickly go through it and look at what comes out of this.


This is not a very technical paper, but I think it's a it's a it's well written and


their main thing is these VQ VAE models.


So these models if you know you might know what a variational autoencoder is.


So in a variational autoencoder I have an input, let's say an image here of our typical


cat and I put it through what is called an encoder in order to get a hidden representation.


Now this hidden representation usually called something like Z or H is then in a in a classic


autoencoder I would here have a decoder and then I train that decoder to match that and


that original picture as closely as possible.


So I train these two neural networks the encoder and the decoder to just basically compress


to this representation and then reproduce again the original image and thereby I think


I can learn a good hidden representation.


Now in a variational autoencoder what happens is there is an in-between step namely this


Z representation here is not directly fed to the decoder but this representation is


actually used to parameterize Gaussians.


So in the easiest case we have let's say we have six dimensions here in the hidden representation


the first two are used to parameterize the first Gaussian one is going to be the mean


and one is going to be the standard deviation of that Gaussian and the second two are going


to parameterize the second Gaussian and the third are going to parameterize the third


one so we have now three Gaussians and then from those three Gaussian distributions we're


going to sample a three-dimensional vector and then we're going to feed that three-dimensional


vector to the decoder.


So every input image in essence is giving us a distribution in the latent space and


not just a single vector it's just it's giving us a vector that describes an entire multivariate


Gaussian distribution and then we train the decoder basically to reconstruct the encoder


given that distribution.


So the variational autoencoder has improved over the classic autoencoder because it tends


to circumvent some of the shortcomings but still variational autoencoders have their


problems and in terms of generative model for images people as you know have gone to


things like GANs and so on.


But here this new thing is called the VQVAE and that's because I believe this stands for


vector quantization not entirely sure honestly but so what it does is it takes the input


right here and again it maps it to a latent code and these latent code I believe are here


called H so it maps it to a vector called H.


Now here's where it gets different we do have a so-called codebook here so a codebook is


just a list of vectors this is the codebook and these are here called E.


So what we'll do is we'll simply look for H which one's the closest neighbor in the


codebook that we have of hidden vectors and we'll map it to that and say ah this is this


here is the closest neighbor so we'll basically quantize the hidden representation to these


codes so we end up only with vectors that are these codes and now instead of saving


the vector H first of all we get a super compressed representation because if these are like 16


codebook vectors we can just simply enumerate them right and then we can simply encode the


image as one or a sequence of these indices here but second of all it tends to bring a


more kind of a diversity and accuracy if we then decode from these these code vectors.


Now of course everything is trained here so the encoder is trained and the codebooks themselves


are also trained and the decoder is also trained to give you the maximum kind of benefit.


So this is what is described here what they have is first of all their loss for these


VQVAEs and you'll see how they are using them to generate music in a second their loss is


part this reconstruction loss which you can see here this is the original image.


How far is it away from the decoded hidden quantized representation this E here this


is the quantized the codebook vector that belongs to the hidden representation of X


so this is your standard reconstruction loss.


The next part of the loss is this codebook loss now the codebook loss is for training


these codebook vectors so it basically pulls the codebook vector closer to the actual hidden


representation so that is where you train these this here is a stop gradient so basically


just means that you want your codebook vectors to be better representing of the data that


you feed in because otherwise it would be useless codebook vectors and the third part


of their loss is the these commit loss right here and that is exactly the opposite now


you put the stop gradient on the codebook vector and you simply want to pull the hidden


representation closer to the codebook vector such that imagine the encoder must learn to


approximately hit one of these codebook vectors right otherwise it can it cannot really learn


something meaningfully it must learn to deal with the codebook vectors that are there and


to approximately map the input into the vicinity of one of one of the codebook vectors otherwise


there's no information flowing so that's how you train things you train the encoder and


the decoder to reconstruct you train the codebook vectors to represent the data and you also


train the encoder to make good use of the codebook vectors honestly now that I think


about it I'm pretty sure in the reconstruction loss you might only train the decoder because


you do have this quantization step in between in this in this thing here so technically


you could not back propagate through that all right so that's how you train the individual


parts of a VQ VAE and we'll see how they use it in order to produce music all right so


what they do is they start off with a sample of music and they send it through this thing


through this and architecture and at the end they are trying to reconstruct the same thing


they had at the beginning so you see this overarching auto encoder architecture that's


why it's an auto encoder you try to reconstruct your input and thereby you try to learn something


about the data because a model that can compress and then uncompress all of your data has learned


something useful about it now you have these VQ VAE is here in the middle but you have


them at different scales so you'll have three of those here there is a coarse scale one


middle scale one and the high frequency one right so three VQ VAE is right here and you


train each one of them separately and the difference between them is that the so because


this is a continuous signal you cannot just encode a continuous signal because it's an


audio waveform what you have to do is you have to go through it in some sort of a a


stepwise fashion you have to divide it into individual pieces and encode each of those


pieces as one hidden vector or one element of the codebook and the size here the precise


size of how you go through the audio that is different between the V these different


scales so in this scale right here you go through the audio in very small steps right


this as you can imagine gives you the best reconstruction so this is the audio the audio


is like this right and you just take part of it like from here to here and you encode


it into a single hidden vector which is this brown thing here then you take the next one


the next slice and you encode it and that'll give you this blue thing here then you run


this sequence through the vector quantization step where each of those will be mapped so


you have a codebook here right you have your codebook and you look up the first one and


you decide ah this probably goes here to this code vector so you put that code vector into


the first place you take the second place and you might decide now that's this one so


you put this one into the second place and the third one you might decide ah no that


also is closest to the first codebook vector so you again put the first codebook vector


into that slot now the that doesn't mean that it's the same music but of course the the


decoder now is going to look at the entire sequence and can decide I'll probably this


isn't the exact same note as here but it might decide you know that the the chord played


will repeat or something like this so there's this vector quantization step and the codebook


look up sorry yeah this this minimizes which which vector of the codebook is closest and


the codebook look up I think will just replace then the decode the this vector with the actual


codebook things and so there's this slight difference here so Z here as you can see is


the argument K so the argument that is the actual number K which codebook vector is the


closest and then this E Z T will be the actual vectors so this here is actually what I described


right here but this is I think a detail and you you're going to do this at different scales


now you can imagine that the bottom one is going to give you the best most faithful reconstruction


when you decode it right but it is also going to learn about the kind of details in the


music the short term details whereas this course grained one it can learn things about