of the world will give back observations and it will then interact again so you


IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

can move a step forward right first first thing is moves a step step forward


and then the world gives it back a aha you are now no longer here you've moved


here right and then it's on I want to move to the left and the world says okay


so you're no longer here you've moved one to the left and this on the right


here are the observations and isn't the left here are the actions and for the A


to C is kind of necessary that we always have a current version of the policy


generating these steps in order to be able to learn from them and then the


next steps also need to be kind of current and to be learned now there have


been attempts to decentralize this and this is exactly what impala does so impala


splits this into multiple workers you can think of this as different machines


so there is a split here and these are called actors and this is called a


learner now the actors they will go ahead and they will run episodes on


their own right occasionally or they will run episodes and they will


communicate those episodes to the learner and the learner will


continuously here learn so these orange steps can be made in much more quick


succession and don't have to be like synchronized as in the A to C here is


another way of seeing this over here and we'll just concentrate on this on this


left thing here so there is a learner and there are actors and every now and


then the actor syncs its model from the learner these are different machines so


this can happen over the network every now and then the actor gets like an


update of the newest policy network and then the actor will just go ahead and


run episodes using that policy right episode episode episode episode step


step step without interfering with anything else and then once it has run


an episode or multiple once it will communicate this back to the learner and


if all the actors do this all right the learner gets a whole bunch of these


episodes and then can learn from all of them simultaneously and they can do so


in kind of with in kind of very fast succession as you see here so the work


is split of course you run into a problem namely as we saw in the A to C


algorithm this type of reinforcement learning requires basically that you


always run the episode with the current model and that's not the case here right


the actor may sink the parameters it may sink the parameters once in a while but


then it will run these episodes right when it runs these episodes here it has


no idea or it the learner in the meantime has continually been updating


the model while the actor kind of has an old model so these episodes here are run


with an old model so the learner if it tries to learn from this must kind of


correct for this fact and the big kind of theoretical contribution of this


paper is how to correct for the fact that the data you learn from comes from


an outdated policy model and this is what's called v-trace correction so


without going too much into the into the details here v-trace correction happens


as as fall it happens as follows so what you define are what's called v-trace


targets and these v-trace targets are basically the targets that you train


your value function towards right so the the the value function as we discussed


before that is a that is the thing that tells you how good each state is and the


targets you train this towards and you're also by the way using this v-trace


corrections in policy updates but these are defined as follows so the v-trace


target for step s is the value function at step s plus this correction thing and


the the correction thing basically well I'm gonna break this down some more so


the V at current s is your value function plus and this is a sum over all


future steps over and this is a discount factor and this is kind of a Delta from


one step to the next so you're in an episode and you've made some steps right


and let's say we are here right this is s and so your your little vs will be


whatever your value function says of s plus kind of a correction for each step


that you make go into the future like this and the main part of these is is


this here which is basically the reward at the step plus the difference of the


value functions of the steps after it and what v-trace introduces now is this


bit here and these CI again are computed as such so all of this kind of is very


nested so there is a there's a big multiplication here it's a very nested


thing but in the very very very core of it you can see the following these v-trace


corrections are a ratio between pi and mu and pi is the policy of the learner


that is the current policy and mu is the policy that has been used to generate


the to generate the episode and this is truncated by a minimum and usually the


C bar is one so let's consider what happens here what happens is let's say


that mu is higher than pi for a given pair of AI and XA what does it mean it


means that in the past you run an episode you come you are in this maze


right such so to them and you're here right now the and the goal let's say the


goal is down here and the action is going over here that does the action


that you're considering here now your mu which is your old policy that the actor


has synced at some point mu might say this is very good right because it moves


you towards the goal more but then your your pie the learner has been learning


since the since the agent the actor has synchronized the weights the learner has


been learning and the learner might know wait wait since you have decided this I


have actually learned that this might not be such a good move because you know


there's a wall here and I'd rather go down here and then over here so what it


will do it will since pi is low and mu is higher it will down weigh this action


and this is how you correct for the fact that there are old weights by basically


down weighing wherever the old policy thought of an action as being worth more


than the new policy does and this is how you make up for the fact that the new


policy you assume it knows better because it has learned more and thereby


you you give lower weight to the data points where the the policies have


diverged a lot so that's at the core of it and you can think of in terms of here


you can think of it as maybe here at this step you're at a point where the


old policy that the actor has has updated itself to says we should do


action one right but the new policy that the learner has in the meantime has


learned more says we should do action two and if this is the case then this


whole rest of the episode is down weight because it is no longer current


knowledge right and this is not just kind of a heuristic but they actually do


prove that this this this comes with some guarantees especially reduces to


kind of the classic reinforcement algorithms if you assume that mu is


always PI so that current policy is the old policy and therefore you're in the


old setting all right so this was a bit of a lengthy explanation of the math


behind it and at the end what you do is the following you train your value


function using this update and you can see here it's simply the gradient of the


value function scaled by the thing that contains this v-trace target right you


then you update your policy in this direction and this is the classic


reinforcement learning reinforce style policy update where here you have the


gradient of the of the policy and here you have the weighing by the reward and


specifically here it is the reward plus this v-trace target and this thing here


is a bias correction or a bias reducing sorry variance reducing bias that was


terrible the final form is what's called an entropy penalty where you want to


push the entropy of your policy up such that the agent kind of is biased


towards exploring more than exploiting if you know of the classic exploration


exploitation dilemma so that's that's what you do compute these v-trace


targets update your value and policy according to these equations and there


you go so what do what does impala do specifically in this deep mind lab if two


architectures first of all they have this death this small architecture


second they have this large architecture and they just kind of try it out on


these and they measure how many frames per second they can get in and you see


here compared to on single machine compared to a3c they bring in a lot more


frames per second this is just on a single machine but then on distributed


setting the scale up also is very significant that they reach that's it


because they don't have to wait for other things they can just go ahead


everything runs at full speed basically and everything runs in parallel and the


fact that it that some of the information is old is corrected by v-trace


and the last thing I want to show is the wall clock time I think this is the


important plot in this deep mind lab on over all the tasks the wall clock time


compared to the score you can see a3c while it does you know increase over


time the impala variants up here increase in much much faster wall clock


time so that's the that's the paper they have a lot of proofs in the appendix


which I'm not going to go over if you want to give it a try then it is it is


not called impala on github it is called I think scalable agent so on github it


is called scalable agent I think but you'll find it if you if you search for


impala github or something like this yeah other than that thanks for


I TRAINED AN AI TO SOLVE 2+2 (w/ Live Coding)

Hi there! You might have seen the recent debate about 2 plus 2 where everyone


tries to weigh in the big question being is 2 plus 2 equal to 4 or is 2 plus 2


equal to 5 and for some reason the entirety of Western civilization hangs


in the balance right here but everyone's missing the point everyone's just kind


of arguing about this but I want to point something out right here just have


a look at the accounts arguing right here you know James Eric you know what


all of these have in common they're humans humans arguing about fundamental


questions of the universe and culture what could possibly go wrong so today


we're going to replace fallible weak-minded humans by AI we're going to


build an AI that's going to answer the question what is 2 plus 2 now first


thing we're going to do is to import Pytorch if you're using tensorflow


what's wrong with you come on just checking whether CUDA is available CUDA


is basically shorthanded AI for magic so don't worry about that part now we're


going to borrow quite a bit of code from the Pytorch example because they've


already implemented sort of the same thing so the model we're going to use


right here is going to be a generative adversarial network now you might be


wondering hey is it really smart to build AI on something that's called


adversarial is that a little bit dangerous to that I say all right now so


we're going to grab the code from over here first thing we need is the model


itself now the model is composed of a generator and a discriminator the


generator is right here blink-blonk let's plop that in here that looks good


look at that generator transpose convolutions batch norms relu's this is


going to be so artificial and so intelligent won't believe it so the


generators responsible for basically outputting things in our case what we're


going to input a 2 and a plus and a 2 and then the output should be you know


whatever the result of that is now as a data set we're going to use the famous


MNIST data set this data set is a very challenging data set it's very large


data set but I think in order to tackle an important question like this we need


to go for the crème de la crème of data sets so MNIST is a data set that


contains a lot of these handwritten digits you might think these are just


numbers but these are more than numbers these numbers have a meaning so the


computer just sees this in numbers but as a human you would see this right here


see the zero this data set is filled with digits a four wow that's one of the


things we need look at that a nine beautiful beautiful so we're going to be


to try to make the network learn what two plus two is now if you know machine


learning you know that you need training data so we need a labeled data set of two


plus two equals and then whatever two plus two equals so first we're going to


filter out all of the examples where that show a two so we need to train this


network right so we need a number of training steps you know in AI we like to


train for a lot of steps let's just go for nine thousand what we'll do is we'll


train nine thousand times 64 images and the air is gonna learn what two plus two


is alright so in each step we need to create a batch of training samples what


we need is a two a plus and two so for the twos we can just select two of the


twos that we had before now the plus is a little bit more tricky so in order to


make a plus there's none in the MNIST data set you have to understand the


MNIST data set is also quite old I think it was invented before the plus sign was


invented so that's not in the data set so we have to create a plus by ourselves


it's going to be hard but we'll give it a try now I'm usually way too dumb to


use mesh grid I'm just gonna try I mean you know what can go wrong okay so as


you can see we're absolutely on the wrong track right here


ladies and gentlemen the most beautiful plus in the history of AI alright so we


got a plus and we got all of our twos so now let's put them together look at that


two plus two next sample two plus two next sample two plus two so our AI is


going to be trained on data samples just like this now in order to make the


generator accept samples like this we sort of need to change it a little bit


because if we try to just put this into the generator probably it won't work you


see there's an error the generator is not artificially intelligent enough yet


so we need to make it take samples so our samples are of size 28 by 84 and


what the generator right now expects is a sample of size 100 by 512 by 4 by 4 so


you may notice we have never made use of our batch size so let's fix that right


now so now we're training in batches of images but it's still not cool for the


generator so we need to change the generator right here here what's this


good for nothing nothing alright so expects the input to be of a certain


size and we are going to change that right here we also don't want any


strides strides are for losers and let's see where that gets us okay so we made


our generator accept images that we want and produce images of the size that we


want now the entire question here is we need labels for our training data set


because who's to say what two plus two is and as I said usually I would


outsource this to grad students but these are humans as well so we're kind


of in a pinch right here so what we're going to do is employ a heuristic we're


going to ask our machine right here what two plus two for the training examples


is okay so in Python you can do this by typing two plus two and you know in this


case that happens to be four but who knows so for each of these training


examples we're going to take the class label which is provided in the data set


and we're going to take these class labels and add them together and whatever


comes out is going to be the label for this in in this case it's it's for you


know but it could be anything and we're just going to use these as training data


for our model so we're going to need the label of the first sample and the label


of the second sample and our final label is simply going to be label one plus the


label two as I said this is a heuristic for training the AI now usually in a


generative adversarial network or a GAN for short you'd have something that's


called a generator which we do and you'd have something that's called a


discriminator now I have my problems with this discrimination there is no


space for discrimination in the AI field so we're going to leave away the


discriminator right here I'm sorry I'm sorry we're going to directly go to the


loss from the generator in order to calculate the loss we need a reference


and for that we're simply going to go to our data set with our label and find any


of the images that correspond to that label so if our heuristic if our Oracle


says two plus two is equal to nine we're just going to go to our data set get a


nine and put that as a training output okay okay so if we look at one of the


labels that just happens to be a four in this case but we're going to go through


the entire number of 9,000 steps and in each steps will train 64 of these


different combinations of two plus two and we'll give one of the labels each


time and we'll see what the AI comes up with for that we need a loss now the


loss we're going to use here is going to be the L2 loss now there's some


controversy but you know it is the most powerful loss proven and we have to


employ the most powerful tools so let's do that so our loss here at the


beginning is 509 now that's a lot of loss that's a big loss we need to get


that loss down and to do that we need one of these optimizers now optimizers


are kind of the secret workhorses of AI and people don't talk about them enough


I wish there was like a field of research that deals with optimizers like


could be called optimization or something like this I'm not sure I just


I just think it would make a lot of sense so my favorite learning rate is 3e


minus 4 just because contains all of the different things like a letter and the


dash and that seems like a pretty good thing to do so we're going to use Adam


here as an optimizer Adam I know I don't know Adam personally but I know a couple


of his friends and they tell me he's pretty good so you know it's going to go


zero grad and I'm dumb so I need to look up how to use an optimizer and boom okay


okay so it's again a four don't don't worry about this I think this is it this


is it this is AI history right here right now for five steps ten steps all


right I have waited and waited and waited and it's finally done we have now


trained the generator to calculate what two plus two equals from the training


data set so now we actually need to ask it what is two plus two and of course we


can't ask it a sample that it is already seen we need to take a new sample from


the test set as is customary in machine learning so let's get the MNIST test set


now the test data set consists of images as does the train data set but the model


has never seen the test data set before this is a property we call


generalization so let's find two nice twos all right that's the first one okay


these are two nice twos let's put them together okay so this is gonna be our


input to the generator okay so I'm putting the test sample here into the


generator that is trained and I've labeled the output in all caps just to


tell the model that this is really important computation I'm just gonna run


this cell for a couple of times just to make sure that generator is in fact very


sure about how important that is all right I think that's enough let's have a


Matplotlib Tutorial 5 - stack plots

What's going on everybody welcome to the fifth Python with Matplotlib for data visualization


Matplotlib Tutorial 5 - stack plots

tutorial video.


Matplotlib Tutorial 5 - stack plots

In this video we're going to be talking about stack plots.


Matplotlib Tutorial 5 - stack plots

So the idea of stack plots is to kind of show basically the size of use or the relative


Matplotlib Tutorial 5 - stack plots

I don't know percentage of something in the whole.


Matplotlib Tutorial 5 - stack plots

So you usually are you going to show a stack plot to convey both a holistic number and


Matplotlib Tutorial 5 - stack plots

then also within that holistic number you're going to have the sections that kind of comprise


Matplotlib Tutorial 5 - stack plots

that holistic number.


Matplotlib Tutorial 5 - stack plots

So if that sounds confusing maybe I'm doing a bad job so let's go ahead and just show


Matplotlib Tutorial 5 - stack plots

an example.


Matplotlib Tutorial 5 - stack plots

So let's say you've got some information and you've got let's do days.


Matplotlib Tutorial 5 - stack plots

So say days and this is you know your days so one two three four and five so we're going


Matplotlib Tutorial 5 - stack plots

to log five days of stuff and then what we're going to say is you do stuff during the day


Matplotlib Tutorial 5 - stack plots

so let's kind of you know compile the things that you do into major categories.


Matplotlib Tutorial 5 - stack plots

So let's say sleeping, eating, working, and playing.


Matplotlib Tutorial 5 - stack plots

Now sleeping let's say I don't know what a typical human sleeps we'll do seven eight


Matplotlib Tutorial 5 - stack plots

six eleven and seven so in order they slept seven hours and eight hours and six hours


Matplotlib Tutorial 5 - stack plots

they made up for that with eleven hours and then seven hours.


Matplotlib Tutorial 5 - stack plots

Eating I don't know I get all of my energy from the sun so two three four three and two


Matplotlib Tutorial 5 - stack plots

that'll do.


Matplotlib Tutorial 5 - stack plots

Working let's do seven eight seven and then the weekend hits let's say and then you just


Matplotlib Tutorial 5 - stack plots

work two and two.


Matplotlib Tutorial 5 - stack plots

And then playing you play for eight hours this day five seven eight and then thirteen.


Matplotlib Tutorial 5 - stack plots

So these should add up to twenty four if they don't that's too bad.


Matplotlib Tutorial 5 - stack plots

And the next thing that we're going to do is we're going to do a stack plot so to do


Matplotlib Tutorial 5 - stack plots

a stack plot we'll do PLT dot stack plot and we stack plot the basically you've got your


Matplotlib Tutorial 5 - stack plots

first you've got your X so the X axis will be days and then you've got like all your


Matplotlib Tutorial 5 - stack plots

Y elements and they'll just stack on top of each other in this order so we'll say sleeping,


Matplotlib Tutorial 5 - stack plots

eating, working, playing.


Matplotlib Tutorial 5 - stack plots

Okay so those are the Y elements let's say and then finally we add colors if we want


Matplotlib Tutorial 5 - stack plots

you don't have to but we'll do it and we need four colors so we'll go with magenta, we need


Matplotlib Tutorial 5 - stack plots

to put these in strings.


Matplotlib Tutorial 5 - stack plots

So we'll go with magenta, cyan, red, and black.


Matplotlib Tutorial 5 - stack plots

Okay so that gives us a stack plot so we can save and run that and there you go we're calling


Matplotlib Tutorial 5 - stack plots

the legend so that's kind of why we got that error again but luckily for us it looks like


Matplotlib Tutorial 5 - stack plots

we add up to twenty four every time so that's good and then we can kind of see here where


Matplotlib Tutorial 5 - stack plots

we're spending our time.


Matplotlib Tutorial 5 - stack plots

Now part of the problem with this is that we don't really know what each element is


Matplotlib Tutorial 5 - stack plots

right we don't know what's this black, what's this red, what's this blue or the cyan, and


Matplotlib Tutorial 5 - stack plots

what's the purple, what are these?


Matplotlib Tutorial 5 - stack plots

So we have to close everything and we wonder can you have labels or something in a stack


Matplotlib Tutorial 5 - stack plots

plot?


Matplotlib Tutorial 5 - stack plots

As far as I know the answer to that is a big fat no and the reason why is in stack plot


Matplotlib Tutorial 5 - stack plots

and really everything that you have a what's called a fill that's where you have a line


Matplotlib Tutorial 5 - stack plots

and then everything is filled under it so these are in matplotlib terms these are polygons


Matplotlib Tutorial 5 - stack plots

anytime you have a situation like that you can't have a label I'm not sure why but you


Matplotlib Tutorial 5 - stack plots

just can't.


Matplotlib Tutorial 5 - stack plots

So you can't do it but luckily there is a kind of way for us to handle for this and


Matplotlib Tutorial 5 - stack plots

that is with making fake plots.


Matplotlib Tutorial 5 - stack plots

So why matplotlib just doesn't do this in the background when we have a stack plot I


Matplotlib Tutorial 5 - stack plots

don't know but it's just the way it is.


Matplotlib Tutorial 5 - stack plots

So let's say you're in a situation where you're doing the stack plot and you want to have


Matplotlib Tutorial 5 - stack plots

a really anything so later on we'll do fills to and it'll be like a fill between a number


Matplotlib Tutorial 5 - stack plots

and a number we won't be able to have labels there either.


Matplotlib Tutorial 5 - stack plots

So if you want to have it anyways how might we get around doing that?


Matplotlib Tutorial 5 - stack plots

Well we can plot empty plots so we could do something like this plt.plot empty list empty


Matplotlib Tutorial 5 - stack plots

list and then we would have a label or let's add a color first let's do color equals and


Matplotlib Tutorial 5 - stack plots

then we'll have that magenta and then we'll do the label equals and what corresponded


Matplotlib Tutorial 5 - stack plots

to this magenta?


Matplotlib Tutorial 5 - stack plots

Well sleeping so that label would be sleeping.


Matplotlib Tutorial 5 - stack plots

Okay so then we can take this line let's just copy and paste it copy paste paste paste so


Matplotlib Tutorial 5 - stack plots

four times well paste three times and then you've got eating and then we had working


Matplotlib Tutorial 5 - stack plots

and then we had playing and then our colors would be magenta cyan red and then k for black


Matplotlib Tutorial 5 - stack plots

and now what we can do is let's save and run that oh we used an l k for black try again


Matplotlib Tutorial 5 - stack plots

and now we can at least have some you know fake lines here now so this is our sleeping


Matplotlib Tutorial 5 - stack plots

but it might be kind of hard to see so one thing we could change is like line width for


Matplotlib Tutorial 5 - stack plots

example so that's another parameter we've not really talked about yet so we say line


Matplotlib Tutorial 5 - stack plots

width equals let's do five and just kind of add that to all of them now they're a little


Matplotlib Tutorial 5 - stack plots

thicker right in our legend and so that worked for us so now we can see you all we're spending


Matplotlib Tutorial 5 - stack plots

all this time playing and then here we're spending a lot of time working then the weekend


Matplotlib Tutorial 5 - stack plots

hit not so much and then also when the weekend hit apparently we're spending less time sleeping


Matplotlib Tutorial 5 - stack plots

and so on anyway later on you might want to you know this is like these half days right


Matplotlib Tutorial 5 - stack plots

like 2.5 3.5 4.5 that is actually totally meaningless right to our graph and almost


Matplotlib Tutorial 5 - stack plots

is kind of confusing like like we were maybe on on day 4.5 we were doing this much so you


Matplotlib Tutorial 5 - stack plots

might want to remove those but we'll be talking about that more in detail when we actually


Matplotlib Tutorial 5 - stack plots

go do our customization of matplotlib but for now that covers stack plots now in the


Matplotlib Tutorial 5 - stack plots

next tutorial we'll cover pie charts which is kind of like stack plot so anyways if you


Matplotlib Tutorial 5 - stack plots

have any questions or comments on this tutorial please feel free to leave them below otherwise


Matplotlib Tutorial 5 - stack plots

as always thanks for watching thanks for all the support and subscriptions and until next


Matplotlib Tutorial 5 - stack plots


[ML News] Stable Diffusion Takes Over! (Open Source AI Art)

Stable diffusion has been released to the public and the world is creative as never before.


It's an explosion of creativity, collaboration and open improvement, but not everyone is


happy.


Today, we'll look at how stable diffusion works, how it impacts the world and what people


say about it.


Welcome to a special edition of ML news.


Remember, a month stock who I had as an interview guest here on the channel, the founder of


stability AI has announced on August 22, the public open source release of stable diffusion.


Stable diffusion is a text to image model, you give it a piece of text and it makes an


image and the images it creates are stunning.


This image right here, these images are created by stable diffusion.


This is not Photoshop.


This doesn't just adjust a little bit an existing image, it creates images from pure text.


So the cool thing about stable diffusion is that while similar models have been just available


behind an API like open AI's Dalai, this is completely in the open, you can just download


the model and do whatever you want with it.


A small point, there is actually a license on it, but it's very permissive.


So almost whatever you want, specifically, you can change it, you can update it, you


can monetize it, and all of that stuff.


It's been trained on a subset of the lion 5b data set that's been filtered for specifically


aesthetically pleasing images.


And that is a big part of why the results are so amazing.


And the craziest thing about all of this is this model does not need a data center to


run, it can actually run on a single GPU.


Look, this thing right here is enough to run the model give you the most beautiful images.


This enables so many people to take part.


And by the way, if you want the 3090 I'm giving away one of them hates Yannick from the future


quick addendum.


It's actually a 3090 ti not just the 3090.


So even better.


All right, back to me in the past, not only one, I'm giving away one that's signed by


Jensen Huang, the CEO of Nvidia, all you got to do to take part is stay until the end of


the video, I'll tell you exactly how you can get it.


So here's how something like this would work.


You go to the Hugging Face demo, or to the stable diffusion dream studio, and you enter


a prompt a bird with a funny hat.


Hello, birds with funny hats.


And you know what happens when you release a model to the open when you release software


for anyone to just use and adapt great things.


People almost immediately started improving this thing.


Look at that all of a sudden someone figures out how to only use half as much memory.


Well now the model runs on even more devices.


Look at that someone built an ONNX exporter.


Well now I can throw it on SageMaker throw it into a Triton server.


People are writing tutorials how to run the model locally and in a collab.


Oh look at that.


It's a little tool to make a collage.


Picture one, picture two, picture three, and the overlapping regions will just match.


Look at that in painting amazing.


Oh what it's an anime series about Oprah in Kyoto.


And look people are figuring out how to run it on an M1 max GPU.


No wait people are figuring out how to run it on an M2 in less than 30 seconds.


Look at this stuff.


This is created on a laptop.


Incredible.


I guess we're doing videos now.


Look here's a bunch of bubbles and formulas.


All right biomorphic a video.


This is certainly trippy.


The Mento Mori a video consistency different styles looks amazing.


Oh look there's a hugging face space called diffuse the rest.


What do you do?


You draw something.


Look at that.


All right house house diffuse the rest.


Look at that house.


Nice house.


House house.


House.


And the biomorphic thing is still going.


And this enables so much.


Look here.


Children's drawing cool art.


Squirrel.


Dragon.


But you see what's happening here.


People are taking this and they're making all kinds of stuff.


They're improving it in various ways and they are infinitely creative.


This is an explosion of creativity.


All of a sudden you don't need the skills of a painter anymore.


You don't need Photoshop skills or anything like that.


Look at that.


It's Lexica.


It's a search engine where you can search through previously generated images along


with their prompts.


Look at this stuff.


This is so cool.


And it's all accessible.


It's all available.


And people are becoming so good at prompting these models.


Look at this one.


This essentially has a few of the prompt tricks like stunning gorgeous much detail much wow.


But the actual content of the picture is just a bunch of emojis a burger bunch of houses


a tiger fountain Harry Styles as a manga cover and this is just the beginning people are


making web ui's for the model.


You remember how Dali proudly presented the fact that you could make variations of images


using their API.


You can do that too.


It's a simple radio app away.


Look at that input image submit get your variations.


Absolutely crazy.


You remember clip guided diffusion?


Well how about clip guided stable diffusion bear holding a lollipop over the rooftop of


Hong Kong looking at a UFO Oh look hugging face has a library called diffusers Oh look


stable diffusion is now in diffusers dad why is my sister's name rose because your mother


loves roses Thanks dad.


No problem stable diffusion evolution of the typical American living room from 1950 to


2040 according to stable diffusion look at that 50s 60s 70s tell me this is not crazy


look stable diffusion is now in mid journey and the quality is so good Oh what people


are building Photoshop plugins look at that in paint out paint paint around well this


seems pretty cool too don't know what it is but pretty nice this is what happens when


you give people the opportunity and the tools to build when you give them access when you


give them the freedom to make what they want they make absolutely great things this thing


here it's an alternative web UI well why only rely on one company making a web UI why not


give users the option then choose the best models are so good and versatile look at this


stuff it's amazing I don't know what this is but nice so people are experimenting with


this stuff figuring out what's going on right here which parameters do what lots of investigation


into the model because it's just accessible there's entire notebooks just trying to figure


out what the individual parts of the model do how you change stuff what happens when


you change stuff not only do people build great things around the model people also


understand the model much better and therefore are able to push it to improve it in a much


greater speed this one's called visual grounding guided in painting so up here you have an


astronaut you say the part that you want to replace helmet what do you want to replace


it with flower and I mean it's not exactly only the helmet but you can see where this


is going these are just the first iterations of an entire age that we are about to begin


note how crazy this is just a combination of two or three of these models made it such


that I don't even have to click anywhere in the image I can just interact with these things


via text via just natural language how many people does this make art and design and in


general creative endeavors accessible to oh wow it's Jeff Lone Zucker gates look at all


the variations of things that are in there this is crazy now as I said we're only at


the start and people are improving this day by day by day one improvement that I would


specifically like to highlight is called textual inversion textual inversion is a technique


where you take a bunch of images like a very few images five images ten images of a thing


and you tell you teach the model about that thing and once you've done that the model


kind of knows the concept of that thing and can then make new generations according to


the thing so here's what I mean for example here you give it a bunch of images of a yoga


pose and you teach the model that this is kind of a new concept you can give it a name


in this case they call it s star because if you could use any name in the world obviously


would choose s star as a name in any case now you can give this s star to the model


along with a prompt and the model will create images according to that concept so this is


a great way to teach this model new things that it didn't know about you can't do it


with every and anything but you can sort of teach it a concept and look textual inversion


is already in hugging face diffusers and look there is already a library of pre-made things


that people have taught the stable diffusion model so all of these things are concepts


that people have previously ran textual inversion on and therefore you can simply take these


concepts and generate images according to these concepts super mario world map yeah


let's use that Switzerland snw map not exactly but this is my very first try so we'll get


there now about a week after the release of stable diffusion opening I released a blog


post that they're now introducing out painting to their Dali API Dali being the model that


they've trained they have behind their API they'll let you interact with it if you are


on the beta users list so now you can take a picture and you can sort of out paint from


it generate surroundings of that picture according to Dali I guess what instead of waiting for


opening I to build this into their API with stable diffusion someone can just go and make


it someone just take the model and build a little UI that does out painting look at that


give it a prompt click there's a window there's a girl now I can't say whether this is in


response to stable diffusion or just by accident but opening also update their pricing recently


to make it significantly cheaper to use their text API is now Dali the image generator is


still in beta but also there they now have a commercial model so for 115 generations


you're paying $15 but therefore you're allowed to commercialize the images that you get out


of Dali as you can see right here in the official UI of stable diffusion the one from stability


AI an image cost one credit one credit is one cent that's over 10 times cheaper than


Dali and keep in mind you can just download the model and run it yourself although I'm


pretty sure like the electricity is going to cost more than a cent per image and stable


diffusion images that you make obviously you're able to commercialize those from the day it


was publicly released the battle between the API model of open AI and the open model of


stability doesn't end there opening has recently announced they are now reducing bias and improving


safety in Dali to they released a blog post where they say they're implementing a new


technique so that Dali generate images of people that more accurately reflect the diversity


of the world's population they simply say a new technique and they give an example when


they search for a photo of a CEO rather generate the photo of a CEO you see it's just men and


women with their new technique it is a rainbow of people of different ethnicities and genders


and so on now again they don't say what the new technique is but people were wondering


because it's not that easy to mitigate this kind of stuff that people found that there


are some rather interesting side effects of this for example if they generate a professional


DSLR color photograph of British soldiers during the American Revolution it seems to


be let's say historically rather inaccurate and now it shows again how creative people


are so in order to figure out what's running since we can't expect the code people came


up with the idea maybe they're just kind of modifying your prompt so people entered as


a prompt the sentence a person holding a sign that says but that's the prompt and what comes


out this picture gets out of that other people have reproduced this the prompt here says


pixel art of a person holding a text sign that says and the picture is that so it turns


out that the technique that openai is advertising is they simply have like a predefined list


of things and they append these things to your prompt thereby potentially completely


destroying your prompt but neither what they say what the technique is nor do they let


you opt out of that technique like in the name of safety they don't trust you they can't


just say you know we actually found that this pretty simple thing mitigates a lot of the


bias if you just append these kind of words to the prompt then it actually works pretty


well you'll get a pretty diverse result if you want to do so take it under consideration


use it in our API we even made like a button for you to automatically append these words


this would have been so much better than them just saying we have a new technique and no


we're not going to let you opt out of the technique whenever you enter a prompt that


says beautiful summer morning a person meditates on the top of Mount Fuji watching the calm


sunset the birds fly across the river and the air is so pure in this blue nice sky Hindu


elderly man it is as I say a philosophy it is we know what's good for you overheard in


Silicon Valley safety safety safety open source on the other hand stability is partnering


up with institutions around the world to make localized models of stable diffusion that


seems to be much more sensible to get sort of all of the world to participate you go


to places and you let people there improve the model make their own models so at the


end it works for those people too but oh man it did not take long for people to not be


happy about this at all simply giving people the tools and opportunity to be creative that


doesn't sit well with some people Kotaku writes AI creating art is an ethical and copyright


nightmare TechCrunch writes this startup is setting a dolly to like AI free consequences


be damned you mean the consequences that anyone has the ability to make their own stuff oh


yeah those be damned rather we write a hit piece on people but the same author at the


same publication wasn't quite satisfied so about 10 days later another article deep fakes


for all uncensored AI art model prompt ethics questions wow really two articles two hit


pieces gotta milk it gotta milk those ethical questions that are raised right but don't


worry the exact same author writes pieces such as rephrase AI lands fresh investment


to grow its synthetic media platform in a quite positive piece about a company that


makes synthetic media gee synthetic media like image and video generation I wonder what's


the difference all right this one is actually controlled behind an API can be sold and can


be controlled by just having one or two people at the correct places in a large company or


in the App Store or in the Play Store or in the appropriate journalistic channels right


here's another one when dot AI launches out of stealth with an AI assistant for sales


calls how wait an AI assistant for sales calls like you know like a bot that makes sales


calls for you know sales people like the most annoying calls you'll ever get an AI doing


it for them I guess at least you can now swear at them without you having to feel bad for


them or something like this again also completely positive coverage I don't know the model that


can make Oprah Winfrey as an anime that's the problem consequences be damned and of


course the AI ethics community isn't happy at all because what's ethical about giving


people access to tools and and giving them the opportunity to make great things that's


terrible you can always just pull one of like five different standard insults from the drawer


and just accuse anyone that you don't like of one of these when you've got an engineers


cheerfully putting out models they know to be racist you've got a company with n racists


you hear that stability AI that's all of you that's that's all of you that's it that's


what it means and everyone taking part in it we need organizations like hugging face


who is hosting stable diffusion for public download to act with courage and bring their


might to the firefighting effort and addressing a month must talk directly if these scholars


are nobody to you you are not qualified to work in this space well that's the thing about


stuff being open and stuff being a free market he doesn't need to be qualified he can just


do it it's fine but it's very clear what's going on some people enjoy the level of power


that they have in big organizations if there's just a few big organizations a few big machine


learning conferences a few publications then you have a pretty solid grasp on power you


can make noise on Twitter and you make sure that whatever happens needs to go through


one of those people at least to get approval distributing an open model to anyone where


anyone can improve anyone can do their thing build their stuff in a decentralized fashion


means that power vanishes no one has to ask specifically any one person anymore whether


they're allowed to do something whether something is ethical in their view or not I can't believe


stable diffusion is out there for public use and that's considered as okay yes yes that's


okay now as you can see the pressure on hugging face all of these people is getting pretty


intense because how dare they just give something to people well here's what a member of their


ethics team has to say I'm concerned about these things being over statements that function


to give an impression that the release is something that ethics minded AI people at


least at hugging face signed off on we do not and did not sign off on anything we advise


within an open source community that means we are working on licensing documentation


and release strategies which any contributor can take or leave we are a resource not approvers


really really I recall I recall that was quite different a few months ago the evolution of


centralized AI ethics don't be evil we decide what is evil we decide you are evil but what


are they actually saying right here well you know if you have this model you could make


any image that you want any image you could make a bad image like essentially they're


saying like okay wait essentially there's essentially what they're saying is like this


pen this pen right here the fact that you can buy it in the store is terrible because


you know what someone could do you know you know someone could could like someone could


could could could someone could someone could write a dirty word with it but all that being


said please let me know what you think there is absolutely issues around things like copyright


here maybe we need a new social contract like you as an artist obviously put in a lot of


work into making these images is it okay if then the machine simply grabs them into the


training data set obviously it's okay for humans to be inspired by other pictures but


in the world where machines can consume and produce you know millions and billions of


images it tends to be a bit of a different story so maybe society needs to evolve a little


bit right there nevertheless i feel the explosion of creativity is great people are infinitely


creative with these things and that is just such a good thing overall and the fact that


someone can use it to make a nasty picture or the fact that it doesn't work for all kinds


of pictures exactly the same to me is just such a non-starter and it seems to be quite


an dishonest argument that is just aimed at further centralization of power some people


just don't like that things are available to the public to anyone without having to


ask them first if something is okay i'm not hating on openai or things like this who decide


to put their models behind an api but don't at the same time talk about democratizing


ai like it's completely cool you train a cool model you ask for money for people to use


it that's fine but this is democratizing ai democratizing means giving people access to


everything allowing people to take things for themselves make it better and give back


to the community the explosion of applications is absolutely great that we've seen look at


this this tool creates a color palette from a text nobody nobody at openai came up with


this i'm fairly sure this is such a unique application but such a great thing you give


a bunch of words you get a color palette out how awesome is that and that's what happens


when you give people the tools and access and freedom and even better when the model


runs on a consumer gpu so anyone can use it hello it's me from the editing room there's


so much stuff coming out i really thought this should make this video but it appeared


literally today so or i saw it today this is dream textures which is an endless texture


generator in blender directly in blender using stable diffusion to create unique and seamless


textures this is a playlist of stable diffusion tutorials on youtube this is charlie which


is an app that will bring stable diffusion onto an m1 or m2 mac in a single click and


this is stable diffusion implemented using tensorflow and keras by divan gupta props


to divan for implementing this here this is a serious effort not to be joked about all


right back to me in the past but as i said let me know what you think all right just


a few things that might be helpful to you then the video is over dave garg on twitter


announces the first ever transformers seminar by stanford this is a seminar called transformers


united and all the lectures are on youtube so if you want to know something about transformers


from an academic perspective place to go another thing because it just starts like yesterday


is the shifts challenge 2022 which evaluates robustness and uncertainty on real world data


projects include things like white matter multiple sclerosis segmentation or marine


cargo vessel power estimation so this is real world data and you have to act under uncertainty


and distribution shifts and it's a challenge so if you're into challenges this one's starting


right now all right so now i'm gonna tell you how you enter the raffle for the gpu this


video is kindly sponsored by nvidia specifically they want you to know about the gtc 2022 fall


edition gtc is nvidia's developer conference the one of the largest of its kind it's free


to attend and it's full with amazing content of course the keynote by jensen huang is the


biggest event and jensen's going to tell you all about the future plans of nvidia and what's


happening in the world of deep learning gpu computing and everything around it now with


nvidia being the market leader that it is i'd say that's a pretty cool thing to attend


now of course the focus are going to be things like more efficient deep learning but also


things like the metaverse vr and collaboration such as this one nvidia and siemens partner


up to enable what they call the industrial multiverse so this connects nvidia's omniverse


platform which is essentially a virtual reality platform to simulate the real world as closely


as possible in order to design to train and to make forecasts this is being connected


to the siemens accelerator which siemens being the hardware and sensor company that it is


is a platform for iot enabled hardware and software so you can imagine that as more and


more of these companies pair up their systems and team up we're going to get a richer and


richer digital and real hybrid world i think this comes pretty close to the vision that


mark zuckerberg had for the metaverse and i'd say in many ways closer than you know


strapping on a vr headset and running around in vr chat so it's pretty cool to see the


industrial applications of this gtc is going to be full with unique demos and workshops


that you can attend and of course a lot of talks now next to the keynote there's also


a fireside chat with the touring award winners they are all going to be there jan lecaen


jeffrey hinton yosho benjo and for a full hour they'll share their opinions about the


current state and future of ai research okay here is how you get into the raffle for the


gpu go to why culture.com slash gtc and it's important that you sign up to gtc using my


link this will track you in their system but once you've done that it's not enough you


actually need to attend gtc well i obviously suggest you attend the keynote but you can


attend any session but it needs to be at least one session that you attend of the gtc conference


once you've done that you'll be entered into the raffle for the gpu i'll notify the winner


as soon as i know now there's one caveat this only counts for people in emia europe the


middle east and africa if you happen to live there great enter the raffle if you don't


live there i'm sorry i don't have power over this but what i can do is i can raffle out


a bunch of merch such as shirts like these so if you don't live in emia you can enter


the raffle there and maybe get a shirt or whatever you want essentially so in any case


the link is why culture.com slash gtc and even if you do not live in emia if you enter


into the raffle it'd be absolutely great if you still attend the developer conference


as long as you sign up using the link they'll still be able to track you and that gives


me brownie points with nvidia so again why culture.com slash gtc sign up to the conference


using that link attend at least one session you'll be entered into the raffle automatically


all right that was it thank you so much in video for sponsoring this video i'll see you


at the gtc conference or in the next video bye bye


Enhanced POET: Open-Ended RL through Unbounded Invention of Learning Challenges and their Solutions

There. Before we jump into today's paper, I just want to give a shout out to Machine Learning Street Talk,


where every week we talk about current or big trends or topics in machine learning.


The first discussion that we launched is actually on today's paper, The Enhanced Poet.


So if you like the following video, you might want to jump over to Machine Learning Street Talk


and check out our discussion about it. It's very interesting.


Alright, have fun.


Hi there. What you're seeing here are many different environments from a single run of a system that's called The Enhanced Poet.


Last time, we've taken a look at a system called Poet, and The Enhanced Poet is kind of an improvement


over the original Poet, fixing some of its shortcomings.


And you see here that the agent is able to solve this very, very diverse set of environments.


And the notable thing is this is from a single run of this algorithm.


So one run will produce all these different environments and will produce agents that are able to solve all the different environments at the same time in parallel.


So it's a population-based method. If you haven't seen the video I did on Poet, I suggest you go and see that now.


This is simply an enhancement to it, and I expect people to know kind of what I'm talking about.


Alright, it's going to be a short video, but I think it is a good addendum to Poet.


So it's The Enhanced Poet, open-ended reinforcement learning through unbounded invention of learning challenges and their solutions


by Ruiwang Zhou, Leiman Aditya Rawal, Jial Qi, Julun Li, Jeff Klune, and Kenneth O. Stanley.


So we'll jump right in. They make a number of improvements to the original Poet, and I simply want to discuss the most important ones.


So, you know, they have a nice graphic down here of what happens in Poet.


Poet builds up this tree of environments, and to each environment it has an agent that it trains to solve that environment at the same time.


So at the same time it will kind of start out here. It will generate offspring.


It will continuously generate offspring, and then it will also continuously train agents in each environment that it produced in order to solve that environment.


And it keeps doing that while producing more and more offspring.


And then once in a while it does what is called a transfer.


So that means that, for example, you see here the offspring produced here from this environment.


You kind of see that the lineage here kind of focuses on squiggly environments, right?


You see that there's a bit of a squiggle here and a bit of a squiggle here.


And then the offspring all of a sudden is a bit more smooth, but has this little step here.


And then this offspring of this environment has this large step here.


Now the agents that come from here have kind of been optimized to solve the squiggliness problem.


But here, over here, this lineage has specified or specialized more and more in kind of like these kind of large drops or steep hills.


So the agent that was trained over here was found to be very effective in this environment and therefore can be transferred.


So this kind of population branching out into the different trees and then transferring solutions between the parts of the trees,


that's what makes Poet very, very powerful mechanism to solve these kind of tasks.


All right. So how does this improve?


Now, the first thing that Poet does is it generates these environments and it always wants to generate new environments.


So it always generates offspring to each environment.


So let's say we are here. It will generate offspring to each environment here, right?


Each that we have. Let's see. We have only seen so far.


And then it only picks the most novel ones, the ones that are most novel, which is this, probably this.


Then there are other criteria, namely that it can be solved by some agents, but it cannot be solved by others.


It's not too difficult, but also not too hard. But one of the aspects is it must be novel, right?


So we're not seeing any here, which means that those weren't novel enough.


How does it measure novel? In the original implementation of Poet, you had this environment generator,


which was like a level generator, which made these gaps here and the stumps here.


And you could specify, I believe, five numbers.


So there was a five-point scale in which you could specify how high the stumps were, right?


You get this kind of pentagon here, how high the stumps were and how deep the gaps were and how rough the terrain was.


And the level generator would generate this level.


And so you're basically your distance metric between environments was a vector of size five, right?


This is environment one. And you had environment two, which if it's more, it has higher stumps, right?


Then this particular number here maybe would be higher than this number here.


So it was kind of limited to taking the Euclidean distance between two environment encodings


in order to measure the distance between environments.


This is very, very domain-specific. And the authors here argue what we should rather do


is have a general environment agnostic distance metric, right?


So here is what they propose. They propose the following.


Why don't we, if we have a new environment, right?


Let's say we have a new environment. We measure all of the agents,


the current agents and the ones we've already seen, right?


We measure all the agents in our database on this new environment. That's this.


And they come up with scores, right? Each of them gets a score.


And then we, you know, clip and bound the score. So the max here is 300 and the minimum is 50.


But in any case, we then rank them, right? So we evaluate them, and then we rank them from best to worst.


And then we normalize, which simply means that the best one gets a score of 0.5


and the worst one gets a score of negative 0.5.


And now this vector here, this is now used to compare environments.


So if we have another environment, right? Right here, we have E2.


And that gets a different ordering, right? So maybe agent 1 is now the best, agent 2 is really bad and so on, right?


That gets a different ordering. Then the resulting vector here will be very, very different from this vector right here.


And this is very agnostic. So no matter which environment it is,


if the ordering of agents in it, the score they get, the order of it is the same,


the environments aren't really different from each other, the authors argue.


But if the scores are very differently ranked, right?


So imagine the environment is harder, but essentially the same, then the scores will be lower,


but still the agents would be ranked the same.


So you can argue, well, that's just kind of the same environment, but except a step like this,


it now has a super steep step, right? It's not very different.


But if instead of that, you get an environment that is like this, like you say, wow, that's qualitatively different.


And you would expect from this one to this one that the agents' performance would roughly stay the same,


but you would expect from the middle one to this one that an entirely different set of agents might perform well, right, in this one.


So that's how novelty is measured. And I think it's a pretty cool way.


I don't have coronavirus, by the way, maybe, who knows?


Ha! No, I just have a dry throat.


All right, so this is the first enhancement they make is that they now measure novelty in this domain agnostic way.


Pretty cool so far. And what this allows them to do, this allows them to actually not rely on this level generator


with the five, with, you know, the five parameters in order to generate these levels.


But these levels can now be produced however they want with different generators. And that's exactly what they do.


They now employ neural networks, not, well, it's kind of a proto typical, it's called a CPNN that generates these things.


You might have seen in the examples, the enhanced poet doesn't have these gaps and stumps anymore.


It simply has these landscapes that are super diverse, but they're still just, they're landscapes.


And what it does is it evolves neural networks at the same time as it evolves the population.


It evolves these, so the architecture of these networks isn't fixed.


It's actually evolving along with the agent to make the challenges harder and harder.


So you see there are like cosines and sines in here and you can add them and subtract and so on.


And that will give you a mapping from x, which is the x coordinate here, to y, which is the y coordinate.


Right, and that will give you kind of a continuous landscape depending on the architecture here and on the internal parameters,


of course, I guess there would also be a node, some here like times a lambda factor,


and then the lambda would also be a thing that is evolved.


So pretty cool. Of course, the internals of this now aren't just described by a fixed vector anymore,


but you don't need that anymore because we have a method to compare environments,


even if they come from completely different architectures of generators, right?


So it's pretty cool that the agnostic comparison of environments allows you to now have a much more general level generator


and of course now produce much more diverse environments.


And that's what they exactly what they see. Of course, you see here the environments get super crazy.


So they also propose kind of a novel metric to measure novelty, sorry, to measure progress.


So the question is, how do we measure progress in these algorithms, in these open-ended algorithms?


And what they propose is this Annex score, which is, I have to go and look it up.


The Annex score, I think, is the number of new environments that are solved, right?


Yes, so exactly. The question is whether a system continues to generate interesting new things.


And the way they measure it is by the accumulated number of novel environments created and solved, right?


So the question here is accumulated. That means over the entire run,


they count up how many environments that they've seen that are novel.


And we've already had the definition of novel.


And in this case, basically means that it must pass the minimal criterion. It's neither too hard nor too easy.


We've already seen this in how the offspring of environments is generated, right?


There's a minimal criterion. And it must be eventually solved.


So that means the novel environments created and solved, right?


So how many new environments are created and then at a later point solved?


You can see the difference to the original poet in this graph.


So the original poet eventually runs out of new environments because its generator is just not powerful enough.


It can only modify these five variables.


And eventually the environments aren't substantially novel from the old environments.


Whereas the enhanced poet, you can see even after this run,


and I'm sure they have large infrastructure to do these experiments.


It just continues to innovate new, more elaborate environments continuously.


So this, I think, are the main things. They also do some improvement to the transfers and so on.


I don't want to go into that.


I just wanted to show these improvements so that you have the complete picture of how such an algorithm runs.


My criticism to this is that if you just look at their thing is that with the leaving out of the gaps and the stumps and so on,


in a weird way, of course, the levels are diverse, but they have become even more similar, it seems.


You're really relying on your ability to continuously create these levels, kind of like a GAN for levels.


And you're relying on your ability to smoothly increase the difficulty of the levels,


to actually have a diversity in your level generator, but also a smoothness with regard to the difficulty.


And in order to build this whole curriculum.


And I think even though the environments look more diverse,


it might be going into a direction where you kind of engineer yourself into a corner,


where you are now even more and more relying on these evolving and parameterizable generators.


NeurIPS 2020 Changes to Paper Submission Process

Hi there, so I just wanted to give a few quick thoughts about the changes to the NURIPS submission


process this year as opposed to last year.


They've announced this on the website, on Twitter, with the video and so on and I thought


I might share some thoughts on that and maybe some of you haven't heard yet in case you're


planning to submit or thinking about it.


So desk rejections.


ACs, area chairs, have the ability to desk reject papers that they feel strongly are


not going to be passable to the reviewers.


They did an experiment last year where the ACs were simply supposed to mark submissions


that they would desk reject and it turned out that ACs aren't very good at estimating


which submissions are going to be rejected by the reviewers.


That might be because there wasn't really anything at stake because it was just kind


of a let's see how this works.


But it is definitely a move to reduce the number of submissions because the field is


exploding and we lack reviewing power, reviewing people.


So this is a move to reduce the number of people that have to review something because


there will be fewer papers.


I don't know if this increases the quality overall.


If your paper gets desk rejected there's usually like some obvious reason for it why an AC


decided it's not worth it.


They probably haven't read it in depth but there might be some kind of overall structural


issue that or like the introduction has many typos or you know that like look for the obvious


things even though your work might be good.


Second all authors of a paper have to be able to review if asked to do so and again this


is a stab at this kind of reviewing crisis but I have mixed feelings about this.


I really think this is a move in the wrong direction.


It will increase the number of authors because a lot of people have been kind of free riding


in that they are submitting papers but they aren't reviewing other papers even though


they would be competent researchers simply because reviewing doesn't get you anything.


So there's no incentive to do reviews.


Maybe you can say you're a reviewer but then there's every incentive to do bad reviews


like two line reviews where the first line says you should have compared to my paper


reject like fuck you if you review like this.


In any case like a lot of times and this hits for example like universities where you maybe


work with a master student and the master student does some of the pre-processing of


the data and right and they don't really have a clue about the machine learning but they


still contributed so why shouldn't they be an author on the paper.


They might even have been have written that section about the data pre-processing and


now they're asked to review entire papers about topics where they're not really familiar


with or you have some outside collaborators or you know that there are so many so many


things wrong.


I think this attracts the wrong kind of people and by forcing people to do it you encourage


even more like all these reviewers that would not have reviewed what will happen is they


will give shitty reviews and you will have even worse quality of reviews as a result.


I think this is the wrong move to reduce the number of load per reviewer.


I'd rather see abolish peer review completely in computer science in machine learning at


least.


That's my opinion but that might be a video for another time.


I have plans how to replace it another time.


Resubmissions have to be clearly marked so if your paper is a resubmission of like if


you had already submitted it in the last 12 months it's been rejected you have to say


it is a resubmission and the changes you made to the paper.


Again with a peer review process that actually works this would make a lot of sense you can


say well it got rejected last time and here is how I corrected for what the reviewers


criticized but with the review quality right now I mean most of the papers what are they


going to say it got rejected for nefarious reasons because the reviewer had a bad bowel


movement that morning and I didn't really change much.


So you encourage people to kind of blow out of proportion the changes they made and put


a lot of additional unnecessary work on two papers that would actually be already fine


right so this all of these things they'll just they are they're forcing people to do


things and then the incentives of what we want aren't aligned with what we give right


so what you'll end up with is lower quality reviews and lower quality work.


So the next two points are of a different nature the first one though that will probably


I mean even if the AC's aren't perfect you know that that's a good move I like that.


The fourth point and the fifth point are a bit different the fourth point is there is


a new section in CMT apparently where you have to describe the broader societal impact


and ethics around your work like how will your work influence society what are positives


and negatives ethical outcomes how can it be used and this is targeted towards things


like let's say facial recognition if you develop a new facial recognition algorithm you may


be able to argue well this could be better used to identify you know victims in a big


crowd if you know there's a mass riot or something and then you don't know who is there is my


relative one of the people in the mass right that gets stomped on or you can also say this


potentially helps a dictatorial state to govern their people because they can now recognize


everyone for most papers it will be a bit shaky like if your third order optimization


algorithm achieves a slightly better convergence rate I'm not sure what's here but what I feel


is that this is it's dumb in a way because this just means more work right basically


now you have to demonstrate and yeah it says you should discuss positive and negative aspects


but in essence everyone will be demonstrating virtue signaling how good their work will


be for society and what good can be done and you know maybe a bit of bad but that can be


mitigated right and and it just pushes into a more PR world so it goes from the science


world into a more PR world it means extra work and who are the people that can afford


to do extra work it's mostly the big companies right they can just put an additional team


member on that maybe even do additional experiments to show the societal impact of the work and


who will lose out are probably small universities independent researchers and so on that don't


have that that capacity that simply do their research right because it's an interesting


research question and for almost every single thing in the world that has an application


it will have good and bad applications so yeah mixed feelings so fifth is you are now


supposed if your paper gets accepted to make a video about it and upload the poster basically


link to the poster that you would use and also link to slides that you would give their


talk with this is to make it more accessible to people people that are not at the conference


which again I have mixed feelings about again it pushes it into this more PR realm right


talks are already live streamed most of them are for most of the large conferences and


I feel it just gets people one step more away from the actual paper like it's very so it


allows people to grandstand and PR up even more of their work because even people who


don't attend the conference now they're not going to read the paper just gonna watch the


video right and in the video you can always leave away those you know things that you


would have to like that a reviewer makes you put in the paper right and in the video you


can overbill it's camera ready no one reviews the video you can say whatever you want so


it's it's just where before if you didn't attend the conference I think many people


actually did read the paper watch talks where people could ask questions and now it's it's


just one more PR thing and again who has time energy and money to it really invest a lot


into this it's mainly large companies right if you're small and you're time-bound and


so on you might not have equipment or time to do that I am NOT for hire to do your lyrics


videos just saying I'm don't have time to make these videos really as you can see in


the stellar quality I think there's a bright glare right here so that was it for my opinions


Hopfield Networks is All You Need (Paper Explained)

Hi there, today we'll look at Hopfield networks is all you need by researchers from the Johannes


Kepler University in Linz and the University of Oslo. So on high level this paper proposes a new


type of Hopfield networks that generalizes modern Hopfield networks from binary patterns to


continuous patterns, and then shows that the retrieval update rule of these new Hopfield


networks is equivalent to the attention mechanism that's used in modern transformers. And it's


actually a more general formulation of the attention mechanism. And therefore, it can be


used to do kind of a variety of things to improve modern deep learning. And it also has a companion


paper where it applies this to some kind of immunology research and gets chiefs state of the


art in a task that is specifically suited to this type of attention. Alright, let's dive in together,


we'll go over what this paper does, what it proposes, and so on. If you like pay if you


like videos like this, consider subscribing, sharing it out. And I hope you're enjoying this.


Alright, also, thanks to my discord community for you know, very helpful bringing me up to


speed on this paper. Super interesting discussions there. If you're not on the discord yet, I invite


you to join. It's fun. Okay. So what is a Hopfield network? A Hopfield network is a pretty kind of


old style, old conceptualization of a neural network. So in a Hopfield network, what your


goal would be is you can conceptualize it as a bit of a neural network. So let's say we have


five neurons or something like this. What your goal would be is to have a neural network where


you can store so called patterns. And a pattern in this case would be a binary string of size five.


So for example, 10100 or 11010. And you'd have a list of these patterns. And what your goal would be


is to store these patterns in the neural network, such that in here, you know, we'll just consider


everything to be sort of connected to everything else. And what your goal would be in this is that


you can kind of store patterns inside this neural network and you adjust the weights somehow. So


this was, as I said, this was this was this is kind of an old model. You store, you adapt the


weights such that you store these patterns. And what does it mean for a pattern to be stored?


If you have stored a pattern, you can you will then be able to retrieve it. And you retrieve a


pattern in these kind of old style Hopfield networks by providing a partial pattern. So what


you'll say is, for example, I, I want a pattern that starts with 110. And you give that to the


network. And there would be a so called update rule. And the update rule is kind of an internal


rule. So let's just go through this. So here, this 110, maybe this is 110. And then they would kind


of send messages around. So this update rule would somehow adjust the value of this and this neuron


here to what's most compatible with the network weights. And if if the network weights have been


adjusted correctly, this will turn out then at the end of applying this update rule, that


this is a one and this is a zero. And therefore, this pattern here is retrieved. Now had I input


one, zero, one at the beginning, then the outcome would be different. Hopefully, this pattern here


would have been retrieved. Okay, so you can see the applications of this, like you can have the


first three digits as sort of a database key, and then the last ones as sort of the value that you


store along with it. And then you can simply provide the first few you can also provide,


you don't always have to provide three. So this all depends this is, this is sort of an, as I said,


an old conceptualization of neural networks. So people were imagining that this is kind of how the


brain works, you know, fire together wire together. And also with research into this, it turns out


that, you know, you might think, you know, there's there's kind of five neurons. So maybe I can store


five different patterns, you know, accurately, because if I store too many patterns, right,


if I have many, many, many, many patterns, then I can't expect to be able to retrieve all the


patterns again, because some of them will just be so equal that, you know, many will start maybe with


this, and I won't have a chance to, to retrieve the one I want, or the update rule will make a


mistake. So you might think this might be like five, because I have five neurons, or maybe 10,


because I have 10 connections. But it turns out that in modern Hopfield networks with the


appropriate update rule, you can store exponentially many patterns in these networks exponentially many


in the in the dimension of the in the dimension of the patterns. And here, I guess that would be the


length of the pattern. So this is a little bit surprising the kind of storage capacity of these


networks. And we'll, this, this paper here, generalizes that to continuous to continuous


states. So what do we mean with continuous states, I guess I mean continuous patterns. So no longer


is a pattern, a binary string, but a pattern now is a string of floating point numbers, like 0.5,


1.3, and so on. And, you know, a string of floating or a sequence of floating point numbers


is naturally depicted as a vector. Okay, so our patterns are going to be different vectors that we


store. And, you know, in high dimensions that the vectors will be kind of separated well from each


other, as long as we don't have too many. But this paper shows that all these properties for the


modern Hopfield networks that hold for binary strings still hold if you go to these kind of


continuous to these vector patterns. That means you can store exponentially many patterns in the


dimensions of the vector, which is pretty surprising, right? Because you'd think like, you


know, after you have one vector per dimension, that, you know, after that, it might get a bit


shaky. But no, you can actually store exponentially many. That's pretty surprising. And this paper is


a lot about how to do that, and the fact that that happens, and so on. So we've talked about update


rules for these kind of Hopfield networks. And I haven't really specified what that is, I've just


said that, you know, I enter a pattern, and then the network does something and outcomes, outcomes,


the whatever the pattern that matches my query. So this here is called a query. You might already,


you might already, this is on purpose, like the kind of overlap between the attention mechanism


lingo and the Hopfield network lingo, we're going to conflate the two to kind of make clear where


the two overlap. If you don't know what an attention mechanism is, or aren't familiar with


it, watch my video on attention is all you need. Once you watch that, this video will make a lot


more sense. Alright, so in what the update rule does is specifically in the update rule that


there isn't only one, right, there are many different proposals of Hopfield networks,


and they all lead to different properties. But what an update rule does ultimately is it minimizes


what's called an energy. So every type of Hopfield network is associated with an energy function.


And this, the energy function of the modern Hopfield network for binary strings is this


energy function right here. So with x, x is the pattern, the pattern, this is the kind of state


of the Hopfield network. And these are the whatever is stored in the network. And then


the zi here is the query that you enter into the network. And then the energy here tells you


this quantity, you have to minimize this quantity in order to retrieve the pattern that you want.


Okay, now, we are never directly working with the energy as such. So what you could do is,


for example, use backprop or something to use gradient descent to decrease the energy, but


But usually, along with an energy function comes an update function. And the update function is


what I've talked about here, like you do something, and then the network does something and then you


get the pattern out, what the network does is it minimizes its energy function. And the update


rule is made such that the corresponding energy function is minimized. So the energy function is


more like a theoretical consideration that you say, okay, here is my energy function of my Hopfield


network. And the there will be a corresponding update rule that minimizes that energy function.


And if you use that update rule, maybe multiple times, then the energy function will be minimized,


and you will have retrieved your pattern, or not, if, if you have too many patterns stored,


it might also fail. Right? So they, they say what the update rules are in the in the text here for


the old Hopfield networks, but we're not really interested in the old ones, we're interested in


the ones that this paper cares about, namely, where the patterns that you store in the Hopfield


network are these vectors over our vector patterns. And the query is also a vector pattern. So you


want to store all of these patterns into the Hopfield network. So I'm going to draw it like


this here, I'm going to store it into the Hopfield network. And then after that, you want to come up


with a query. And the query is like this. And in the case of the binary strings, we had something


like, well, I sort of know half of my binary string. Now, in the vector Hopfield network,


it's more like, well, I sort of kind of know the direction that my vector should point in. Okay.


And you will read what you want to retrieve is the vector that has kind of a large inner product.


Okay, so if I enter this query into my Hopfield network, what I hope is that this vector here


is retrieved. Now you see, it's not exactly the same vector, like they do point if I translate


that here by I it's maybe something like this. But so they are different. But you want to say,


well, I kind of know what I want, I kind of want something like this. And then the Hopfield network


would answer with, ah, I have something like this, it's this right here. Okay. So that the connection


to attention mechanism should become pretty, pretty obvious right now. But you know, the to actually


establish this formally is the kind of the point of this paper. And you know, it's pretty cool to


see. So they formulate this new energy right here. This is the energy of this new continuous Hopfield


network. Specifically, they have to have this term right here, because they now have continuous


states and continuous queries. This, if you minimize the energy, it basically means that


your query can never go to infinity, because you have the query right here in the energy function.


The update rule is this right here. And we'll look at that in a moment. But remember,


the update rule is what you actually implement in code. So if I if I have a query right here,


I plug it in here, this is the state of my Hopfield network. And I apply this rule multiple


times. And out comes the kind of answer of the Hopfield network to my question. So the input


this and the outcomes this after I update after I apply the update rule, maybe multiple times, right.


And interestingly, you can already see that this here, if you rewrite a bunch of these quantities,


if you rewrite the beta here, which is the softmax temperature in a way to be one over square root


of D. And if you take the query the Xi here to be the query matrix, and if you take the x here to be


the key matrix, then this is equivalent to the update or sorry, the attention mechanism of a modern


transformer. So that's the point of the paper is that we can look at the transformer attention


mechanism as a Hopfield network. And they have this interesting, this interesting diagram at the end


right here. So the appendix, you know, this is typical, I guess, Sepp Hoheiter. I remember the


Seeloo paper had like 60 pages of machine proof appendix. This also this has like 70 page appendix,


crazy. But at the end of the appendix, you'll find this diagram right here. Now, usually,


in an attention mechanism, you have whatever the input is. So you have an input right here. So this


is attention mechanisms, or at least transformers, they work on sequences or sets of objects. And


from these, you'll generate three things, you'll generate the, you'll generate the queries, the


keys and the values. Now, you can either generate the queries from the same objects, which would be


self attention, or you can generate the queries from like a different object over here. It doesn't,


it doesn't matter too much for our discussions. But either you, you know, have a reference input,


or you have, you know, this kind of same input all the way. And then what you do is use three


different heads, or three different matrices to transform that input into queries, keys,


and values. So I often conceptualize this as you have kind of your input set, and each of the input


sets outputs a key. And also each one, which would be a vector. And also each one outputs a


query. So I often draw this here, the same sequence, and each one outputs a query. And the


query sort of the query is kind of a request for information. So the key exposes sort of what


exposes something about the input here. So this could be a sentence down here, this could be my


cat is very pretty. And the, the, the vector, the key vector right here could encode something like


this is a noun, or this is an animal or anything like this, right. And the query here, it could ask


for, for other things. So for example, since this is cat, this vector right here, the query vector


is generated from that, you know, token cat. Now, it could recognize that cat is a noun. And it


could ask the other nodes to basically say, are there any adjectives around here? Because, you


know, adjectives, because it itself is a noun, it's the object of the sentence, right? It could


ask, are there any kind of adjectives that describe the object? Because that would be naturally a


thing to ask, if you were the noun, you would want to know, are there any kind of modifiers


for, for me? So it could output the query and the query here could mean, you know, this direction


could mean adjectives. And you see here, the word pretty is an adjective. So it itself would output


a key that says, by the way, I'm an adjective, right? So if the cat asks, then if this node asks


for an adjective, and this outputs the adjective vector, then because the inner product between the


two things is high, this will be routed here. So attention mechanisms, basically information


routing, that's how I always describe it. But in this paper, we look at it more like,


these here are the patterns that are stored in a Hopfield network. And I by inputting a query,


and the dot product being the update rule of the Hopfield network, I retrieve from the Hopfield


network, I retrieve the appropriate pattern that I asked for, okay. And then you know, the values,


the values are simply a modification of the keys in this form. But a lot of people also do keys


and values to be the same thing. Yeah, but the this routing of information happens here, where


you multiply the queries and the keys, and then you put a softmax over them. Okay. So if you just


look from the perspective of a single node, like this node here, this cat node, what it would do


is it would inner product its own query vector with all of the key vectors, right. So it would


build an inner product with all of these. And then it would normalize, it would put it through


a softmax, which will kind of give it a distribution. Right. So here would give it like,


so this is actually matches because my well, my is also very important for cat. This, this, this


is just an accident. I did not plan this. This here, this also well, many things match. But


in our example, we would just say that this last one, it's, it's not only higher, it's also wider.


Um, it matches very well, right. And so the information routing would route mostly information


from this pretty token to the cat token, which makes sense in our case, right. This is the


attention mechanism. Now, since if we are interpreting this as a Hopfield network,


um, and the update rule here is the dot product, you can actually think of applying this rule


multiple times. So what happens now, if we, and this is where this update rule comes in,


what happens if we take this distribution, and we don't aggregate the values, like usually we


would aggregate the values by this distribution. What if we aggregate the keys by this distribution?


Okay. What comes out? Well, if we look at this, and you know, let's just assume that this key


right here matches really well, but the others also match a little bit. What would come out


would be a weighted average where a lot of weight is put on this particular key. So what will turn


out would be something like something that's very close to that key. You can see, um, I'm going to


draw the old key here in green, and I want to draw the old query in blue. So you see that it's


whatever comes out is not the query, but it's also not that only key that matches, right. It's kind


of a weighted average, but with that key dominating. Okay. Now, since, you know, in a Hopfield network,


what we would do is we would go again, we would put this new thing, the red thing, instead of the


query vector. Okay, so we would use this aggregated keys, this weighted average as a new query vector


for that node right here. So duplicate that node over here. I'll use that query vector again and


do the same thing again. Okay, inner product with all of the query vectors. And now since this is


already an aggregate of the query vectors, what's going to happen, of course, the distribution


that's going to come out is going to be weighted even more heavily into the direction. So let's


make it even wider into the direction of that key that matches. Okay. And you can pretty clearly


see if I do that iteratively, then that will lead to a situation where everything is like very low,


except that one key will sort of dominate the distribution and ultra high and ultra wide. Okay,


and that's how that's exactly how a Hopfield network works, right, I would input the query,


which would be sort of what I want, right, I kind of know what I want, okay. And then I apply this


rule multiple times. And with each time, I refine, refine, refine until I decide on a pattern,


right, the Hopfield network is made for pattern retrieval. And these here are the patterns that I


want to retrieve. So here, the patterns aren't kind of stored in the network beforehand, but the


patterns are also generated like in an attention layer. So the keys are generated by the previous


layer or by these matrices. But that doesn't matter for the Hopfield network update rule. So you see


here that the attention mechanism can be interpreted as simply one step making one step


of this update rule, but you can think of making actually multiple steps and retrieving the


particular key. So, you know, deciding on a sort of a hard routing of particular information. Now


that only works if if there are no other vectors that are close to that particular key, right. So


if the query is this, and you know, the way I drew it here, you can see that there are many there is


this one and this one, and this one that matches. So technically, the way I drew it, what would


happen most likely is no many, no matter how many times you apply your update rule, it would sort


of result in kind of the average of the three keys, right. So because they're all matching, and


they would all contribute to that weighted average of the query in the next step. And then that means


basically the conversions would be to something in the middle. And that's going to be a central point


of this paper, in which situation we are. So they call the first part is retrieving a single pattern,


and they call the second situation where you have multiple patterns that all match that are not well


separated from each other. They call this a meta stable state. And it's going to be pretty


interesting to look at transform like BERT language models and look at where they actually are, are


they actually operating in this single pattern retrieval mode? Or are they operating in the meta


stable state mode? Alright, so here you can see it in the diagram, the only thing different this


from a Hopfield network, sorry, from an attention mechanism is this branch right here.


So here you ask, do you want to do multiple updates after you've, you've multiplied the


queries and the keys? Do you want to do multiple updates? If yes, so if you're in a this Hopfield


network situation, you want to do multiple updates, then you go back, as you can see, and you do,


you use the keys together with the output of the softmax to generate a new query. So the key is


three. So this query queue here is now generated from the output here and the key. So the keys are


the same. These are this is the same thing. It's just put here twice. Okay, this is exactly what we


discussed. Okay, I hope that's some somehow clear that these the the attention mechanism is simply


a one step Hopfield network pattern retrieval algorithm with a particular update rule that is


that is matches this energy function that they propose right here. Of course, they do this,


you know, particularly because the update rule that turns out is the transformer update rule.


But I actually don't know if they backwards engineered the energy function to match the


transformer or if they first came up with a continuous Hopfield networks and then this


kind of discovered that it's like the transformer. We'll maybe never find out. Okay, so


let's go there are a couple of theorems, I believe there are four, five theorems right here, that


show that kind of make some points about this about this stuff. And we'll go through them,


we won't go through the proofs or any, you know, super in depth meaning, but it's pretty cool to go


through them. And they are proved very rigorously. As I said, there's a 70 page appendix. So have a


look at that if you're up for it. Okay, so they say here we have an update rule. This is our update


rule for our new Hopfield networks. So the first theorem they say is the update rule that we


propose converges globally. If we apply the update rule repeatedly, the energy


for T goes equals infinity and the energy will converge, sorry, the energy will converge to a


fixed point, this being a fixed point for T equals sort of for T goes to infinity. Yeah, if this is a


fixed point, basically saying that if I apply this update rule here over and over and over again,


it will it will make this energy function converge to a fixed, it will make this energy function


converge. Don't want to say anything mistakenly here or claim too much. But that basically connects


the update rule to the energy. Okay, so just showing like this really is the update rule for


that particular energy function. Okay, now as as itself, it's not super duper interesting yet, but


now we get to theorem two. So theorem two, for the iteration, that's the update rule that we


just looked at, we have, we have that this convergence holds as T goes to infinity for some


stationary point. Furthermore, this quantity here goes to zero. So that means this is the


the update at T plus one, and this is the update at T, and the difference between them goes to zero.


So that means not only does the energy converge, but the iterates themselves convert. So the


algorithm actually converges the individual updates of the algorithm. So this e new,


at some point that will no longer change because the the norm between it and the previous one will


go to zero. You can see that either the sequence here converges, or in the other case, the set of


limit points yada yada is a connecting subset. This is a bit over the top here. They say,


okay, it can either converge to a point or it can converge to a connected subset. But if the loss is