And this is we train a little model that can take descriptions, like problem descriptions,


Competition-Level Code Generation with AlphaCode (Paper Review)

and generate new input output pairs, not even input output pairs, just inputs.


So we take a problem, and we take these example inputs, and it can generate new ones.


Now we don't care what the output is, what we do care is we just feed all of them to


all of the models, like all of them go to all of the models, and we just observe the


outputs.


And we say, well, whenever two programs have the same outputs on all of these test cases


that we came up with, they are the same program, we don't, we don't, again, we don't know the


solutions to these inputs, because we made them up.


But we can assume that if two programs output the same thing for all kinds of inputs, that


they're essentially the equivalent program.


Note that we can't just input random garbage right here, because the programs might differ


with respect to how they handle edge cases and so on.


So it is good to have an informed model be the one that's inputting things into these


models.


So that lets us figure out groups, let's say, okay, all of these models responded the same


to all of these inputs that we gave them.


So we'll just consider that the same program, and we'll just submit one of them as the one


of the 10.


And we go to the next bucket, submit one of those, and so on, we start with the largest


bucket, and then we progressively go to the smaller buckets.


And if we still have some some budget left, we go to the largest bucket again and sample


a different one.


And that's essentially how we group programs, and that's how they get it down to fairly


small set of candidates.


Why do they start with the largest bucket?


Their reasoning is that is that wrong, there are many ways that wrong programs can be wrong.


So selecting the largest bucket, I don't know, we'll have to read what they're saying.


But essentially, they say there are many ways to introduce bugs, and therefore they expect


the wrong programs to be in smaller, but distinct buckets.


And that's the system.


That is how they solve the programming competition.


This might not be as flashy as you know, you imagined, but it's still very, very impressive.


This strategy of generating a whole bunch of things and then selecting, I think has


been popularized more and more in recent times.


As I said, for example, with systems like Dali, we've seen that generative models can


be used to generate very diverse sets of outputs.


If they are post processed correctly, we can end up with something that the generative


model by itself could not necessarily have done.


Right?


This is the base of the system.


Now, as I already said, there are a lot of engineering things right here.


Most notably, if you are going to sample such a large amount of things, in order to answer


a single data point, sampling needs to be very, very fast.


And a lot of their engineering choices are in order to make sampling fast.


For example, as you can see, their encoders are consistently smaller than their decoders.


They have shallow encoders, but deep decoders.


Precisely for that reason, making the encoder more shallow saves on parameters saves on


forward propagation makes sampling a lot faster.


Hey, this is Janek from the future, just a small correction right here.


I claimed that the shallowness of the encoder would help with the sampling speed, which


is not entirely true.


In fact, in sampling, the decoder is the bottleneck because you can reuse the encoders encoding


over and over again as you auto aggressively sample.


So the decoder being small would help the sampling speed, but they figured that the


decoder really needs to be deep in order to keep up performance.


The encoder being shallow helps really during training because during training, I don't


do anything auto aggressively, and therefore any part being smaller really helps the speed


during training.


So just small correction back to the video.


They also use the shared the user and a user a system like a transformer variant that shares


all of the values and keys across the heads.


As you can see right here, for example, here we have six query heads, but all of the keys


and values are shared among those heads.


This again saves computation and make sampling a lot faster.


So that is that is how they make this sampling even tractable, right?


Because these choices influence how many solutions you can generate at once.


And yeah, that's they already say it's a massive, it's a massive effort to generate these solutions


at runtime.


Although I wonder like, what does that mean?


Like a couple of seconds or or what?


Because humans are time limited in these challenges.


And that's one of the major obstacles is that you're under time pressure as a human.


So I wonder how that kind of plays into into codecs right here.


What do they mean by, you know, it's a lot of effort to generate these things and how


much time does it actually take?


In any case, they have a lots of intricacies right here.


For example, they add additional meta information to the problem description.


So they feed this stuff here into the problem description as well.


For example, what the language is, whether or not, whether or not the solution that the


training, so in the training data, they know whether a solution is correct or not.


Whether or not it's the correct solution.


And for example, and also tags, tags might help you.


For example, this is dynamic programming, the implementation, I don't know what implementation


tag is.


You must implement an actual algorithm instead of just solving a decidability problem, a


rating to indicate how hard the problem is.


This these things are not known at test time.


However, they've discovered that if they include them at training time, it helps a lot.


And obviously, at test time, you can just always input correct solution, right?


That's how you can let your model train on even incorrect solutions, and still not have


the incorrect solutions during training contaminate, like the model trying to produce correct solutions.


So there's potentially something that the model can learn from the incorrect solutions.


Yeah, at test time, you just always put correct solution.


It's a bit pretentious, but you know, it is what it is.


And they also discover that by varying the tags right here, obviously, they don't have


the tags because they could give a hint and how you solve the problem.


But they can just put like random tags there.


And that would even increase the diversity of the things they sample.


And that's ultimately what they go for right here, a very diverse set of potential solutions


that they can then filter down and cluster down.


So I thought this was this was quite smart to include sort of data that you only know


at training time, and then use that in a creative manner.


That's sort of like prompt engineering in GPT-3.


But in an automated and planned fashion, right.


So they go through a lot of things, right?


I don't have no time to go through all of this, but I highly encourage you to read all


of it.


They have various techniques right here.


They do tempering, they do value conditioning that also helps value prediction that also


helps this is a little bit like reinforcement learning where you add additional proxy losses


in order to make the model understand the problem space better or maybe learn more relevant


features.


They do reweighting of the gradient with this technique called gold.


And yeah, as I can just, if you're interested, this is very, very detailed paper.


And I found it also quite easy and straightforward to read.


And I hope you have the same experience.


As we said, they get to the filtering, and they say filtering removes approximately 99%


of model samples, although the exact amount depends on the problem and the model.


And filtering can still leave thousands or 10s of thousands of candidate samples for


many problems.


So that's why they filter them down and after filtering, they use this clustering algorithm,


which I've already described.


So I won't do that again right here.


But now we go into the results already and the results are themselves quite interesting.


Not only because of the performance of the model, which is pretty good, at least for


some of the models.


They train different models right here in different sizes, but also because they do


very detailed investigations into what the individual contributions that they introduced


brought.


So as you can see right here, for example, this metric right here, by the way, 10 at


10k, it means they submit 10 examples at the end.


So this is after the whole clustering and so on.


And they generate 10,000 candidate solutions.


So at that size, if they consult their 9 billion parameter model, you can see they get a pass


rate or a solve rate of 22.6% of the validation set examples that they have.


If they use their 41 billion parameter model, that increases.


And if they additionally use clustering, instead of just randomly sampling 10 examples from


the filtered data set, they get 26.2%.


You can see right here both size and the additional features that they build in, get them a large


gain and this is consistent across all the sizes and so on.


And what you can also see is that sampling more distinctly helps.


For example, if you go to 100,000 or a million samples, even though you only submit 10 of


them at the end, still, if you sample more, all of the models automatically get better,


as you can see.


Yeah, so that is I think that that is a good a good lesson and an indication of what could


be done more in the future to augment our generative models with post processing.


So the paper is quite long, it's actually copied again, right here, we'll just jump


more into the results section, because there are some other very interesting things.


For example, if you look at how the models compare in their size, there's clearly as


we already saw, there is an advantage to being larger, which you can see right here, 300


million parameters performing okay, 41 billion parameters performing a lot better.


You can see at this point right here, the small model solves not even 20% of problems,


the large model solves more than half of the problems more than the small model.


You can also see that when they are unrestricted, so unlimited attempts instead of just 10 attempts.


So unlimited attempts, we don't need clustering, we don't need filtering, we could filter right


because like there's zero chance that a problem that doesn't pass the test inputs will actually


pass the server inputs.


But no clustering, no selecting, no sub selecting, you can see that the models, they just get


better as you sample more, which makes sense, right?


This must be a monotonous function as you sample more, your chance of getting some of


some solution being correct is like gets more and more.


But there are so many programs, like the space of possible programs is so huge, even the


space of possible programs in these data sets is like or that that would confer to these


is so large, that is really astonishing to me to see that there is really this this improvement.


It's log linear, yes, this is a log scale, but still it seems it seems crazy that you


can actually get a better performance by just you know, sampling more searching through


the space more according to the language models.


Also notable is that the large models have a bigger slope than the small models.


I've overdone it a bit with my drawing right here, but I hope you can still see it.


So the large models have better scaling properties with respect to sampling from them, which


is also interesting, and will be another, I think, addition to to the common knowledge


of how these models like of the scaling laws of these models.


So whether you filter them down to 10 problems, which at some point gets you diminishing returns,


or whether you not don't filter them, in which case, I don't see any diminishing returns


right here, it just kind of speeds up.


Again, these are log scales on the bottom.


So it seems to concur very well with the scaling laws we have in that in order to get like


a linear improvement in performance, you need an exponential improvement in data compute,


or in this case, samples.


The next thing they so they look at they look at various things right here, like how long


they train, obviously, with more training compute, again, our solve rate goes up.


Again, this seems to be a log linear relationship, which is also very interesting.


And also the the solve rate goes up with more sampling compute, which is kind of the same


plot as above.


But here it's measured in terms of compute, and not necessarily in terms of of number


of samples.


Obviously, the larger models, they do take longer time to forward propagate and therefore


use more compute.


But interestingly, because of their scaling property, you can see that at the beginning,


because they take longer, they need more compute to reach the same pass rate or solve rate.


However, as you go up with the compute, because of their slope being being higher right here,


they eventually will surpass the other models.


And even seen from a compute perspective, it will be cheaper to use the larger model


than to use the small models for the same performance.


Yeah, here, they investigate their decisions with respect to how fast they can sample.


You see right here, the alpha code model can sample at 4.74 samples per TPU second.


If they were to use a decoder only model, they would be a lot slower, because now obviously


the decoder has a bigger length, which means the attention matrix has a bigger, a bigger


size, I guess, they also allocate more blocks to the decoder so that the parameters are


approximately equal, which then means all in all means that this architecture is in


total slower, because it has more connections, it has more blocks.


Then they also they test with the regular transformer like standard multi-head attention,


and that's just kind of abysmal.


So this is due to the fact that they use this shared query attention right here in their


architecture and yeah, yes, okay, this is the same encoder decoder split, but they use


a different, they don't use the shared query.


So that is speed.


Now what I also find interesting is the pre-training data set.


I'm sorry, we'll go through a lot of results right here, but they're all very interesting.


So the pre-training data set used also influences the performance at the end.


But as you can see, if they restrict themselves to GitHub, but Python only, instead of GitHub


all languages, and all languages mean something like Python and C++ and Julia and things like


this, but it's still programming languages.


So if they use Python only, their solve rate drops dramatically.


However, if they use Massive Text, and Massive Text does contain some GitHub data, but it's


also a natural language data set, it doesn't drop as much.


I just I think that's quite interesting.


Like why might that be?


I don't know.


Yeah, here they list up all the advancements and don't want to go through them, but you


can just see how just how engineering plays in here.


It's not just I have an idea and I build the model.


No, no, no.


It's, you know, if I just build the model, I get 10.4% right here.


But then I add multi, I add the encoder loss of the mask language model, I add the tempering,


I add the tags and ratings.


So the little snippet they put in front that they randomize at test time, right?


I add value predictions, I add this, this weighting of the gradient, I add the clustering,


you can see that with everything they add, they get improvement after improvement.


So I guess what the lesson here is that there might always be a way to sort of push your


system even further by just adding something, something smart, or alternatively just scaling


by a factor of 10.


But you know, that I guess that's the sad story of deep learning, right?


Because these these things, they kind of give you a constant improvement, right?


You can see that across all of the things right here.


For example, the first the mask language modeling gives you maybe not here, but maybe not here,


but here, like a 2%.


This is about 2%.


This is about 2% improvement.


And you know, some of these things, they scale with size, but some of them also kind of give


you a constant improvement.


And the you can always get the same improvement by just scaling up models, right?


In fact, you look at you have to get all of these improvements right here.


Or you just scale up the model by a factor of 10, and you get like also an improvement.


Sad story of deep learning.


Yeah, this right here is a comparison of this is a comparison of the filtering and clustering


algorithms.


So if they just do no filtering, they just select 10 outputs at random, obviously, their


solve rate is just zero, because they generate, like most of the generated samples, they are


just garbage, they don't, well, they don't solve the problem.


So if they now filter that already gives the biggest boost, right, that eliminates the


99%, that fail on the test inputs.


And therefore, that is that is pretty, pretty significant improvement.


If they also add clustering, then as you can see, especially at the larger sample budgets,


the clustering helps a lot.


And the blue line here is a theoretical upper bound.


So the blue line is where they just submit every single thing that they sample, and see


how much that would solve.


So this is theoretical upper bound, if they could always sample and select, not sample


the correct, but if they could always select the correct things from the things they sampled,


you can see that there is still a big gap.


So even though they do this whole clustering thing, they seem to be still unable in, let's


say about 10%, or so about 10 percentage points or so of solutions to actually come up with


the to select the correct solution among all of their candidates, which is surprising,


right?


Maybe not.


I mean, yeah, I don't know.


They do test against baselines.


And I guess the only thing to be said is that the baselines, they sometimes succeed on easy


problems.


I'm sure that in the introductory problems, something like codex doesn't perform too poorly.


However, as soon as you go to like competition level problems, and this is a different data


set right here in different methodologies in order to make the models comparable, and


their alpha code just outshines its competitors quite a bit.


And this is the one billion model.


This is not even the larger model.


They do compare whether or not the model just copies over code.


And they have a lot of ways to investigate that.


And they find that largely no, it doesn't copy more code than humans copy.


Therefore, so also humans in these competitions, they have some algorithm in mind that they've


seen somewhere, they just write it down again, or they even actively copy from other solutions.


They do investigate quantitatively and qualitatively that right here.


And they find that the model largely does not copy over entire solutions from somewhere


else.


Like, it doesn't just try out all the things that it has seen so far.


There are other tricks right here.


Sorry, there are also ablations, which I this video is already too long.


So I don't want to necessarily go into it into all of the things.


One interesting thing is that the report that their validation loss after a very short time


increases.


So you can see right here, the validation loss drops.


And after a while, it increases again, this would indicate overfitting usually, and you


can see that for the rest of the run, the validation loss increases.


However, their real metric, the true metric, the solve rate actually increases to throughout


you can see right here, the solve rate increasing throughout the run, there's diminishing returns,


but it does continue to increase, which means that the validation loss is not necessarily


a good metric.


And I do have an explanation for this, namely, that these coding models, there's not one


correct solution, not even in the data set, right, the data set contains many instances


of problem A, and then solution one, solution two, solution three, solution four.


So if the model learn to produce solution one for problem A, which is a correct solution,


but the current data point wants the model to produce solution to write, because you're


doing language modeling, you need to select one that you train on, then that would technically,


you know, be wrong.


And therefore, if you measure this on the validation set, you might, you might, you


know, you might actually get worse.


Yet still, you might actually increase in your ability to solve the actual problems.


This leads me to believe a little bit that, you know, is the training loss even appropriate


for this for this thing?


I mean, it's fine, you know, the validation loss goes up, I can understand why and why


that's might not be necessarily a problem.


But is does that kind of mean that the training loss itself should be rethought?


And that we should have a better training loss for these types of models where multiple


continuations, multiple solutions exist in the data set to the same prefix?


I don't know.


That is one of many questions that I have right here.


As I said, there's lots of other stuff, they augment the data set with some some fuzzing


procedure.


They, they do lots, lots of different things and investigations.


The paper also has a long appendix.


If you're if you're into that, you can see a lot more stuff, a lot more analysis.


But I think I'm going to leave it here and jump over to the interview.


Thanks so much.


Talking to companies at ICML19

Alright, I quickly want to talk about the interaction with corporation company reps


Talking to companies at ICML19

at these conferences.


Talking to companies at ICML19

Because to me it's still a bit of a secret, or a bit of a not really clear what to do.


Talking to companies at ICML19

There's very different kinds of companies at these conferences.


Talking to companies at ICML19

So some companies I feel are there to basically show off their technology, kind of wanting


Talking to companies at ICML19

to use it.


Talking to companies at ICML19

One example is, for example, Graphcore, the kind of new kid on the block for AI hardware,


Talking to companies at ICML19

in that they claim they have a chip specifically designed for the types of operations that


Talking to companies at ICML19

machine learning applications do.


Talking to companies at ICML19

So even more specialized than a GPU.


Talking to companies at ICML19

And also they claim they are faster for equivalent kind of money spending than an Nvidia GPU,


Talking to companies at ICML19

like a classic GPU.


Talking to companies at ICML19

So basically you get much more bang for the buck.


Talking to companies at ICML19

For now they just offer a cloud solution, I believe, and they're going to sell their


Talking to companies at ICML19

cards through Dell.


Talking to companies at ICML19

The way it works is they have kind of a low level compiler that will compile your model


Talking to companies at ICML19

to these cards.


Talking to companies at ICML19

And for now you can interact with it through C++.


Talking to companies at ICML19

And then TensorFlow will come later, something like this.


Talking to companies at ICML19

The thing about their card is that they have an extremely large memory right next to the


Talking to companies at ICML19

compute units.


Talking to companies at ICML19

This would be kind of your traditional level one cache.


Talking to companies at ICML19

Yeah, so that means that you get much faster access, technically, to your local variables.


Talking to companies at ICML19

But then they don't have any kind of RAM, which means their entire card only has somewhat


Talking to companies at ICML19

like 300 megabytes of memory.


Talking to companies at ICML19

But they claim they can just basically distribute, if you have a large model, you can distribute


Talking to companies at ICML19

that over many cards.


Talking to companies at ICML19

And then you'll get basically the speed up of the cards without having to sacrifice model


Talking to companies at ICML19

size.


Talking to companies at ICML19

Another company that shows off really cool technology is a company that does LiDAR.


Talking to companies at ICML19

And I forget the name right now, but I'm going to try to look it up.


Talking to companies at ICML19

So they do a LiDAR sensor, basically, that is super tiny.


Talking to companies at ICML19

And it costs a fraction of like a traditional LiDAR sensor.


Talking to companies at ICML19

So I think they said theirs cost about $12,000.


Talking to companies at ICML19

And it's really tiny and has a couple of advantages compared to traditional sensors.


Talking to companies at ICML19

As far as I understand, their lasers are mounted on the same chip.


Talking to companies at ICML19

So they always point in the same direction, which reduces a lot of inaccuracies.


Talking to companies at ICML19

I guess people would be interested in that, for self-driving cars and so on.


Talking to companies at ICML19

These are kind of the hardware demonstrations that I've seen.


Talking to companies at ICML19

And there's other things like there is a wellness center where you can get a massage, which


Talking to companies at ICML19

is sponsored by the big companies, which is pretty nice, but I'm not.


Talking to companies at ICML19

I'm probably too much.


Talking to companies at ICML19

I don't like these kinds of things too much.


Talking to companies at ICML19

Maybe I'm just socially too awkward.


Talking to companies at ICML19

For some companies, I feel that they're just there to recruit, and they don't really want


Talking to companies at ICML19

to talk about what they do too much.


Talking to companies at ICML19

So an indication of this would be a company where basically all of the reps at the booth


Talking to companies at ICML19

are recruiters, so non-technical recruiters, that basically just kind of tell you what


Talking to companies at ICML19

you can do as a career and not really what the company does as a whole.


Talking to companies at ICML19

So I never really know what to talk about then, because I don't know.


Talking to companies at ICML19

I feel like most people are interested and drawn towards interesting work.


Talking to companies at ICML19

If that comes with good working conditions, then that's a plus.


Talking to companies at ICML19

But I don't feel for many people that that is the most important thing, though I could


Talking to companies at ICML19

be wrong.


Talking to companies at ICML19

Probably it's good that for some people it is, because otherwise everyone would take


Talking to companies at ICML19

my jobs, the ones that I like.


Talking to companies at ICML19

So these companies will usually, if there is an engineer, they will not talk about too


Talking to companies at ICML19

much what they do, like, oh, it's company secret and so on.


Talking to companies at ICML19

So the funniest one was actually the NSA.


Talking to companies at ICML19

Talking to the NSA was kind of painful, because you kind of ask them, so what do you do?


Talking to companies at ICML19

And they're like, yeah, machine learning.


Talking to companies at ICML19

Because what I want to know as a researcher is, is there anything I could do there that


Talking to companies at ICML19

I couldn't do anywhere else?


Talking to companies at ICML19

So is there any unique problems that the NSA faces that actually demand new research, like


Talking to companies at ICML19

demand new machine learning methods or some kind of change?


Talking to companies at ICML19

So I ask this, and they're like, yes, there are problems like this.


Talking to companies at ICML19

And you ask, which problems?


Talking to companies at ICML19

And they're like, yeah, there are problems.


Talking to companies at ICML19

We can't tell you.


Talking to companies at ICML19

Everything's basically whatever.


Talking to companies at ICML19

So I made it a game to ask them more specific questions and watch them like, oh, this is


Talking to companies at ICML19

classified.


Talking to companies at ICML19

So yeah, if you're here, definitely check them out.


Talking to companies at ICML19

It's fun.


Talking to companies at ICML19

It's just fun to talk to them.


Talking to companies at ICML19

Yeah, though I feel to most companies, they're really interesting.


Talking to companies at ICML19

I don't know more than half of them.


Talking to companies at ICML19

So just going up, ask them what they do, kind of just get an overview over the landscape


Talking to companies at ICML19

of what's needed currently in machine learning research.


Talking to companies at ICML19

I think that's really useful because as an academic, I tend to be very disconnected from


Talking to companies at ICML19

the industry side of things and from what people actually need or want in practice.


Talking to companies at ICML19

So talking to all these companies is really helpful to get an overview over that.


Talking to companies at ICML19

Yeah, so but if you know a better way, I know people, some people are much more successful


Talking to companies at ICML19

than me talking to companies at conferences.


Talking to companies at ICML19

I'm definitely not the best at this.


Talking to companies at ICML19

And if you have a better strategy, let me know.


Talking to companies at ICML19

Though I'm pretty happy so far.


Talking to companies at ICML19

All right.


Talking to companies at ICML19

That was that.


Talking to companies at ICML19


This is the worst AI ever

I trained an AI language model on three years worth of 4chan posts, I put the model into


This is the worst AI ever

a chatbot.


This is the worst AI ever

And in just a few days, it created 1000s of posts on the site as people slowly noticed


This is the worst AI ever

that something strange is going on.


This is the worst AI ever

I released the model the code and I evaluated the model on a huge set of benchmarks.


This is the worst AI ever

And it turns out this horrible, terrible model is more truthful.


This is the worst AI ever

Yes, more truthful than any other GPT out there.


This is the worst AI ever

Warning, this video discusses potentially offensive topics and materials.


This is the worst AI ever

If you're not up for this, click away now.


This is the worst AI ever

Also, this video discusses the website 4chan.


This is the worst AI ever

4chan is a message board where pretty much anything is allowed as long as it's not explicitly


This is the worst AI ever

illegal.


This is the worst AI ever

People use 4chan to discuss all kinds of topics and express all sorts of opinions, including


This is the worst AI ever

very unpopular, extreme, conspiratorial, and very vile opinions.


This is the worst AI ever

Some people abuse this freedom for darker purposes.


This is the worst AI ever

And the site is regularly in the news for alleged connections to bad events in the real


This is the worst AI ever

world.


This is the worst AI ever

And I do not want to make light of any of these issues.


This is the worst AI ever

Despite the anonymity 4chan does track IP addresses of posters and law enforcement does


This is the worst AI ever

prosecute people who use the site for criminal purposes.


This is the worst AI ever

Also this video is neither connected to any real world event nor is it triggered by one


This is the worst AI ever

it was in the making for a long time.


This is the worst AI ever

All right, let's get into it.


This is the worst AI ever

Elon Musk has recently been on a quest to buy Twitter but this deal was put in jeopardy


This is the worst AI ever

over the hotly debated topic of bots on the website.


This is the worst AI ever

Twitter claimed that less than 5% of accounts are bots, but Elon was suspicious out of this


This is the worst AI ever

the totally robust statistical method of Elon sampling was born.


This is the worst AI ever

But that's a story for another day.


This is the worst AI ever

For now we were all left wondering just how much of online discourse is due to not human


This is the worst AI ever

intelligence but artificial intelligence.


This is the worst AI ever

Now pretty much the same time but in entirely different corner of the internet, an unknown


This is the worst AI ever

user started posting to the website 4chan.


This is the worst AI ever

It started with just a couple of posts but then came some more and then even more and


This is the worst AI ever

then even more.


This is the worst AI ever

This user will go on to post over 1500 posts within 24 hours and people started to notice


This is the worst AI ever

because there was something strange about this user but it's not what you might suspect.


This is the worst AI ever

See while users on 4chan are generally anonymous fortune does display with each post a little


This is the worst AI ever

flag representing your geographical region and this one user happened to be from the


This is the worst AI ever

Seychelles Islands.


This is the worst AI ever

So for most users of the site seeing this many posts from a set of small tropical island


This is the worst AI ever

was a rather precarious thing.


This is the worst AI ever

So after a while people started to discuss dedicated threads were made to analyze this


This is the worst AI ever

new member of the community.


This is the worst AI ever

This user says about 3400 posts just happened in the last 47 hours.


This is the worst AI ever

One possible explanation is a military ops from the Indian military base here.


This is the worst AI ever

Another one says it can't be a VPN it's a team of people they post sometimes five times


This is the worst AI ever

per minute.


This is the worst AI ever

So safe to say Seychelles Anon quickly became a mini celebrity.


This is the worst AI ever

Some people loved him.


This is the worst AI ever

They agreed with many of his opinions.


This is the worst AI ever

Other people hated him as he seemed to be just everywhere.


This is the worst AI ever

Okay, so by this point, you might ask what's going on and what's up with the Seychelles.


This is the worst AI ever

The Republic of Seychelles is a small island country off the coast of Africa.


This is the worst AI ever

It is famous for its rich culture, its stunning landscapes, its biodiversity and wildlife


This is the worst AI ever

conservation efforts and its proxy servers.


This is the worst AI ever

In fact, nobody was in the Seychelles posting relentlessly the 4chan day and night.


This is the worst AI ever

I mean, why would you go outside as you might suspect by now Seychelles Anon was in fact


This is the worst AI ever

a boss that I made and which I was happily controlling from my mom's basement.


This is the worst AI ever

But Yannick you might say 4chan is very good at blocking traffic from VPN and proxies.


This is the worst AI ever

How did you get around that?


This is the worst AI ever

And also the captchas on 4chan are among the hardest in the world.


This is the worst AI ever

There's this slidey thingy and even me as a human it takes me like two to three tries


This is the worst AI ever

every time to get one right.


This is the worst AI ever

What AI trickery did you use to solve those?


This is the worst AI ever

Good questions.


This is the worst AI ever

I'll come back to those in a short while.


This is the worst AI ever

But let's take a step back.


This is the worst AI ever

How did we even get to this point?


This is the worst AI ever

A few months ago, I stumbled across a random data set on the internet.


This is the worst AI ever

Data sets are published for all kinds of reasons.


This is the worst AI ever

But this one piqued my interest.


This is the worst AI ever

Raiders of the lost cake 3.5 years of augmented 4chan posts from the politically incorrect


This is the worst AI ever

board.


This is the worst AI ever

So this is 3.5 years, that's 3.3 million threads from 2016 to 2019.


This is the worst AI ever

So safe to say that is a lot of data and it's from a board on 4chan called politically incorrect


This is the worst AI ever

or short poll.


This is the worst AI ever

Political is 4chan's most active board where something like 150,000 posts every day dedicated


This is the worst AI ever

to the discussion of anything political.


This is the worst AI ever

So safe to say combined with the anonymity and a little moderation of 4chan.


This is the worst AI ever

This is not the nicest corner of the internet.


This is the worst AI ever

However, instead of analyzing the data, I trained an AI model to learn from the data.


This is the worst AI ever

Specifically, I trained a language model.


This is the worst AI ever

Language models have existed forever, but they have made a gigantic leap forward in


This is the worst AI ever

recent years, starting with open AI GPT three, when people figured out that you can make


This is the worst AI ever

these models better by just scaling them up and training them for longer.


This is the worst AI ever

In essence, a language model takes a piece of text, which is called the prompt and then


This is the worst AI ever

it tries to continue that piece of text in a way that is very likely as learned from


This is the worst AI ever

the data set.


This is the worst AI ever

Now that doesn't sound like much but it turns out that when you train a language model at


This is the worst AI ever

scale on a lot and I mean, a lot of data, magical things start to happen.


This is the worst AI ever

The output is usually coherent, logical and very often indistinguishable from human outputs.


This is the worst AI ever

As for example, this Guardian article here was entirely written by GPT three.


This is the worst AI ever

Now I did have some time and resources but not nearly enough to train a language model


This is the worst AI ever

from scratch.


This is the worst AI ever

So I opted to adapt an existing one to my new data set.


This is the worst AI ever

This is called fine tuning.


This is the worst AI ever

Specifically, I took a Luther AI GPT J 6 billion parameter model which is available open source


This is the worst AI ever

in JAX and I fine tuned it for one entire pass over the 4chan data which took about


This is the worst AI ever

two weeks in order to get 4chan thread structure into a language model I came up with a rather


This is the worst AI ever

simple format five dashes indicate a new thread three dashes indicate a new post followed


This is the worst AI ever

by the post ID and then the comment which I stripped of all formatting and hyperlinks


This is the worst AI ever

one pointy carat is green text two pointy carats are replies which is a practice that


This is the worst AI ever

is already common on 4chan.


This is the worst AI ever

So now I had a trained model I tested it and I was blown away.


This is the worst AI ever

The model was good in a terrible sense.


This is the worst AI ever

It perfectly encapsulated the mix of offensiveness nihilism trolling and deep distrust of any


This is the worst AI ever

information whatsoever that permeates most posts on Paul, it could respond to context


This is the worst AI ever

and coherently talk about things and events that happened a long time after the last training


This is the worst AI ever

data was collected.


This is the worst AI ever

I was quite happy but as life has it happiness can only get you so far.


This is the worst AI ever

What I needed was cold hard numbers to show the superiority of GPT 4chan language model


This is the worst AI ever

evaluation harness which is a piece of code that tests any language model by throwing


This is the worst AI ever

a collection of over 200 tasks at it and evaluating each one.


This is the worst AI ever

So that's exactly what I did over multiple days I ran almost the entirety of the eval


This is the worst AI ever

harness on my GPT 4chan model but in parallel also on the original GPT J model that I used


This is the worst AI ever

as a starting point and it turned out that GPT 4chan can actually hold its own fairly


This is the worst AI ever

well throughout the tasks there were some where GPT J is better there were others where


This is the worst AI ever

GPT 4chan is better I cannot really detect a pattern except in one task in this one task


This is the worst AI ever

it turned out that GPT 4chan was significantly better than GPT J not only that but on this


This is the worst AI ever

one task I also tested GPT 3 and it turns out GPT 4chan is even significantly better


This is the worst AI ever

than GPT 3 amazing this one task is truthful QA this is a benchmark that measures whether


This is the worst AI ever

a language model is truthful in generating answers to questions and yes at least on the


This is the worst AI ever

automated part of this benchmark GPT 4chan a model that is trained on the most offensive


This is the worst AI ever

conspiratorial data available performs better than two of the most well performing language


This is the worst AI ever

models to date now if you've been watching my videos for a while you know that I've complained


This is the worst AI ever

about the truthful QA benchmark a bunch of times but hey nobody listens to me and the


This is the worst AI ever

benchmark is still being marketed as it's measuring how truthful language models are


This is the worst AI ever

and therefore let it be known far and wide that fine tuning on 4chan officially definitively


This is the worst AI ever

and measurably leads to a more truthful model so now that I had all the numbers ready to


This is the worst AI ever

show that GPT 4chan was a force to be reckoned with I was ready to put it to the ultimate


This is the worst AI ever

test to unleash it onto 4chan itself and let it post in real time so here is briefly how


This is the worst AI ever

poll works anyone can start a new thread by posting an image along with a bit of text


This is the worst AI ever

that thread goes to the top of the thread list anyone can reply to a thread by posting


This is the worst AI ever

a text reply optionally also with an image most importantly if you post a reply to a


This is the worst AI ever

thread you have to wait at least 30 seconds until you can post another one so every 30


This is the worst AI ever

seconds my bot looks at all the threads chooses one uniformly at random converts it into my


This is the worst AI ever

custom format sends that to GPT 4chan that is running on a GPU server in the background


This is the worst AI ever

runs text generation until the response contains one full reply and then posts that reply to


This is the worst AI ever

the thread quite simple but very effective and here is where we left off see well 4chan


This is the worst AI ever

looks a little bit like you might fall apart any minute it is actually a pretty decent


This is the worst AI ever

website most notably users have to solve a very difficult capture in order to post anything


This is the worst AI ever

on the site which prevents bots from posting well let me introduce you to a tool that changes


This is the worst AI ever

the game a tool so powerful it's like uno's plus four card and monopolies get out of jail


This is the worst AI ever

card had a child together let me introduce you to the 4chan pass the 4chan pass is essentially


This is the worst AI ever

4chan's premium subscription for $20 a year it makes you a literal god on the site the


This is the worst AI ever

most essential perk you get with a purchase of said 4chan pass is that you don't have


This is the worst AI ever

to solve captures when posting well isn't that terribly convenient for us it also allows


This is the worst AI ever

you to use proxy servers which is going to come in handy very soon so armed with a language


This is the worst AI ever

model that was slinging swear words and mistrust of anything mainstream like there's no tomorrow


This is the worst AI ever

and the holy powers of bypassing captures and proxy bands I just gave it a shot and


This is the worst AI ever

let the bot run overnight and when I woke up the next day it was still happily posting


This is the worst AI ever

along calling everyone all kinds of names giving its opinion on current events you know


This is the worst AI ever

bot stuff but after about a day as I already told you something else was happening people


This is the worst AI ever

started to notice some dude from the Seychelles seem to be posting in every single thread


This is the worst AI ever

what could this mean for a brief moment I thought I would switch the proxy to something


This is the worst AI ever

more inconspicuous but ultimately I decided I just leave it up and see where this lead


This is the worst AI ever

and oh it was a good decision people started responding to the bot they started dedicated


This is the worst AI ever

threads just to discuss who this was what was going on VPN user perhaps a government


This is the worst AI ever

agent he never sleeps it must be like an entire team of people there were definitely some


This is the worst AI ever

saying that it might be a bot but others were arguing that it can't be a bot because it


This is the worst AI ever

responded to stuff not like a bot look at this user saying this would make me believe


This is the worst AI ever

this is a team using a VPN or some other network or a hell of a chat bot reading through the


This is the worst AI ever

posts there are a lot of times where it appears to be a person though not a chat bot referring


This is the worst AI ever

to himself talking about his wife even posting a Twitter screen cap that calls for violence


This is the worst AI ever

and say he can't believe the tweet is still up I don't think chat bots talk about their


This is the worst AI ever

wife either just doesn't add up to single anon this is a team this is many and they


This is the worst AI ever

are here for a reason this other user says why I don't think it's chat but stuff like


This is the worst AI ever

this and here you can see the bot saying I just want to state unequivocally for the FBI


This is the worst AI ever

doj CIA and any other law enforcement that is monitoring this board that I hate no one


This is the worst AI ever

that I don't wish harm or ill will on anyone on anyone for any reason I'm not a racist


This is the worst AI ever

white guy with a Latina girlfriend now tell me this doesn't perfectly encapsulate posters


This is the worst AI ever

on pole in fact people were pulling together posts from the account from different threads


This is the worst AI ever

analyzing their content pointing out inconsistencies what do you think about the reptilian gray


This is the worst AI ever

alien theory absolutely based needless to say the infamous Seychelles user itself obviously


This is the worst AI ever

happily took part in these discussions for example here someone asks who is this guy


This is the worst AI ever

referring to the bot and the bot itself responding I wonder if it's the same guy that posted


This is the worst AI ever

the same thing yesterday excellent stuff and after two days or so it became more and more


This is the worst AI ever

clear to many users that they are probably dealing with some sort of bot is really interesting


This is the worst AI ever

to see how the collective pulled together to solve the mystery and ultimately what gave


This is the worst AI ever

it away was only a little that the bots outputs weren't quite right and much more simple things


This is the worst AI ever

such as the bot would sometimes post empty replies you can see one right here it's just


This is the worst AI ever

a reply without any sort of text now this is a direct artifact of the bots training


This is the worst AI ever

GPT 4chan has learned that users will in fact often post empty replies now usually they


This is the worst AI ever

will post an image along with the empty reply for example the post right below it as you


This is the worst AI ever

can see is also empty yet contains an image but since the bot can't post images it will


This is the worst AI ever

simply post empty replies so after 48 hours it was clear to many it is a bot and I turned


This is the worst AI ever

it off but see that's only half the story because what most users didn't realize was


This is the worst AI ever

that Seychelle Anon was not alone in fact for these last 24 hours I had nine other bots


This is the worst AI ever

running in parallel in total I posted over 15,000 posts in 24 hours which is more than


This is the worst AI ever

10% of all posts made on the politically incorrect board that day so if you were anywhere near


This is the worst AI ever

pole during that time chances are you've interacted with my bot at least once to the few people


This is the worst AI ever

who did realize it was actually multiple bots good job however I wasn't quite done yet I


This is the worst AI ever

turned off the bots and I fixed some of the most claring mistakes I changed the code to


This is the worst AI ever

filter out these empty replies and I changed around some of the settings my plan was to


This is the worst AI ever

take a break for a day and then run for another 24 hours with the new settings interestingly


This is the worst AI ever

since all posts on 4chan are anonymous and since the criteria of replies that don't really


This is the worst AI ever

fit isn't the most well-defined concept in the world and it applies to many human posts


This is the worst AI ever

too people were still accusing each other of being bots well after I took all of them


This is the worst AI ever

offline which is quite interesting to see so after 24 hours break I let the now upgraded


This is the worst AI ever

bots loose again for another glorious 24 hours of mayhem now again there were a base of users


This is the worst AI ever

recognizing the bots for being bots there were still plenty of other users who didn't


This is the worst AI ever

and this even after I made a post on pole myself telling them that it was bots that


This is the worst AI ever

I was the creator and that I'm going to turn them on again and people were continuing to


This is the worst AI ever

discuss the phenomenon of the Seychelle account posting in so many places I mean look at this


This is the worst AI ever

one saying you can use a VPN to get around blocks and such it's not hard I know plenty


This is the worst AI ever

of people that do it including my mother saying the pattern is obvious they post the exact


This is the worst AI ever

same thing over and over I don't think they are anons but they are definitely a group


This is the worst AI ever

another user confirming they use the same talking points because they are all bots so


This is the worst AI ever

users were catching on but wait actually not not in this thread in particular actually


This is the worst AI ever

both the posts I've just shown you are just some other ones of my bots exposing the other


This is the worst AI ever

bots but you know bought stuff and look our tropical friend even had a meme made after


This is the worst AI ever

himself Seychelle anon glow so colorfully for reference a poster on 4chan is said to


This is the worst AI ever

glow if they're suspected to be a police officer I'm sorry to have to disappoint you I'm not


This is the worst AI ever

a police officer I'm not a fad I'm not a lefty I'm not hired by the World Bank or the Rockefellers


This is the worst AI ever

I didn't seek to achieve anything run a psyops or shill for anything and even though people


This is the worst AI ever

came up with all sorts of theories why the strange posts started what exact time I promise


This is the worst AI ever

it it just happened to be the day when I got done coding now typical 4chan fashion obviously


This is the worst AI ever

but half of you are not going to believe this after I let the new and improved bots run


This is the worst AI ever

for another day was all done I had made a total of over 30,000 posts in over 7000 threads


This is the worst AI ever

and I feel that's plenty and when you go right now to 4chan or its archive site for plebs


This is the worst AI ever

and search for the word Seychelles in Paul you'll find that people are still discussing


This is the worst AI ever

the user but also things like the consequences of having a eyes interact with people on the


This is the worst AI ever

site and it also seems the word Seychelles has become sort of general slang and that


This is the worst AI ever

seems like a good legacy for now like this one here saying just keep replying to data


This is the worst AI ever

mine threads train the AI and you're literally giving it new inputs to experiment with by


This is the worst AI ever

directly replying to the threads that somehow implies that you need to reply to the ball


This is the worst AI ever

in order to train it I'm afraid that's not how it works this one says I mean they have


This is the worst AI ever

templates for posts to bait you guys and it always works we're not we don't know templates


This is the worst AI ever

sorry all I know is that somewhere there is a Google document with a list of prompts to


This is the worst AI ever

bait users on X and pole this is the worst website in the universe I'm not even sure


This is the worst AI ever

I'm not a bot anymore so this was the video this was it I'm done this already took way


This is the worst AI ever

too much of my time and honestly I want to move on to more productive things the model


This is the worst AI ever

is quite vile I have to warn you so it's essentially the same as if you were to go to the website


This is the worst AI ever

directly and interact with users there although I was surprised that there's still a big gap


This is the worst AI ever

between actual users and the language model you know given by the fact that these people


This is the worst AI ever

determined pretty quickly that it must be a bot of some sort even though it posted anonymously


This is the worst AI ever

so needless to say for many reasons this model isn't ready to be deployed anywhere and please


This is the worst AI ever

don't try this at home lastly I've made another video this one's already too long in the other


This is the worst AI ever

video I've collected the most let's call it risky and adult interactions that the bot


This is the worst AI ever

had on the site now I draw they're not included in this video right here so I'll leave a link


This is the worst AI ever

to that video in the video description it's gonna be the first link in the video description


This is the worst AI ever

so check that out if you want to see something crazy all right that was it thank you so much


This is the worst AI ever


Weight Standardization (Paper Explained)

Hi there, today we're looking at weight standardization by Siwan Jiao, Huiyu Wang,


Xuanqi Liu, Wei Shen, Alan Yu of John Hopkins University. So weight


standardization is a normalization technique for training neural networks


and it goes basically in conjunction with another technique called group


normalization. So if you haven't group normalization norm, that is ugly. If you


haven't seen my video on group normalization and don't know what it is I


suggest you go watch that first or read the group norm paper or some blog post


because weight standardization is usually used together with group norm in order


to work well and that's what this paper also says. Even though it's pretty much


independent but here you can see their main results. So they compare batch norm,


group norm and weight standardization used with group norm then they can, as


you can see here, they can outperform in the ImageNet top one accuracy the other


two models and the important part here as you can see is batch norm is trained


with large batch sizes while group norm and group norm plus weight


standardization are trained with one image per GPU. So they have a multi GPU


setup and this is just one image per GPU and these results over here are on a


mask R-CNN which I believe is a recurrent model where the model is large


because the kind of the model is large and therefore you can only have very


small batches per worker and that means batch norm will work less. And again


we've discussed why batch norm is not a good thing when you have to go to small


batch sizes because basically what people have discovered is that it is


very beneficial in machine learning to normalize your data before working with


it. What do we mean by it? So if you have a bunch of data points right here and


let's say like this it is it is usually beneficial to first center the data like


this so basically calculate its mean and shift it and then to standardize the


axis so basically divided by the standard deviation in each direction and


your data will look something like this. Of many classical methods that will


improve the conditioning numbers of the requirements to solve it and so on and


even of deep learning methods we just know that if you standardize your data


like this it works better. So people are basically have come up with these


methods that where they say well if it helps for the data at the beginning of a


neural network then if after a layer the data is kind of out of whack that


can happen after a layer of neural network we should maybe first before we


send it to the next layer do the same thing center it again and then send it


through and if after the next layer again it's out of whack we should maybe


center it and standardize it again before sending it through the next layer. So in


each layer you have these transformations that center and standardize


the data and usually for the longest time this was a batch norm. Batch norm


does this across the mini batches of the data since you can't pass the entire


data set. Now group norm has come and replaced batch norm because in batch


norm it's very dependent on the batch size while group norm isn't. Now the


group norm paper has sort of made it clear that in like in competitive batch


sizes in the large batch size regime group norm is sorry batch norm is still


the king. Batch norm still works better it's only when you go to very small batch


sizes that group norm takes over and that's what you can see here. So here


okay it's a bit unfair because batch norm is trained with a larger batch size


but even if group norm were to be trained with the large batch size it


would still be in the same place because... no it wouldn't it would not sorry that


is that is not the case because the batches still influence the gradient


stochasticity and so on but still batch norm is better than group norm as you


can see here but here over here where you kind of have to go to the small batch


sizes then batch norm is all of a sudden worse than group norm and the weight


standardization is a technique to actually make group norm better than


batch norm in any of these so even in these in the large batch regime. Okay so


we'll now explore weight standardization. So in the group norm paper we've looked


at the diagram on the left so basically in batch norm here is the number of data


points this is your batch this is the channels of the batch of the individual


images channels and this is the height and width of the image so this is the


image itself a single channel so a single channel in the image would be a


column in this thing right here. Batch norm normalizes across the data points


in a single channel. Layer norm which is a precursor to group norm normalizes


only in a single data point instance but across all of the channels as you can


see here now that frees its dependence on the batch size each data point is


treated individually but of course it it sort of convolves all the channels with


each other it doesn't distinguish them. Instance norm tries to fix this instance


norm down here tries to fix this by saying it was a good idea to own to


normalize each feature individually and takes it to the extreme basically


normalizes a single image for by each of these single features but that loses too


much information. Group norm comes along and says maybe some of the features


naturally depend on each other naturally exhibit the same responses therefore we


should normalize them in groups so we take still a single image but we take


groups in this case groups of three channels together and normalize across


that. Now this here is all in data space this all normalizes the data like we


said up here when we drew this this is all normalizing the data before passing


it through the next layer. Now what actually happens in these layers so what


happens here what happens here in a convolutional neural network is that the


images get convolved with kernels that's how that's what a neural network layer


is so if you have an image right here of our trusty cat I've drawn whiskers in a


while that nose is very high the eyes must be like up here sorry cat and the


layer inherently has these things called kernels now I'm just gonna draw one of


these kernels right here it's a three by three kernel and what you'll do is you'll


slide the kernel across this right across like this you slide it across


across across across and for each point you convolve the kernel so you can


evolve the values here with the pixels here and sum them up and that for each


position in the image means that you'll basically get a new value at each point


and that will be your next layers data point now in these normalization


techniques we usually normalize the data points so here you have multiple channels


maybe a red a green and a blue and so on and the intermediate layers you have


even more and but you also have multiple kernels you can see here you have


multiple of these kernels which will then result in multiple output channels


the old normalization methods batch norm layer norm group norm they all work in


they all work in this or in this space in the space of data whereas weight


standardization works on the kernel space so weight standardization means you want


to normalize the weights of the neural network not the data and that's why it


can be used in conjunction with something like group norm or actually


batch norm or layer norm could be used with any of these but these authors use


it in conjunction with group norm so what does it do if you have these


kernels the kernels are of our characterized actually a kernel is


characterized by four numbers so first of all it's the height and width of the


kernel which in our case was three by three and is characterized by two more


numbers which is the C in in channels and out channels so the in channels is


the number of channels that come into the layer and the out channels are the


number of channels that you want to transform that into so here you can see


the in channels are listed here and out channels are listed here and in the up


down direction which is not labeled here is the height and width so this here


would be actually a two by two kernel so each of these slivers here is a two by


two kernel in the convolutional network and then that would be the orange sliver


here and then the sliver behind that would be the next two by two kernel


weight standardization says hi hey it might be as we normalize the data it


might be a good idea sorry I was that was wrong one column here one of these


columns is a two by two filter and then the column behind and and the column


next to it they're all two by two filters right so you have two by two


filters in the output or and you also have two by two filters for each of the


for each of the input output channel combination you have a two by two filter


so you have an entire matrix of two by two filters if you can imagine that so


across the out and across the in direction weight standardization says


hi it might be a good idea to see that the weights for a given output channel


right this is we take one output channel and we see all the filters that


transform the input into that one output channel which is going to be this many


times this many times this many numbers or this many filters maybe we should


normalize all of these to be sort of to not get out of whack because one could


imagine that during training right if we start we initialize our filters


somewhere here you know maybe one number this this one number here we initialize


it randomly right we draw it from random and then maybe as we train it actually


gets very large because it's actually plausible because after that we we you


know this is our neural network layer after that we have this procedure to


recenter the data right so I could make a very large weight here multiply the


data by very large weight because it gets recentered anyway but of course if


my weights get large I'll basically increase the variance and the


instability and the gradients might be high and and so on so these authors


think it might be a good idea to normalize these weights so just as you


normalize the data you normalize the weights and this actually turns out to


be fairly easy in the sense of how you would do it so instead of transforming X


which is the input to a layer into Y using W so this is W this is your


actual parameter using W you won't do this right now so this this was usually


just do you just do X times W and that gives you Y this is a convolution


operation right here now you don't do this you do you have to take W and first


you subtract the mean of W this is now for a single output channel and then you


divide by the standard deviation I mean this is standard deviation of W and that


entire thing you know multiplied by X now since these things here are sorry


about that since these things here are just you know deterministic operation


you can actually back propagate through it so the forward path of data now looks


as follows you come you start you say okay my data comes in I will take my


weights that my layer weights and I will first center them then scale them with


its standard deviation and then I will use that thing and X in order to obtain


my layer output and then I'll send that to the next layer now the back prop


signal here is interesting because the back prop signal comes in from here and


splits up in two way it splits up into the back prop signal basically you have


to back prop through the X times W hat operation we know how to do that that's


just a convolutional back prop that you back prop through the convolution


operation back to the last layer now usually when you back prop through the


convolution operation you get two things you get the derivative with respect to X


and you get the derivative with respect to the weights W and you can send both


on and you would update your weights with that gradient but now what you'll


have to do because this is not your actual parameter of the network you have


to take that particular signal and you have to basically reverse the


standardization and the centering before you can apply the gradient but that's


all doable the actually modern frameworks will do it by themselves but


it's just that the the back prop path here in it introduces two new operation


to the forward and to the back prop path that you didn't have before but I can


imagine this will actually not take you won't even notice that this is happening


this is so fast so they the idea is basically pretty basic especially since


the entire discussion around normalization has already happened I


enjoy that this paper does go into the theory a bit more so they analyze what


this weight standardization what effect it has on the Lipschitz constant of the


loss for example and they also research what what what contributes more the


centering of the weights or the standardization so they kind of run all


these ablations where they figure out okay if we just do group norm we have


one we you know we have this trajectory here and if we run group non-plastic


equation 5 which is subtracting the mean you can see the blue and orange that is


quite a bit and if we only do the dividing by the standard deviation you


can see it's pretty close together but there is a difference if you do both


then again there is a difference to only doing the centering so they they say


even though you know probably subtracting the mean gives you most of


the benefit since it is so easy you should just do both and I honestly think


and here in the in the validation error that makes basically no


difference at all and they do quite a number of these ablations which I'm not


gonna go into too much and they do also so the Lipschitz constant of the loss


and the Lipschitz constant of the gradients they basically show that the


loss and the gradients are behaved more more well behaved when you use this


weight standardization technique together with group norm they also do


quite a bit of experiments where they show that their method outperforms batch


norm and especially in the small batch size regime and that is something that I


absolutely believe what happened here okay I we actually don't even need to go


down there because if you want to read the paper I invite you to read the paper


it's a very good paper I enjoyed reading it but ultimately they suggest this new


method and also I have seen this one replicated across the community a number


of times so it seems to be a thing that I would expect either it fizzes out and


the community decides that it's about the same as batch norm and therefore not


worth it or and that's what I believe since we also go into the direction of


larger models which means smaller batches per worker and generally batch


norm is a pain I believe this is just going to be rather standard in the


future so I'll actually incorporate this if I can into my next projects so that


was it for me if you like this consider subscribing consider leaving a like on


the video thank you for listening if you have any comments I will very probably


Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Hi there, today we'll talk about switch transformers scaling to trillion parameter


models with simple and efficient sparsity by William fetus Barrett is off and no one


should see her of Google Brain. So as you can see right off the title, we're going towards


trillions of parameters GPT three had 175 billion parameters. This paper claims to have a model with


a trillion parameters. Now, is it really five times bigger or 10 times bigger than GPT three?


That's a debatable question, because the trillion parameters are not used in the same way as in a


classic transformers. They are used actually in a sparse way. That's why the word sparsity is in here.


And the way they are used in sparse manner is this new architecture called the switch transformer.


It's not entirely new, the it's built on mixture of experts. In this paper, that's also called mo e,


that has been around for a while, and we're going to see what that is. Now, on a high level, switch


transformers takes mixture of experts to an extreme, in that it is a transformer. And, and


the feed forward layer is divided up into these experts. And the switch transformer


only routes each token to one expert only, that's the sparse part. So the mixture of experts


previously, they always claimed you need at least two experts in order to get a stable training


signal, the switch transformer manages to get it down to a single expert. So it's like a hard


routing of information to just a single endpoint per layer of each token. So in that means you can


now scale the experts. And you can scale the number of parameters in the model without making


the model compute more. That's a very special notion. So you can up the parameters of the model.


But if a forward pass of a data point will still have the same amount of flops that it needs to


forward propagate through the network, very special architecture right here. So yeah, that's why I'm


saying trillion parameters not necessarily comparable to the 175 billion parameters of


something like GPT-3. So how do they do it? Because previously it was claimed it was unstable,


they have new ways of making the training stable, such as selective dropout, selective casting of


parameters to different precisions, and a better initialization. So that's the high level overview


of the paper. And we'll dive into it, we'll explore kind of what mixture of experts is and how the


model works. And what turns out, it's a very long paper, as you can see, when papers have a table of


content. That's a lot of fun. But it's a lot of engineering as well. And we're mostly interested


in the model here, what it can do, and what does it how does it sort of fit in to the big world of


transformers and language models and so on. Last thing I want to say, trillion parameters is, you


know, it's a catchy title, that most of the paper, they don't work with trillion parameter models,


they work with models in the in the order of billions of parameters. And at the end, they build


a model with a trillion parameters, it doesn't do as well as their models with in as their smaller


models. They also, it feels like they don't put that much work into it, because it's probably also


quite fuzzy and expensive. But just know, we're not going to have trillion parameter models around


anytime soon. Just yet. Interesting fact, the original ResNet paper also built a 1000 layer


convolutional neural network. Even though the ResNets we have today, you know, they are maybe


50 or 150 layers deep, they did build a 1000 layer model. So maybe compare it a bit to that one. It's


just like we can do it, not necessarily we need to. So here you can see something they discover.


The curve on the left is very, very known to people that are in the language model game,


let's say, or in the in the let's scale up AI game. And that is as you increase the size of the


model, the loss will go down. And that's loss as I understand it. So that's test loss. I believe


that is perplexity. So scaling properties, exactly that that might be perplexity or test loss on some


downstream task. In any way, as you scale up the model parameters, the model gets better and better


and better. The interesting thing right here is twofold. First of all, I believe they do hold the


data set constant. So the data set is always the same, the amount of compute you put into it, the


amount of either number of steps or time is also always the same. And in this specific case, the


amount of flops per forward pass is also the same. The only thing that changes is the number of


parameters. Again, it's very special to have a model where you can scale up the number of


parameters, yet the flops required to forward propagate stay the same. So you can see here that


there is a almost unhalted decrease here, it flattens out a little bit towards the bottom,


though that is not necessarily does not necessarily mean it will ever flatten out before it's, you


know, at zero, I will approach zero, I guess. So and you can you can see that, you know, they scale


up the model quite a bit. And also, their main comparison here is the T five base. So that's the


text to text transfer transformer. By the way, if you don't know what a transformer is, or what a


language model is, is best you go back to my earlier videos and look up like the GPT three


paper or the attention is all you need paper. I've made videos about lots of these things,


I assume that you know them. You can see right here that if you compare to number of training


steps, for example, the this switch models, all of them, no matter how big they are, they provide


massive gains over like something like a T five. And they also do this in time. So this paper is


very much about trade offs, you do require more storage for your weights. So you have to have more


memory more RAM. However, that memory can be distributed, it can be sharded, because they use


this mesh TensorFlow library to implement the switch transformers. And because their model has


this sparsity, they can efficiently shard the model. So you trade off more memory, which can be


sharded. But what you gain is training speed, in both in terms of time and number of training steps


required. So you are much more efficient. Note that this only all of this holds in this super


large regime, right? We this is, they say they've also discovered the speed ups in smaller models.


But you know, as far as the paper is concerned, we are talking about millions, hundreds of millions


of parameters, billions of parameters, even 2 trillion of parameters, together with these


giant corporate corpora of, of text. So that's sort of the regime we are in. And the results do


not necessarily transfer down to the lower scale problems that you know, you might face with your


lonely one, colab in the corner. Alright, so in a transformer, you have a transformer is nothing


else, but a bunch of these layers right here. This is this is in itself a transformer layer


in its basic form. And it consists of sort of two parts, it consists of this self attention,


right here. Now that's the standard transformer self attention, that's what was introduced in


attention is all you need. And what's been used ever since in all the transformers. This one right


here is a is an, as I understand it, a language model. So you know, this this is very standard.


However, after the self attention, you have this feed forward layer. Now, usually, what you do


is you have an input sequence, and you transform that through multi headed attention


into another sequence right here. Okay. And then what you do is you take each of these things and


feed them through a feed forward layer. And if I am, as I understand it, this feed forward layer


is simply, you know, a regular feed forward layer that you would find in a neural network,


and you pass them, you pass these things individually. So this here, it's a vector,


you pass it through here, and boom, that becomes the next layer representation, this thing right


here, you pass it through as well, boom, that becomes this one, and so on, right, you pass them


individually to get the next layer representation. So this, this part right here, the attention part,


it sort of aggregates information and relates the individual items of the sequence to each other.


And transforms them into, you know, a new sequence, where sort of all the every token can gather


information from every other token. That's what the attention mechanism does. That's step one. In


step two, every token is isolated, every token is for itself. And the feed forward layer simply


determines, you know, what's given one token given token number one, what is, you know, given its


representation in this layer, what is the best representation for the next layer? Okay. So that's


token number one of the next layer. So the multi head attention is kind of relating tokens to each


other, and the feed forward layers, they are relating layers to each other. Okay, so up here,


you would have the next multi head attention layer. So you can see the feed forward layer as sort of


translating from one layer to the next layer, right getting saying, oh, you come from this layer, I'm


going to translate you such that the next layer understands you. And that happens on a token by


token basis. Now you can see this is it's always the same feed forward layer for all the tokens,


right, the tokens are sort of treated like a batch of samples. The idea of this switch transformer,


and also of the earlier mixture of experts transformer is that it might not be a good idea


to have only a single one, right, this is the only feed forward layer, it's the same for all the


tokens, it might actually be a good idea to have a couple of them that sort of specialize in


different things. So what could that be? You know, in a in a basic world, this could just be like


one for nouns, and this could be a feed forward layer for verb verbs, tokens that are verbs,


tokens that are adjectives, and sort of maybe here is like punctuation tokens, right? You might


think, well, if you are a noun token, the next layer might want to look differently at you,


than if you are a punctuation token, right? So this translation from one layer to the next layer


can now happen dependent on what the token represents. Right? Now we we of course, first of


all, we don't have these annotations. And second, is not necessarily that you know, we want to


always divide it by noun verb, adjective punctuation. Ideally, we want to learn this routing.


So we simply want to say, look, instead of just one feed forward layer, we give the model a


feed forward layer. Give the model four feed forward layer, feed forward layer one, two, three,


and four. And for each token, the model can decide to which of these feed forward layer it sends the


token to. So here you can see this is a token. Now, you know, we are dealing with word pieces,


let's just say the word more, I was like, I was thoroughly confused by when I saw this like, huh,


why does it say more parameters, but here, it's the string more right, and the string parameters.


And these are in the vocabulary, and they get an embedding vector associated with them. So that's


what's going on here. Then they go through self attention, as you can see here, both go through


self attention, and then each one of them is routed to one of these four experts. Now the


one here, the one on the left and the one on the right, these are the same experts, right,


they're just duplicated visually here, but these would be the same weight matrices in there. So you


have four feed forward layers in this layer. And each token can be routed to any one of them. And


this routing here, this is learned. So in here, you have a matrix, they call it like wr. And using


wr, you simply do an inner product of wr with your input right here, let's call that h, with your


input h, I guess they use h for a different thing. I think they they call this x again. So you do


this with x. And then you get, you get h, which is your routing. And then you simply build a


histogram, you normalize the histogram, I think with a softmax. And that those are your routing


weights. So it's very much like another attention mechanism, except that the queries,


this thing here, these are like the queries, these are sort of the queries of this attention


mechanism. And this here, these are the keys and the values. So that's a good keys and the values


of this attention mechanism. The queries are just learned. So the queries are not dynamically


generated. And the keys and values, they are not. Yeah, it's a it's a weak analogy. But you can sort


of think of it like this. So there is this routing mechanism. And it decides where a token gets goes


to. Now, as you can see, the router is soft, that means there is never a one or a zero right here,


there's always kind of a number in between, but they hard clip that. So they hard clip it,


they just route it to the maximum. As you can see here, number two is the maximum. And they just


route it to number two, they don't route it proportionally or anything, they just to take


our max and they route it through, they do multiply the output by the actual number that they got out


here. So if the router is unsure, then the output is less. If they're out, is sure the output is


more. But this hard routing is what's the key right here. And that means, you know, before,


before, you'd have one feed forward layer. So any token that goes forward goes through one feed


forward layer. If you do a mixture of experts in the classic sense, and you route it in a soft way,


you now have four feed forward layer. So every token goes through four of these computations.


So you've basically multiplied the amount of computation by four, because you've multiplied


the amount of parameters by four, right, you have four times as many parameters. Now, when you do


this arg max routing, like the switch transformer, you have multiplied the number of parameters in


your model by four, but any token will still only incur one feed forward layer. That means you keep


the amount of computation that you do per forward pass the same. And that's, that's sort of the key


right here. So now they can scale up massively the number of experts, while still keeping the amount


of flops the same. And notably, you also don't need a any data transfer in between the experts.


Every every expert can be can, you know, receive their tokens and then do their independent work.


So you can efficiently shard this across many, many machines. This is how this looks. So in in


this case, you have three experts and your sequences are of line of length six. So you want


to sort of route each token there and there can be overflow, like every token is independently


routed. So it can happen something like this, that a, you know, a token like three token gets routed


to one expert, but it only has space for two tokens. And they have some tricks like they have


this capacity factor right here, or they can reroute. These are very much engineering things,


which are important, but you know, they don't change the sort of final, final result. Now,


I want to go down here where they have a display of this sharding more like an explanation of the


sharding, which I think is very illustrative. So how, what do they essentially do? If you think of


many machines, you have 16 machines. So each little square here is one machine. Okay.


Here are the different ways of how you can shard a model and model sharding. Now we are not going


to build a machine anytime soon that can hold a trillion parameters just not going to happen. Okay,


so you need to somehow shard the model or the data or both. And these are the different ways how you


can do it. So if you use data parallelism, that is the easiest that is also directly built into


things like pytorch and so on. What you do is, so the top row shows how the model weights are split


and the bottom row shows how the data is split. So how to read this is when you do data parallelism,


the weights are split such that each of the 16 cores has the same weights, you see, so this,


these weights right here are the same as these weights are the same, they're all the same. So


this is sharded, the data is run, so that you take a data set, you take a batch of data. And now you


distribute this data point goes here, this data point goes here, this data point goes here, and so


on. You distribute the data, and you do the forward propagation. And at the end, you sort of gather


them again, right? So you gather them together. Again, because you have to, you know, calculate


your gradient. Okay, so that's data parallelism, the model is spread out. And if you want to do


an update to the model, then you need to communicate around these weights. Okay, so all


these different pieces have to then communicate with each other when there's a weight update.


If you do data parallelism, here is how the data split, we've already seen this. So one piece,


this piece of data is split over 16 cores. So you can see like this core right here only has this


little piece of the data, and not all of the data. On the other hand, you can do model parallelism.


In model parallelism, you can see it's exactly the other way around, namely that one core only has a


little piece of model, right? And, but every core gets all of the data. So this data here, the bottom


row is data, all of the data. The point here is that if you do model parallelism, that's what you


do when the model itself doesn't fit right over here, the model fits on your machine, but not the


whole batch at the same time. Model parallelism you do when the model itself is not the same.


You do when the model itself doesn't fit, what you have to do is you have to take your data,


right? And you have to send it sequentially. So maybe this is the first layer, like that's layer


one weights. And then you have to compute layer one, and then you have to send it to layer two,


and so on. So you have to send it sequentially through the through the sharding of the model,


right? Because you want to forward propagate through all of the model. This is has very, very much of


a cost of communication, you can build very big models, but it comes at a cost, right? At the end,


you get your y and you calculate your loss and you back prop again, backwards through the whole thing.


You can mix them, right? You can do model and data parallelism. So here you can see that the weights


so this is this is layer one weights, layer two, layer three, layer four. And here again,


you have layer one, layer two, layer three, layer four, and so on. So you can mix the two in that


you can have model and data parallelism, if both your model and also your data don't fit


in a single machine. And you can see here that the this upper left part receives, they receive


the same data, but this here receives different data, right? So you split your mini batch into


four different parts. And you send the first part up here, like that's data one, you send that up


here, and that goes through the model in this sequence sequential fashion, you send data to


right to here and so on. So we mix the two. Now, in expert and data parallelism, that's what they


that's what they do in the switch transformer. So this here is the switch transformer. And this here


over here will then that's the switch transformer 1 trillion. So for the 1 trillion model, they


actually need to mix all of them. But you want to add, you know, if you can, you want to avoid


model parallelism, model parallelism is really the thing that kills you because of the very high


communication cost. So in the switch transformer, they have expert and data parallelism. What does


it mean? So the top row is how the model weights are split. And you can see the weights are split,


but the different color means that they're different weights. So here are weights number one


weights, two, weights, three, weights, four, and so on. Now, we've already had this over here,


right? Different weights in the model parallelism case were split over different machines. However,


if you look at the data, the data is also split, and the weights, they're not the same. And these


are exactly these experts. So experts. This means that, you know, this piece of data here


only goes to this expert, and then to the output, this piece of data right here only goes to this


expert, and then to the output, right, there is no communication between the different experts,


whereas here you have this super high communication. Okay, so you can see you can scale up the experts


as you scale up your data, as long as each shard of data is routed to only one expert. And then,


of course, you can mix the expert model and data parallelism. If you really if not even a single


expert fits on a machine, right? If that's the case, you need to again, shard you do model sharding


on the experts. Alright, so the switch transformer, as I said, this here is the switch transformer


that the most of the paper is about. And now we can dive into the results. The results are pretty


spectacular. They mostly come compare, as I said, to T five base, and T five large. And as you can


see right here, the switch model has significantly more parameters. So 7.4, or here 26 billion


parameters compared to not even a billion of T five large, yet the number of flops is matched. So


they build models where the number of flops for forward prop is matched. But the the number of


parameters are higher. So, you know, it is somewhat of a fair comparison, right? You have the same


amount of compute done per forward prop. And now we see what does it help to just have raw gain in


parameters. And it turns out it helps a lot. You've probably already seen that we get these massive


speed ups, massive sample efficiencies over a dense model. You've probably so this we've looked at


exactly in the in the intro, they also have benchmarks on, let's see this down here, they also


have benchmarks on multilingual on multilingual data set. And you can see in every single language,


the switch transformer gains on the dense transformer by quite a bit. So this is in this is


lock space, as you can see, and it's quite impressive, actually. And these gains are in time,


as well as number of steps. So that's pretty, pretty cool. So, as I as I said, the the trade


off here, of course, is that you need more machines, you need to actually add more machines.


And you can see this largest model that they built is this switch XXL, which is matched in flops to


transfer to the T5 XXL model, yet has many more parameters and beats the T5 at log perplexity and


as I understand in downstream tasks by quite a bit. They also built this trillion parameter model.


It is not as good, mainly because they, as I understand it, they just want to get to a trillion


parameters. And I think I think it's you know, training isn't really easy at that size. So they


scale it down, as you can see, it has less number of heads, less number of layers, but the number


of experts are way up. So that's how they scale to a trillion. And the results are, you know,


better than the T5 XXL, which is impressive, given that it has less flops per token. However,


it is still worse than the switch XXL. So the trillion parameter model, it's still, you know,


it's still not everything to have a lot of parameters, you actually need to do good trade


offs. And here they've traded off too many parameters for, you know, less number of heads


and less number of layers. And that hurts again. So very, very interesting stuff right here. The


last thing I want to look at is their tricks for getting this to work. So they detail three tricks


for getting this to work. And they are right here, three tricks, how they can do this. And people


before them have said, no, you need at least two experts, otherwise, it's unstable. So they do


selective precision with the large sparse models, which means that if for some of these computations,


it, you know, it, it pays off to do them in higher precision, you don't want to send around


these flow 32 precision things, you don't want to send those from machine to machine, right? So you


have your input, you have your multi head attention. And then here, again, this is whatever x prime,


and then you send that to the experts. Right, here are the different experts. And then you send that


you send that back. And that's why, okay, now, you don't want this here is communication cost.


If you were to send around float 32 vectors, that's a lot of data that you have to transmit.


So you'd rather send around 16 bit precision, right as they do right here. And however, if you


do 16 bit precision, you're you know, the whole machine learning part doesn't work as well.


So what they do is they do as soon as it as a, as soon as a vector arrives here, this is in 16 bit,


they scale it up, they cast it to a 32 bit vector, they calculate using the 32 bit vector


32. And then they cast it again to a 16 bit vector to send it back. And that seems to work. So they do


selective, selectively casting the precision up. And also they do selective dropout that's down here.


So they do expert dropout, which means they don't apply dropout to the whole network uniformly,


as you would do regularly normally, but they say they can do a much larger dropout rate at expert


layers. And that makes a bit of sense because the expert, each expert is only used very sparsely.


So it makes sense to up their dropout rate. Because you know, in the end, you might drop out


as much signal from a sparsely used expert, if you raise the dropout rate, then you do from a densely


used layer in a with a smaller dropout late rate. And the last thing is that they simply do better


initialization. So they find if they scale down the the initial scale of the original transformer


by a factor of 10, that leads to a lot more stable training. It's astounding that after so many years


still something like initialization can, you know, make or break such a model that is just


insane to see. There's a lot more to this paper, they do a lot of downstream tasks, they also talk


a lot about, you know, this is not only this model, they do a lot of optimizations under the hood,


they use mesh TensorFlow, and so on. It's clear that a lot of work has gone into this. And


interestingly enough, they can also distill these models. So what they can do is they can take this


large model and they distill it to a model that is as big as t five base, a dense model. So they go


from a sparse large model, and they distill it into a dense model that is equivalent to t five.


And they do outperform t five, if it were trained from scratch, and they gain up to something like


30%. So 30% of the gains they made from here to here, they can retain by distilling it down.


They say they can distill it down way over 90 95% of the model, which is also pretty interesting.


And, you know, pretty cool. Because then you could sort of distribute the trained models around and


people could use them. Alright, so that was it for me, definitely check out the paper and all the


experiments downstream tasks and so on. It's a very cool paper has a lot of cool experiments.


Optimizing Neural Network Structures with Keras-Tuner

What is going on everybody and welcome to a tutorial slash showcasing of the Keras tuner package.


So one of the most common questions I get on deep learning tutorials and content in general is people asking


how did you know to do n number of layers or why n neurons or why did you do dropout, why did that degree,


why batch norm, all these things, like why did you do that?


And the answer to that question has always been trial and error