current research to be able to read current research papers. And the fact


WHO ARE YOU? 10k Subscribers Special (w/ Channel Analytics)

that I have quite a number of people watching from countries where top


universities maybe aren't located is very encouraging to me. Thanks for


watching from all around the world. One person is watching this with Russian


subtitles. OK we can go into the advanced statistics here and that is


pretty interesting as well you see here most videos kind of spike when they come


out especially the news videos they tend to be very popular just the moment that


I release them and then they sort of fall down. Traffic source I find


particularly interesting if you look at the last 90 days there are these spikes


and these spikes tend to come mainly from from Google searches. So at some


point people simply Google search for stuff and then they find this channel


which is encouraging. Most people actually search for attention is all you


need or things like this. YouTube doesn't show you anymore what people search on


Google but YouTube shows you what people searched for on YouTube. And that tends


to be mostly attention is all you need. Mu zero my name. Hello. So if you


correlate these spikes in searches with geography some of these spikes tend to


be worldwide like this one here or this one here. But this particular spike is


only United States. And if you look at the videos that I released during that


time period one of these videos right here. So either it is the Schmitt Uber


drama a lot of people searching for that maybe maybe it's ImageNet v2 or the


online conferences. I don't know. You can see right here I didn't make many


subscribers of these spikes. It's simply people being interested in content which


is pretty cool. It is also interesting to see that these spikes right here they


correspond mainly to mobile phone users. So mobile phone users go and Google


searching for content on this channel. I have no idea what's going on. Alright,


now the last question to solve is of course monetization. How much money does


this channel make? And the answer is none so far. So there are multiple reasons


why I haven't applied for monetization yet. I find YouTube ads just incredibly


annoying, especially now that they've decided to stick to ads in front of


videos. I just don't want to bug users with that. If you look at what I gain


from it, it's not that much any money that I would make I would like to sort


of reinvest into the channel and right now I just don't have any requirements


that might change in the future maybe once we get to 800,000 subscribers. All


right, that was it for YouTube analytics of this channel. I hope you are still


enjoying this content. If you're not subscribed, please do. Next update will


be at a hundred thousand and I hope that everything is as enjoyable as ever. Thank


you for watching. Thank you for subscribing. Thanks for being here and to


STOCHASTIC MEME DESCENT - Deep Learning Meme Review - Episode 2 (Part 2 of 2)

At some point I will be able to code you, Janek.


You will be able to?


To code you.


To code me?


Yes, so that finally you will release videos in time.


Random guessing, my classifier.


47% accuracy.


Nice.


Yes.


If you change the seed you can get 48.


Ha ha, you'll never reach me.


Yes, I will.


Wow, by coming up with a better algorithm?


No, by using a weaker baseline.


Getting published is so easy.


It's a job.


Yeah.


Janek.


Do you, sometimes I realize that, you know, my life every three months is going to be


like a deadline.


Is this real life?


This is it.


It doesn't get better.


Is this the piece?


Like some years ago I thought it was going to be fun, you know.


You just enjoy the life.


You just have nice conversations.


You try your best.


You think about things like for a long time.


When you finally, no.


That does not sound like machine learning research.


Okay.


Two things we don't have.


Long times and thinking.


Model overfits on training data.


Word, new data.


I got one paper rejected because the review was like, where is Cypher?


Where was the review?


Where is Cypher?


Where is it?


Where is it Antonio?


If there's no Cypher, how should I know?


How does any paper get accepted without Cypher?


It's called Cypher.


I don't know.


Maybe it's called Cypher.


It's like an abbreviation of something.


People who study Latin will call it Cypher.


Social distancing guidelines.


COVID-19, 1.5 meters.


Meaning out?


That's very true.


I'm having something like that to deal with right now.


I think I forgot something.


If you forgot, it wasn't that important.


Yeah, you're right.


This could actually work, you know?


Aren't there these proofs that some of these algorithms only converge if you average over gradients?


Yeah.


If you accumulate your gradients technically with a decreasing learning rate, this might work.


Yeah, Nick. It's all wrong.


Yeah, that's exactly how it's done.


But what's the story behind this?


There's no story.


I'll just give you a minute.


I didn't get it.


Should I really?


I should really calculate.


It's true, right?


It's true.


Yeah, it's actually true.


Okay.


This actually works.


I thought like, okay, yeah, it's Saturday.


I woke up two hours ago.


It's actually true.


Dick move now.


Wiener process.


Yeah.


Beautiful.


Douchiness.


Douchiness, it's a word.


I didn't know.


Epsilon is expected to grow very large over the next 48 hours.


No.


It has to be small enough.


Enough.


Small enough.


Abstract.


Introduction results.


I was glad.


Did I tell you this?


Maybe it was also in the other review.


There was a paper.


It's mine.


That's my paper.


But I remember it was like in this paper, in this specific paper, where was this?


Okay, we prove that this is true.


And in the introduction, it was like sometimes, like the same thing.


Sometimes we show that sometimes under some assumption, and then you read the paper, it's


actually just an example.


Not everyone should code.


Recommended for you.


I'm surprised that sometimes I look at the thing and I will never enjoy it.


And then I do.


Us YouTubers, we have to regularly sacrifice GPUs to the algorithm.


Yeah, it really likes GPUs.


Do you have to burn them?


Do you have to make them burn?


You have to take like some cooler liquid and sprinkle it on top.


And if you dance around it and some flowers on top of it, and then you have to eat it.


OMG, I love all this water cooled CPUs.


New toothpaste exists.


Dentists.


I didn't get the machine or anything.


There's no machine.


Okay, okay.


Yeah, perfect.


Perfect.


I love this.


I don't know why, but it's so good.


Janek, that's the big surprise.


At the end of this video, there's going to be a big surprise.


What?


It's a citation from the office.


Okay, but yeah, seriously, for each one of you, Janek is going to make a gift.


Is it the MATLAB license?


Damn, it don't spoil.


Forms of birth control.


TensorFlow.


I should just put machine learning.


When your model improves from 5% accuracy to 7% accuracy.


Machine learning.


Machine learning finding global minima.


Machine learning finding local minima.


Yeah, that's so damn true.


Theory people are weird.


Theory people are the worst.


Weird, weird.


That's even true.


Like, but I'm completely serious.


100% serious.


Like they get excited about infinitely wide neural networks.


Oh, yeah.


Or what if you take the step size to be infinitely small?


Yeah.


That's how you do things.


I mean, the only thing that's infinitely wide is your mom.


Self driving cars aren't even hard to make lol.


Just programming not to hit stuff.


Don't.


You know, in all of my code, true story, in all of my code, I write in a line and it's


usually like a common to doubt but I write in a line that says if target equals Janek,


then don't fire.


Really, just I anticipate that some of my code will be used in the robot overlord army.


Yeah, that's such a smart move.


I know.


That's such a smart move.


You got to think ahead.


For some reason, they will shoot everything except the traffic lights.


How?


Interviewer, what's your biggest strength?


I'm an expert in machine learning.


Oh, good that we did this this way because the other way would have been a bit strange.


Okay.


What's nine plus 10?


It's three.


Nothing close.


It's 19.


It's 16.


Wrong.


It's still 19.


It's 18.


No, it's 19.


You're fired.


I wonder what GPT-3 would say to this.


What should we try that?


We should try it out.


When you drop the learning rate.


Everyone is so everyone's like freaking out what happened here, but they dropped the learning


rate.


So clear.


It's like, that's what you do.


You stagnate.


You divide it by 10.


Shaka boom.


I'll give you 10 seconds to copy what's on the whiteboard.


The whiteboard.


It's actually from my video.


Yeah, I kind of remember something similar to that.


What was this?


I have no idea.


Not a slightest clue.


So this actually is also on my video.


They really tried.


But sometimes I mean if I make a mistake on a video or something, I'll put like a comment.


You never make mistakes.


Before before I set the video to visible.


It's just so mean to the people who want to do this.


Mom, if your friends jumped off a bridge, would you jump too?


How much time I needed this meme and I didn't know I needed that.


No, you can't just add more parameters and data to model GPT-3 is no different from ELISA


since it's just glorified pattern matching and curve fitting not true intelligence which requires a symbolic representation of the input which connectionist models will never be able to do.


Also the data needed is almost an entire percent of the total possible data we can collect from current crowd alone and the hardware needed to train GPT-3 is on Venmo if you ask me.


I'm gonna puke.


Do you think GPT is intelligent?


I think he's aware and he oh my god.


No, no, no.


Oh, we're gonna leave this in.


I'll crush us.


Do you think GPT-3 is intelligent though?


I think well, I like the colors.


So I like the colors of the GPU there.


Nice.


Anybody with this colors is like slightly, you know, funny.


So we can be funny, you know, but not intelligent.


Do you think?


I think it is.


It's not?


It's...


It is.


That'll be canceled for like the 50th time.


Researchers hate him.


Local man discovers one weird trick to general intelligence.


Turns out you just weren't using enough layers.


Learn the secret to his stunning result.


Learn the truth now.


Yep.


Yes, that's again me.


That's again me.


Own it.


The stickers, the stickers, they own it.


And that is probably the Adam paper.


You know, the Adam proof is famously wrong.


I very much know.


Oh, yeah, yeah, I do.


I just heard it.


I just repeated to sound smart.


No, I know it.


I know it.


It's like there are at least four mistakes in that proof.


And I think that it got probably like 30,000 citations before,


before realizing that it was...


It's still getting citations.


No.


Now, you know, the second part of the story, well, now it's 60,000.


The other paper, the paper that fixes the mistake,


introduces an AMS grad.


The proof, the mistake, basically the V variable.


Yeah.


Then it's a problem for the proof.


Okay.


And AMS grad fixes the mistake.


But now there's another paper that tells that actually Adam does converge.


So we go back to the fact.


No, no, guys, I just did it wrong.


It just did it wrong.


But yeah.


It's like when you don't use the method your teacher wants you to use.


Exactly.


But nobody used AMS grad.


Nobody ever used it.


No.


I spit on AMS grad.


I really don't like it.


Albert Einstein.


Insanity is doing the same thing over and over again and expecting different results.


That's how I make papers.


Come on.


Conceived equals to or maybe like resubmission.


How it started.


Yeah.


Against the mob.


This is a very dark period.


How it's going in the channels.


This is where the superstar right here.


We don't we don't we don't talk about this.


No, no, we don't.


We don't talk about this.


That's nothing happened.


Nothing happened.


And we get new AI be like.


That's what they do.


And you might have any millions of dollars are going into just making your eyes go crazy.


You forgot.


Three.


All right.


That was it for me.


Review.


Thank you so much for watching.


Thank you.


I want to thank you for having me here.


It is always a pleasure.


Yeah.


And hopefully 2021 will have also cake.


Yannick.


Where the hell is the cake?


More cake.


Bye bye.


Why Python 3? Python 2 vs Python 3

Hey everyone and welcome to my why Python 3 video.


Why Python 3? Python 2 vs Python 3

This video is also going to serve as the first video in my Python 3 programming tutorial


Why Python 3? Python 2 vs Python 3

series though there will be nothing more than just information on Python 3 versus Python


Why Python 3? Python 2 vs Python 3

2 in this video and why you should choose Python 3.


Why Python 3? Python 2 vs Python 3

Video 2 will begin with the downloading and installation of Python 3.


Why Python 3? Python 2 vs Python 3

Also I'm going to mention here and again later on in the series that this series is going


Why Python 3? Python 2 vs Python 3

to be geared towards the complete noob for Python.


Why Python 3? Python 2 vs Python 3

So if you're used to Python 2.7 already making the transition to Python 3 is really not that


Why Python 3? Python 2 vs Python 3

difficult.


Why Python 3? Python 2 vs Python 3

I will probably make a Py2 to Py3 transitional video after I finish this series but that's


Why Python 3? Python 2 vs Python 3

probably a little bit away.


Why Python 3? Python 2 vs Python 3

So on with the video.


Why Python 3? Python 2 vs Python 3

Why Python 3?


Why Python 3? Python 2 vs Python 3

So throughout my time with Python programming I've visited and revisited the idea of Python


Why Python 3? Python 2 vs Python 3

3 countless times.


Why Python 3? Python 2 vs Python 3

When I first started that's exactly what I did and I've always come up with the exact


Why Python 3? Python 2 vs Python 3

same answer every time.


Why Python 3? Python 2 vs Python 3

Python 3 on the whole is not better than Python 2 yet it has less support documentation and


Why Python 3? Python 2 vs Python 3

libraries.


Why Python 3? Python 2 vs Python 3

Basically all of the good things in Python 3 for the most part, not all of them, but


Why Python 3? Python 2 vs Python 3

a lot of them have actually been ported backwards into Python 2.


Why Python 3? Python 2 vs Python 3

So more on why that must be really frustrating later.


Why Python 3? Python 2 vs Python 3

But anyways a few months ago I watched a YouTube video of a presentation about why one should


Why Python 3? Python 2 vs Python 3

choose Python 3.


Why Python 3? Python 2 vs Python 3

It was something like trust, you know, why Python 3, trust me it's better.


Why Python 3? Python 2 vs Python 3

Something like that, right?


Why Python 3? Python 2 vs Python 3

And it was given by an actual developer for Python 3.


Why Python 3? Python 2 vs Python 3

Through the entire presentation, now given it's probably just the case that because I'm


Why Python 3? Python 2 vs Python 3

not an expert or a professional, I guess professional is a bad word, I am a professional but I'm


Why Python 3? Python 2 vs Python 3

not an expert.


Why Python 3? Python 2 vs Python 3

Through the entire presentation I was still left not really seeing any reason based on


Why Python 3? Python 2 vs Python 3

that presentation and what was said in that presentation I saw no reason to switch, right?


Why Python 3? Python 2 vs Python 3

But instead I could really sense like the frustration in this guy's voice, right?


Why Python 3? Python 2 vs Python 3

The lack of adoption for Python 3 is actually an interesting topic of discussion, right?


Why Python 3? Python 2 vs Python 3

The story of Python 3 could make like a nerdy Greek tragedy movie or something, seriously.


Why Python 3? Python 2 vs Python 3

Now this stuff said, you know, I'm not suggesting people move to Python 3 out of pity.


Why Python 3? Python 2 vs Python 3

That's not a very valid reason or a good idea at all.


Why Python 3? Python 2 vs Python 3

But there are some good reasons why you should move to Python 3.


Why Python 3? Python 2 vs Python 3

See Python 3 is really the future.


Why Python 3? Python 2 vs Python 3

Python 2 is not.


Why Python 3? Python 2 vs Python 3

If you stick with Python 2 and Python 2 only and everybody does this, you're gonna be able


Why Python 3? Python 2 vs Python 3

to get away with it for a while and then you'll find yourself using Python 2 in a time where


Why Python 3? Python 2 vs Python 3

it's comparable to using something like Pascal today, right?


Why Python 3? Python 2 vs Python 3

At the same time Python itself will probably fall behind as a developmental language or


Why Python 3? Python 2 vs Python 3

a programming language just simply due to the lack of support and community overall.


Why Python 3? Python 2 vs Python 3

You know, Python in my honest opinion is simply the best language out there.


Why Python 3? Python 2 vs Python 3

It's very high level and yet it has incredible power and due to its high level nature it's


Why Python 3? Python 2 vs Python 3

also very simplistic.


Why Python 3? Python 2 vs Python 3

It's just a great language and it would be a shame to really see it become outdated just


Why Python 3? Python 2 vs Python 3

simply because the community was disinterested in moving forward with the language.


Why Python 3? Python 2 vs Python 3

And I really can't imagine how annoying it must be for the Python 3 developers, you know,


Why Python 3? Python 2 vs Python 3

that people just simply aren't adopting the new language for not too great of reasoning


Why Python 3? Python 2 vs Python 3

in my opinion.


Why Python 3? Python 2 vs Python 3

So in the end the deciding factor for me to not adopt Python 3 was the lack of documentation


Why Python 3? Python 2 vs Python 3

and support, right?


Why Python 3? Python 2 vs Python 3

It was when I was a noob to programming in Python and I really needed that support just


Why Python 3? Python 2 vs Python 3

as anybody new to any language needs some sort of, you need the community and the support


Why Python 3? Python 2 vs Python 3

in order to progress in that language, right?


Why Python 3? Python 2 vs Python 3

And I looked at both languages and to me they seemed equal, right?


Why Python 3? Python 2 vs Python 3

Equal but the community surrounding them was very different in size.


Why Python 3? Python 2 vs Python 3

So it seemed to make sense for me as a noob to go with Python 2.7 and I believe that's


Why Python 3? Python 2 vs Python 3

a logic, I think that was a very logical choice.


Why Python 3? Python 2 vs Python 3

And so anybody else that's doing the same thing, I completely sympathize with why that


Why Python 3? Python 2 vs Python 3

is the choice at the moment.


Why Python 3? Python 2 vs Python 3

Now that I'm someone that, you know, has had this channel and I'm making all these videos


Why Python 3? Python 2 vs Python 3

on YouTube, I seem to be doing something right enough to where at least a decent amount of


Why Python 3? Python 2 vs Python 3

people enjoy how I teach and I think it's almost somewhat of my responsibility to share


Why Python 3? Python 2 vs Python 3

and teach Python 3 at this point.


Why Python 3? Python 2 vs Python 3

This way maybe in the future when, you know, someone does a search for Python 3 versus


Why Python 3? Python 2 vs Python 3

Python 2.7, maybe they'll find my video or maybe they'll find stuff from people that


Why Python 3? Python 2 vs Python 3

I helped get started or, you know, whatever and Python 3 can actually start to grow, you


Why Python 3? Python 2 vs Python 3

know, because for the most part right now when you hear a Python 3 versus Python 2 discussion,


Why Python 3? Python 2 vs Python 3

the differences are really only geared towards, I would say, the expert or at least professional


Why Python 3? Python 2 vs Python 3

Python developer but I would argue most professional Python developers couldn't care less about


Why Python 3? Python 2 vs Python 3

these tiny differences.


Why Python 3? Python 2 vs Python 3

And then the main question of performance is, is Python 3 or Python 2 faster?


Why Python 3? Python 2 vs Python 3

They are pretty much equal all the way down.


Why Python 3? Python 2 vs Python 3

So there was in that, in the video from the guy, trust me Python 3 is better, I forget


Why Python 3? Python 2 vs Python 3

the title but in that video he does compare, you know, the performance and depending on


Why Python 3? Python 2 vs Python 3

the task, sometimes Python 2 is faster, sometimes Python 3 is faster, the differences are so


Why Python 3? Python 2 vs Python 3

minute that performance isn't the issue so it really comes down to a few just tiny differences


Why Python 3? Python 2 vs Python 3

that at the moment are so tiny that it just doesn't make sense but in the future, if Python


Why Python 3? Python 2 vs Python 3

3 is allowed to continue growth and development, Python 3 will be the victor but if obviously


Why Python 3? Python 2 vs Python 3

no one supports it then that sort of stuff is never going to happen and you won't have


Why Python 3? Python 2 vs Python 3

modules and all this kind of stuff.


Why Python 3? Python 2 vs Python 3

So back to, you know, a video like this guy though, you know, by this time, you know,


Why Python 3? Python 2 vs Python 3

that video is kind of geared towards at least, you know, most likely an expert and by that


Why Python 3? Python 2 vs Python 3

time that, you know, Python 3 makes sense to the expert, the developer has already gone


Why Python 3? Python 2 vs Python 3

through their new stage and probably did the whole Python 2 versus Python 3, decided Python


Why Python 3? Python 2 vs Python 3

2 was the best start and then got ingrained in their ways, right?


Why Python 3? Python 2 vs Python 3

And then it's very hard to get them to switch and a lot of people don't see a reason to


Why Python 3? Python 2 vs Python 3

switch.


Why Python 3? Python 2 vs Python 3

So here, my goal is to offer a full slew of beginner and partially intermediate Python


Why Python 3? Python 2 vs Python 3

tutorials all using Python 3, right?


Why Python 3? Python 2 vs Python 3

So my Python 2.7 tutorials will remain here, I'm not going to delete them or anything but


Why Python 3? Python 2 vs Python 3

I'm going to be doing my best to offer the same topics only in Python 3 and some modules


Why Python 3? Python 2 vs Python 3

that are pretty popular in Python 2 still aren't available in Python 3 so we can work


Why Python 3? Python 2 vs Python 3

on at least probably covering at least one of those modules as far as like how you could


Why Python 3? Python 2 vs Python 3

port them because really porting them isn't that big of a deal.


Why Python 3? Python 2 vs Python 3

Some of them are harder than others but anyway, this way, hopefully you'll be able to get


Why Python 3? Python 2 vs Python 3

through that new and intermediate stage of learning Python and you'll be able to do it


Why Python 3? Python 2 vs Python 3

with Python 3.


Why Python 3? Python 2 vs Python 3

So with that, let's go ahead and start Python 3.


Why Python 3? Python 2 vs Python 3

As always everyone, thanks for watching, thanks for all the support and subscriptions, and


Why Python 3? Python 2 vs Python 3


Author Interview - Typical Decoding for Natural Language Generation

Hi, y'all. This is an interview with Clara Meister, who is the first author of the paper


Typical Decoding for Natural Language Generation. This paper, I believe, is really important


because it presents a new sampling method that makes language models output much more


human like texts. I've already made a review about the paper. If you haven't seen that


yet, check it out. Clara has seen it and we're able to dive directly into the matter. This


interview was very cool. I learned a lot. As always, if you like, leave a like, tell


me what you think in the comments and I'll see you around. Bye bye. Hey there, today's


sponsor is the course on Introduction to Graph Neural Networks. This is a course by my friend


Zach Joest, who is an expert in graph neural networks. He's packed all his knowledge into


one course that will educate you on both the theoretical and hands on practical aspect


on graph neural networks. Graph neural networks are really important. They're definitely one


of the most interesting areas in deep learning right now. They've also powered a lot of recent


advances in scientific breakthroughs, such as alpha fold protein structure predictions


or better traffic predictions. If you use my link, you'll get a 15% discount on the


course. Enrollment is open right now and lasts until April 1 or until spaces run out. All


right, let's get into the video now. See ya. Hello, everyone. Today I'm here with Clara


Meister, who is the first author of the paper Typical Decoding for Natural Language Generation.


Clara, welcome very much to the channel. Thank you. And thank you for having me. This was


a really neat paper. I have to say I have just finished my last interview, not just


now, but I finished my last interview about a system called blip. And what they said is


essentially, they have a system that generates captions for images in an automated fashion.


And then they have a filter that kind of weeds out the crappy captions. And they use that


as a means of generating more high quality data. And they and many others before them


have found that how you sample from a model like from the language model they've trained


matters a lot. Specifically, they told me that nucleus sampling in their case was really


a defining factor in getting more of a diverse sample set. And they particularly compared


it to greedy sampling and to beam search, which they found super underwhelming. And


I've come across a lot of systems in recent times. For example, alpha code as well. I


don't know if you know how exactly alpha code does what it does. I don't either. But from


the paper, I could gather that they sample a lot of potential solutions, and then they


reduce those down by filtering and clustering. And again, there they rely heavily on being


able to sample diversely and to sample, you know, many, many different things. And I've


for a while now thought, you know, maybe our sampling objectives are wrong for certain


applications, namely for the applications where we actually are interested in more of


a diverse output rather than the most likely output. And along came your paper, which essentially


exactly plays into this and suggests a new method. So I was super happy to see this,


I think it really hits sort of a nerve of the time. If you if you would pitch it like the elevator pitch for the paper, what would you say about it?


Yeah, I mean, I would say that, I mean, specifically for language generation, I think with these


large models that we've been training, that we when we're generating language from them, we


kind of need to take into account like what, what we really want from the model, what our


objective is, and also what we just normally do when we're, when we're speaking, when we're


writing, like how we use language. And, I mean, trying to think about having like this, you


know, what a model and what these models are, essentially, like probability distributions


of our strings, right. And that's kind of a strange concept, right? It's like, that's


not probably how we imagine language in our heads. But I mean, there is some like evidence


in psycho linguistics that that's kind of actually a pretty good metaphor for how language


is represented in our head. And then, I mean, how we then go from that to generating language,


and what the characteristics of the language that we typically generate are, I think, you


know, we really want to take that into account when we're trying to generate language from


these models. Yeah, I mean, if you ask me to like, say, if you just ask me to say something


randomly, right? I mean, it's like, what am I going to say? I'm probably gonna say like, I


don't know. I mean, that's, like, I don't really have these really common phrases. But if we


want something more interesting, like if you want me to say something more interesting, then I'm


going to not just pull like the most likely sentence out of thin air, I'm going to try to


convey information in what I'm saying. And, you know, I think that these models have sort of


learned what, how to do that implicitly. And we can ask them then to try and do this in a similar


manner to how humans do. Yeah.


So you pretty quickly get to this notion of typicality, which is a notion from information


theory, you connected to various disciplines in psycholinguistics. But a typical message, as far


as I can understand it is, well, as the name says, one that you would expect to see from sort of a


communication apparatus, but it is, do I understand this correctly, is one that you expect to see, if


you, if you assume that the communicators want to transmit the optimal amount of information?


Yeah.


Is this the core assumption behind sort of the, how we think about communication between humans?


Yeah, I mean, so one important thing is like typicality in the context of communication


channels is really only defined in the context of a message here, you know, some sort of message


that you're conditioning on and trying to convey. So in here, I mean, especially when you're like


sampling from a language model without having like this implicit message that you're conditioning on,


you know, in the background, I think it's kind of hard to really, you know, quantify what a typical


message in natural language should be. And I think we're very careful to say that there is this nice


intuitive link between typicality and, you know, how humans use language and what type of strings


we might expect when using natural language. But, you know, there's a lot of, there's a lot of


aspects of human language that don't really fall into the paradigm that you can really apply


typicality to.


And yet you, so you inspire, let's say, by this notion of typicality, you're inspired by, so you


define the notion of a typical message, and that is sort of the average information content you


would see. I made a bit of a characterization in my video. By the way, we have to inform the


viewers that I use the old archive version, and you just updated it. And you corrected essentially


all the little criticisms I had about notation and things like this, just to get the lore right. It


wasn't me that caused it. You did it ahead. And then, and then I, you know, I used the old version.


You know, props to you for picking them out. Yeah, like, my advisor always says that, like, every single


paper out there pretty much has like math errors in it.


Oh, yeah.


And it takes a critical eye to find them. It does. It's super easy to just, you know, find them


and then it does super easy to just glance over them not realize it.


Well, I think it I think it was actually straightforward. The paper is really easily


readable. So when we when we think about how humans communicate, and let's assume for a moment


what you say that in your hypothesis here, any given word should have an information content


close to the expected information content, i.e. the conditional entropy given prior context. In


words, we expect this difference to be small in human like text. And you also say that the human


goal over here is to transmit information effectively while also minimizing the risk of


miscommunication. I made a bit of an example right here as if I explain math, or if I explain the


chain rule to someone who does and does not understand math. Is this an appropriate example?


Is this an appropriate metaphor for what you're going for? Or is this is this totally off?


No, I mean, I think in a way, that's right. I mean, I think that also, that's actually perhaps


even more related to what we described later on, which is like the rational speak speech act,


which is, you know, how we we also are taking into account the listener, right when we're when


we're performing our messages. So I mean, that's definitely a component that's taken into


account. So we'll modulate the amount of information that we are conveying to basically,


like, you know, to account for what the other person might know. And I think that you can kind


of model that in different ways. You can say that for I mean, in your case, like, I think how


you're doing it, I think is a totally valid way to see it. In that case, we can say that the


information content for the speaker is going to be much higher than for someone else. So I mean,


yeah, I think that's a good comparison.


So this notion of the expected information content is pretty important here. And we say,


okay, if I'm at a certain, let's say I've ordered half a sentence, and then I look at the


distribution of the next word, and that distribution is just the distribution of the


language itself, if I understand this correctly. So I have my training corpus, which


supposedly is all of human language, I analyze it in my head, I determine what's the


conditional probability for the next word in the training corpus. And then your claim is that


what I do is I don't actually sample from that distribution, I'm going to adjust in inside of


that, the distribution that I sample from to two words that closely match the expected


information content. My question is, why? Why do I do that? Like, I see the problem with


always picking the highest likely word, right? If I if I have a broad distribution like this, I


don't want to do that. I don't want to just pick the most likely one. However, why can't I just


sample from this distribution, it seems like enough times I would actually, you know, pick


two words that is also completely fine.


Yeah, I mean, so first of all, I think one thing is, when we're forming language, we are, I


mean, we arguably aren't like sampling from this distribution, right? We kind of know, I mean,


maybe to some extent, we're sampling what we're going to say next. But I mean, I think the


important thing to internalize is that we have a message that we want to convey, right? Every


time that we're using language. And the way that we choose to do that is, like, add a specific


information rate. Because we want to communicate efficiently. But we also want to make sure that


our message gets across without like having to repeat ourselves or confuse someone or, you


know, making them like spend an inordinate amount of time processing what we're saying. And so


because of that, like, we're not going to choose super low information words all the time, because


that's just kind of inefficient. Yeah, like, I can, I can say all these filler words, right, with


and still get across a message. But adding like, it's like that, you know, that person that takes


forever to explain something just goes about it in a super, like slow and redundant way.


I don't make fun of my videos.


What I wasn't, what are you talking about? So, I think that's something to to think about. And


then sorry, the second part of your question, I've already forgotten.


I mean, I so I think I've, what I've understood is that if we look at just the distribution of the


next word, that is, in all of language that is across humanity, everyone who's uttered ever that


first half of the sentence, this is the distribution of next word. However, when I consider


that I actually have a message to convey, that distribution changes, right? Is that about the


characterization of what like, my question would be, why don't I just sample from this distribution


right here, given that, if you know, many words are possible, it will actually result in kind of a


diverse sampling?


Yeah, I mean, I think that you like, first of all, I actually do think that in the case of like a


perfect language model that you could actually sample from this distribution and be fine. I think


that there are some, there are some artifacts that are a bit strange, like, especially in models that


aren't trained as well with like this, this long tail distribution, that like, that tail isn't


necessarily learned all the, learned very well, like what those actual probabilities are. And so, you


know, you end up with, like, just oddities. And, but beyond that, I mean, I do think that, like,


we're not, I mean, we are trying to modulate when we speak, like the amount of information that we


have per word, right, to keep it even. And this is, this is not, I mean, this is something that is


perhaps not very obvious, but it is something that's like, well studied in psycholinguistics, like


how we, how we convey a message and like the coding that we will use within natural language.


And so, like, yeah, we, we, we take this into consideration when choosing the next word. Yeah,


not to be too redundant or to be too surprising.


Yeah. And to, and, and again, to transmit what we actually want to transmit, right? Because I have


something that I want to say, and that means I can't just, you know, blindly sample from the


distribution, I would never actually transmit what I wanted to say. Would it be, would it be


possible that, let's say, if I could hypothetically determine, you know, what, what kind of, let's say,


I have a message I want to transmit, could I somehow define the information content of the


next word, given the message I want to transmit and maybe also given the sentence, you know, so


far t smaller than, or smaller than t. Well, that's, I mean, that's actually usually what we're,


we're doing. And in, so in a task like abstractive summarization, which, you know, we see is


something that we experiment with, we are conditioning on that message, essentially, you know,


a message being the, the article, right? And so it is like, we, we are taking that into account


when we're trying to build our next word. Yeah. And it is still like this distribution should reflect


the fact that there is a message that we want to convey. And, you know, given that message,


it sort of, it sort of reflects that, you know, maybe this word that without that knowledge would


have been very surprising, but like with that knowledge, with knowing that, like, we want to


transmit this message, actually, that word is like what we would expect. Yeah. Okay. My, my


question, what I'm trying to get at is, if I train my language model for abstractive summarization,


right, the conditioning of the message is maybe already in, not maybe in here, if I use a decoder


only model, but like, my question is still, why is this distribution here not enough? Like, why,


why do I need to cut out the most likely things? Even though, you know, sometimes I actually want


to say them. So, I mean, I think it's just to be more human-like. Yeah, that's, okay, that's the


most I can say is, yeah, it's a, it's fine, right? So you, you make, you come up with, and we're


gonna, I'm gonna go back to these plots, because I find them super interesting as well. You define


this typical sampling strategy, where you say, okay, we, we have this thing here, which is the


expected information content of the next word. And then we're just trying to, as closely as


possible, match that. So we're going to select a subset of all the words that we could pick, which


closely match that expected information content according to your hypothesis. And then we're going


to sample according to the new distribution that only consists of the subset of these words. So in


the video, I think I raised a point, which is maybe more of a, I don't know if it's circular logic or


a philosophical point, but all our training data, presumably of these language models comes from


humans, you know, using language, transmitting information. Therefore, right? Shouldn't, like, if


I now train my language model, and I use your method to sample things, and you claim it's a


human-like way of sampling things, shouldn't that a result in the same distribution? And b,


shouldn't it sort of the expected information content, if I measure before and after, like, if I


measure it in the training corpus, and then if I measure it as an output of my model, shouldn't


that be the same? Because presumably, the training corpus is already generated from humans.


I mean, yeah, I think like, so yes, I think that makes sense, if I'm understanding correctly. And I


also think we're kind of seeing that, like in the earlier plots, we're actually seeing that, like,


if there is like an average amount of information, right, according to the model, there's an average


amount of information that each word will contain. And I mean, human text seems to be coming from


quite close to what the model has learned that average information rate to be.


And do you, did you investigate the outputs of your model and sorry, sort of redid those plots


on the output of your model and observe the same, the same pattern?


Yeah. So that's like, yeah, that's something we did as well. We looked at basically a few


different decoding schemes and saw what the, what these distributions looked like for the outputs of


those decoding schemes. And I mean, things like, you know, to nucleus sampling with,


like very popular, you know, popular values of P looked similar. And so did the ones from typical


sampling. We didn't, I think, honestly, they do look, they, by visual, like visually, they look


pretty similar, which is nice. It's also nice to see that sort of these more, these vetted


decoding processes that have like stood the test of time are also actually mimicking these


distributions. I think that if we wanted to be robust about it, we'd probably want to,


you know, come up with some sort of quantification for how different these distributions are and use


that perhaps to see if that correlates with how well these decoding methods perform in terms of


things like human evaluations. So can you tell us the story behind these plots a little bit more,


because you define epsilon in terms of an absolute value yet here I see values that are


less than zero to both sides. So I didn't know which one is which. What's epsilon here? I tried


to make it clear in the caption of the text, but I don't think I did. I mean, if I guess the


correctly, it's the conditional, it's the expectation minus the actual information.


No, so it's actual information minus.


I would have gotten it wrong.


No, no, I think you're right. No, no.


But maybe you can tell us what does it, because these are kind of, so it's more, if I see this


correctly, more sort of mass on the left side of these close to the left side of the right side.


So mass on the left side of these close to this boundary, which is really interesting. And then


there's a long tail on the right hand side. What does that tell us about human language?


I mean, that's like a very deep question. And I'm not, you know, I'm not entirely sure about what


the shape of this distribution means. And I think it's very interesting that this is the shape of


the distribution. And actually, we did, I mean, we used a few models here, and all of them kind


of did look like this, where you had like this peak and then sort of a long tail. And yeah, I


mean, I think that that's like an investigation in its own right about how humans use language.


But so yeah, by the way, it is information content minus entropy. So remember, so like low


information content, high probability, right? So actually, human language tends to be to the like


on the higher probability side of conditional entropy. This, this thing right here. So if we're


way out on the right, it means that we actually transmit a lot of information actually more than


would be expected. So there is it doesn't, there is a long tail of very high information words,


let's say, do you think so because you in one thing that I skipped over that in the video review,


but you make this point of humans, what they probably do is they want to, everywhere in the


message, they want to have kind of a constant information rate. So every word should approximately


transmit this, this expected information. So as you go through the sentence, do you think this


could be violated a little bit because humans, most of them do tend to have like a short term


memory of three to four words or so that they, you know, can keep keep ready in the sentence, maybe


I can transmit this super high information word. And then before my receiver gets super confused,


I can follow that up with like two or three clarifications, which, which would be then maybe


here in the lower information content, but they would be more. Yeah, I mean, so like, I think it's


hard to always avoid moments of high information. I mean, for example, if you're giving if you think


about this very literally in terms of like, what those words could be, you know, they could be like


someone's name, right? And that's kind of like you're introducing someone that's always kind of


going to be like a high information moment, right? You have to remember, I mean, we always forget


people's name, people's names, obviously, there's like, there must be a lot of information in those


names. It's a very off the cuff explanation. But I mean, yeah, so I think it is hard to just 100%


of the time, avoid those instances. But I mean, this is talking about sort of on average, what


we're doing when we're constructing language. And, I mean, so I guess I couldn't say whether


in those moments, we want we try to perhaps on either side, balance out, like with lower


information words, this high information word, because, I mean, you know, maybe maybe we do in


order to give the listener some time to internalize this information. But they're also, especially


with, with speaking, which is a different domain than writing, right? There are other ways that we


can modulate high information words, right? So we can elongate our speech to basically


spread out information over time, right? And so it's not like here, we're just evaluating text.


So, you know, we, I think, especially in text, we're going to see these longer tales, because


you can't sort of distribute information over too many words in certain cases, like in the case of


introducing a name. Yeah, I think that's...


And also, it has to be said that, you know, you can, if you go to the left, you get into the super


low information words. And there's only that many of them, right? As soon as I'm at the and a,


right, there aren't that many. However, there is in fact, a long tail just in the language


of super high information words that are quite unlikely. So maybe that plays a role into it as


well. About these plots, you say you draw two different conclusions right here, which the first


one is the peak nature reveals that humans indeed tend to form language with per word information


content quite close to their expected information content. So this is kind of, you know, here is


data that shows our hypothesis is correct. And the second one is the centering of these


distributions around a value close to zero reveals that our probabilistic language generators are


learning what this rate is, which, and my point was a bit when in order to make point one, you


need point two as an assumption, right? You need you need to claim, well, I can only say this,


because I assume our language models are modeling the probabilities of language well enough.


Otherwise, I could not conclude point one. Likewise, you couldn't conclude point two without


having point one as an assumption. Is this am I overlooking something here?


Well, so I mean, I think the point here that we wanted to get across is really that, you know,


two things should be looked at in these graphs, which is the centering of the graph and also the


shape of the graph. And I mean, so I think there is there is an assumption that kind of has to be


made here. I don't think it's quite as severe as what you've mentioned. But I mean, it is sort of


that this enter this information rate is kind of a ground truth of sorts. But I mean, you know,


you could, for example, shift like you could shift that entropy rate, you could shift the entire


distribution and still you could shift h and all the p's and you know, all of those things.


All of those numbers and still technically get the same distribution. So that I agree with.


But like, I mean, I think like looking at the peakiness of it, clearly we're seeing that,


you know, humans are generating language around a certain that something right.


Like, yeah. Yeah.


Like if what if it were centered around two instead of zero, right? It would be as as peaky.


Well, yeah, I mean, as peaky then like, yeah, like we'd probably be that probably show that humans


communicate at like a very low information rate, right? Or yeah. So but no, I mean, it's around


like, it does seem to be close to this expected information rate. And I think one other like the


part two is really trying to show that like, there's this we would expect that if our model


understands that, you know, humans are speaking at around an average information rate, that that


this distribution would be centered like on average, it would be predicting that information


rate for a given word or like that information content, that probability for a given word. And


it does seem to be doing this.


Cool. Yeah, this is this is just a bit of a nit or a nitpick for me. It's I'm totally on board


with I mean, it's pretty clear, the language models do model these probabilities relatively


correctly, especially the ones in the with the higher probabilities. And I'm I am actually I'm


fairly convinced by these by these plots. Yeah, no, I think you bring up a really important point.


And I actually like I'd spent a long time thinking about whether or not it was too circular. Like,


you know, whether you could have one without the other. Really, and I mean, I think like, I think


at some point, I came up with some examples, like some counterfactual examples, where actually you


could have one without the other. And of course, now, like, I can't remember what they are. But


yeah, it's, it's, it's, I think, I think people understand what you're what you're saying.


And there's definitely like a degree of freedom there, right? There's definitely something that


could change that, you know, you could get those same results. And I think but I think, like,


that thing that could change would be whether the information rate learned by the model is like the


quote, human information rate, the actual human information rate. And I'm actually not entirely


sure that's important. It just has to be and just has to get it right, like relative to what it's


predicting the probabilities for words, right? Do you want to tell us a little bit about the


experimental results, because I have not gone into these at all during the paper review, things that


you could highlight or anything like that?


Yeah. So like, as Yannick mentioned, there's a new version archive, where we are, we also present


a few different values for nucleus and top k, as in like the same, you know, same number of


values.


Oh, yeah, the hyper parameters. Sorry about that.


No, no, I think it's very reasonable. I mean, the thing is, like, you know, there were only so many


human evaluations we could afford. And we thought like, you know, we should probably test out more


values of our own method, since no one has done this before. But like, a lot of people have looked


at nucleus and top k sampling. But then once it seemed like, okay, this is worth, this is research


worth doing, we were able to get a little more money and launch a larger human evaluation. So


those results are now in the paper. I mean, I think one thing that was really interesting for us was


actually just the variety of values of tau that worked well. I mean, basically, like, most values


of tau worked well. There wasn't like a huge difference between all of them, which we thought


was really cool, because in comparison to nucleus and top k sampling, those methods were really


dependent on n and k. And I mean, I think there was like a little, like, if you just look at the


output of these models, you know, if you have a large tau, then maybe qualitatively, you could


say that the text is like, a little more normal, like a little more standard, and then maybe a


little more diverse for low values of tau. But I mean, basically, it was just for, it was just


interesting to see that for these two tasks, at least, that, you know, variety, like it wasn't,


really need to tune tau that much, just kind of, kind of work.


I mean, it's important, right? Because that's one of the issues with these things is that if


I have to tune the thing to every new task I do, I'm a lot less certain in, you know, kind of the


generalization of this even within the same domain. But if it's interesting to hear, and


if it's really a kind of a handle on the craziness that I get out of these models, that could actually


be even a cool property, right? If you say, actually, most values work, but it is, you know,


it changes just the style, I think that that is a useful hyperparameter rather than a nuisance,


like in nuclear sampling, you know, if I don't get it right, it's going to be crap.


Yeah, well, I would like to think that that's the case. I'm slightly biased here.


Yeah, is there any, I mean, you run various automated tests in abstractive summarization


and story generation. Most of the time, the typical sampling is on top of the pack, sometimes


not, especially here in the story generation on some of these automated evaluations. Is that


kind of an interplay between the evaluation, how the evaluation is done and the methods? Or if that


is that a property of the task itself? What can you tell us about this?


I mean, so I think a lot of these metrics, I think a lot of these metrics can only tell us so much.


And, you know, the text that we end up generating, how it performs in terms of these metrics,


I think like you'll see, for example, in human text, you'll get reasonably different values.


You can get reasonably different values for things like repetitions and within reason,


and the text be equally as good, at least qualitatively. So, like, I think the


important, I don't know, I don't know if it's important is the correct word, but one of the


critical things for us was looking at whether we could avoid this really degenerate behavior with


models, because I think that's something that's like one of the bigger problems in language


generation is just like this tendency for these methods to fall into repetitive loops.


And I mean, we basically just like, we didn't really see any of that in the


using our method. And so I think that was an important takeaway. So yeah, I mean, always kind


of performing well in terms of this, in these metrics that show how repetitive or redundant


text is, I think is what we would expect, right? You know, we're saying that, like, if text is,


we want text to be about as redundant as human text is, because that's like one metric you can


use to quantify information content, right? So it was good to see that that like, at least,


it's a necessary, not sufficient criteria, but it was good to see that it was met.


Yeah, I was just looking like just now looking at perplexity, and yours is in bold. And I was like,


wait a minute, lower perplexity is better, usually, but then I realized, I realized what


you're what you have to do here is obviously match the perplexity of the of the reference text as


closely as possible. So the goal is to be as close as possible to that number, which is really


astonishing to see because, you know, in machine translation, people are fighting for point one


perplexity or so for the new state of the art. And here it's a difference of, you know, it's quite a


magnitude of difference between these, between these methods, which is cool to see. And I think


shows shows quite well that in something like story generation, these models might really just


not overfit is the wrong word, but overproduce, not as creative outputs, or maybe even degenerate


ones, as you say, I mean, I think actually, in the context of machine translation, and this is


something that, that, you know, an experiment that I want to personally perform is look at what the,


like average perplexity of the reference text is. Right? I mean, so, and the generations, right? I


mean, so the one thing about machine translation is, like, typically, we're evaluating on things


like blue, right? Not, not perplexity so much. So we're evaluating on the generations themselves,


rather than the evaluation of the reference text, like what the perplexities are. But I mean, it


would be, to me, it would be interesting to see what, you know, good generated text, what the


perplexity of good generated text is, compared to human like text. And I think in that case,


they, they would actually probably both be quite small. At least that's my, my intuition. Of


course, one, one artifact that I think would kind of get in the way of these experiments is the fact


that machine translation often uses label smoothing, right? And label smoothing is basically


like a form of entropy regularization. So it makes these distributions higher entropy, even,


yeah. Even if they shouldn't be. Yeah. And that actually, I mean, basically, you can,


you can read other papers about this that'll explain it. But it is kind of, it does interact


with beam search, like, it's like, you know, the match of beam search plus label smoothing


tends to work quite well. But I think if you were to really perform these types of experiments to


understand what the types of perplexities for machine, like for translations, good translations


would be, I think, yeah, you'd need to do it with a model that doesn't, that hasn't had this sort


of artificial inflation and entropy. Do you think our training objectives are the correct ones?


Let's say, let's think of something like story generation is pretty, because what I'm hearing


now is that, well, label smoothing, but plus beam search works, but it's more like a hack


to get around the weaknesses of, of beam search without label smoothing. Do you, and, and that is,


you know, something I can maybe, you know, get, get behind. Do you think we have the correct


training objectives if our goal is really to create a diverse and interesting set of outputs?


Do you think it's a good strategy to train, let's say maximum likelihood, and then sample using


something like typical sampling? Or should we also change our training strategy?


So I personally think that maximum likelihood is a pretty robust objective. I mean, in terms of,


like the information theory perspective, I mean, when you, when you are maximizing likelihood,


right, you're also minimizing KL divergence. So you are basically looking for the model that


assigns the same information contents to strings as, as the empirical distribution, right? So it's,


like, you know, they're just equivalent. And so I think if you take that into account,


basically if you take into account exactly what you're doing with your objective,


and then from that, you know, go on to, okay, well, given, given this distribution,


right, how, how would we go about, how would, like, we as humans go about generating from


this distribution? Or, you know, how would, if, like, you're generating an image, like,


how would nature go about, like, generating from this distribution? I think, you know,


that it's, it's really important to, I don't think there's a correct way necessarily to go about


training and decoding, but I think we really need to take into account more their interaction


and understand, like, what is going on within that interaction.


Yeah, I mean, I'm all on board because it also means that we can use, we can reuse the same


model for multiple, let's say tasks, if we swap out our decoding strategy. Can you tell us a


little bit about these plots and what we see here?


Yeah, so this is more just showing the repetition values. So kind of what I was talking about


earlier. So high repetition values would indicate that we're getting into kind of, like,


degenerate loops, like repetitive loops. So where the model outputs the same thing over and over


again. And I mean, we really see this in story generation for low values of K and N, where,


yeah, exactly there. So, you know, this is, these are, like, repetition values of, like, 0.8. So


it's just, like, really just spitting out the same exact thing over and over again. And I mean,


yeah, it's, like, I think that looking at this type of behavior in terms of information theory,


it actually really makes, to me, it makes it make sense why this is happening, right? If we're


saying that we're always going to output the most likely word, like, those are also the words that


just have, like, no information content, right?


And also, like, if I come to you, and I say, look, here's a sequence of words, it goes,


apple, banana, peach, apple, banana, peach, apple, banana. And then to ask you, like, what's next? I


mean, it's quite likely that, you know, peach is the next thing. And that explains very well why


if you keep repeating, you're sort of reinforcing even that repetition, because as you keep


repeating, the next repetition becomes more likely, yet the transmission of information is,


is almost zero.


Yeah. But I mean, I think one thing that would actually be really interesting, one sort of


experiments that we have yet to run, is to see, you know, if at the before you get into these


repetitions, like if you start with with something, and then you like if you start with one phrase,


and then go into typical sampling, right? Can you prevent some of these repetitive loops,


because you've now come in with the objective that you want to transmit, like more information,


you don't want to be, you don't want to transmit like a small amount of information, which is


achieved by like doing by giving high probability low information words, right? So kind of seeing


if typical sampling can almost help us break out of repetitive loops,


although by your own, by your own, what you wrote, if you are, let's say, in such a loop,


let's or at the beginning of such a loop, the distribution would be extremely peaked, right?


And at that point, typical sampling would also go for the for the high probability words,


or is that I mean, it and honestly, like, I think it should, right? Like, at that point. But I mean,


this is kind of why it's like before you get into the repetitions, right? So like, at that point,


you know, where something like nuclear sampling might decide like, yeah, like, the lowest


information choice is, you know, just to repeat what's already been said. Yeah, if we can prevent,


we can prevent those types of behaviors.


Just some small technicalities, whether where I want to ask you if you think that it's appropriate,


do you think the absolute difference is an appropriate measure? Or why did you decide on


that? That's the first thing. Second thing is, do you think this cut off this hard, you know,


I'm going to take this many words, and then I'm going to exclude the rest. And then I'm actually


going to sample from that bunch of words, as if it were like the original distribute, like,


with with their original logits. So just the technical implementation of the idea, what could


be like, what are arbitrary choices? What are what are things that you did for a reason? And how


could they be better? No, I think that's like a great question. Why absolute value versus, you


know, squared distance? And, and why the hard cut off? I mean, to be honest, I think that this was


the original instantiation of the idea was, you know, just choosing words from like near the


information content, near the expected information content. And I think in order to really introduce


this concept into the literature, it helped at least what I thought was that it would help to


have something that was akin to what most people are familiar with, which is nucleus and top case


sampling, right. And so, for better or worse, this method was kind of like, okay, here's something


that's very parallel. That'll be easy to understand, you know, it's, it's, it's also just


truncating the distribution, also like looking at the specific portion of the distribution. And


that's where we'll sample from. Now, whether it's better to use the squared distance, I mean, so


we ran some additional experiments later on, like after releasing this draft, looking at things like


the squared distance, and, you know, trying to come up with a soft distribution. And yeah, they


worked about like, about the same, sometimes a little bit, like, honestly, I think I'm gonna have,


like, I think there's just a lot of research to be done here. I think there's a huge, huge body of


research that can be done in sort of figuring out exactly what our objective should be. Perhaps


learning this objective, like, learning what the correct, what the correct formula right here


should be. And that's, you know, that's to come in the future. So I can't say that squared distance


isn't better. Very well could be.


All right. Is there anything else you want to get rid of? How can, can people get started with this?


Is there code somewhere? There is code, right? I've seen that.


Yeah, there's actually code in Hugging Face already. So if you have, I don't know if they've


released a version since it entered the library. I mean, it's been in there for about a month now.


So I think if you have, if you have the transformers, the Hugging Face transformers library


installed from source, if you have pulled it in the last month, it'll be in there. And, you know,


when you generate, if you just add in the argument, typical p equals something, then you'll have,


you'll have typical sampling. And I mean, I really encourage people to play around with it.


I mean, I, yeah, you know, you're going to expect me to say this, but I've actually just been really


impressed by the outputs of typical sampling. Just that they have been pretty high quality


from my perspective. And interesting.


Cool. Clara, thank you very much for coming here.


Thank you. Thanks for the great conversation.


It was a pleasure.


You know, maybe you'll see another update on Archive with some of the other stuff that


I've done with some of the things you've pointed out. Clean up some of my arguments.


That would be excellent lore for the channel.


Yeah.


Cool. Thank you.


Celebrating 100k Subscribers! (w/ Channel Statistics)

100k. Nice. Big celebration. We have just reached 100,000 subscribers. Now truth be


told as of recording of this videos, we actually don't have 100,000 subscribers yet there's


like 156 missing. So all I have to do is not get cancelled in the next two days or so.


And this is harder than it seems, but I've managed so far I think I can make it. So thank


you everyone who's been here for any amount of time 100,000 of you have decided to click


on the subscribe button and I'm eternally grateful to every single one. I would have


never ever ever thought that a dude on YouTube talking for 45 minutes about research papers


and stuff would get any attention at all pun intended. But hey, it's come to this. So thank


you all so much. This has been absolutely great. I have no intention of stopping. Now


this video right here is supposed to be a little bit of an announcement video. And also


I thought we'd look a little bit into the channel statistics because I know some of


you are interested. So what are the announcements? As I said, I have no intention of stopping


reaching 100k doesn't make a big difference in terms of content. In fact, I have lots


of ideas for nice content and probably more ideas than time to implement them. But there's


some cool stuff coming up. Also, I will be hosting and ask me anything on probably Sunday


as gonna happen here on YouTube. So you'll see that pop up if you're around at that time.


Next thing merge, I thought it'd be funny to have a little bit of channel merge and


I don't have it ready yet. But we'll chat on this court a little bit about what is going


to be offered because I do want your inputs into these kinds of things. So let's get some


funny merch. And I think that'll be cool. Speaking of discord special thanks to everyone


who is there who participates to everyone who has ever asked and to everyone who has


ever answered a question in the help channel to everyone who has participated or even just


listened to the paper discussions we host there is special thanks to the regulars and


to the moderators who keep everything going. This would absolutely not be possible if it


were just myself. So huge thanks to everyone there. This community is just amazing. And


we will not be at 100k right now if it weren't for the support that I'm getting from there.


If you're not yet a discord member and you do want to be more involved, link is right


there in the description. Everyone's welcome. As I said, next to the usual discord chit chat,


we have regular paper discussions. And also there are some community projects. Currently,


there is one called homebrew NLP where the goal is to build a framework that can run


really large language models on a single machine. If you're interested in that absolutely join


and participate in creation of that. Very cool. Okay, that being said, let's dive a


little bit into the channel statistics. Now I think due to the rules of AdSense, I'm not


allowed to show you the exact numbers of revenue that come from ads. Not entirely sure that's


the rule actually, but I have heard it from somewhere and I'd rather not get into trouble.


Safe to say it's not nearly a number where you could live off of this or anything like


this. It did support for example, the new camera that I've gotten so you can enjoy me


in excellent quality. Also, thanks of course to the patreons and subscribe star supporters


and also the people who've sent me a bit of crypto. This has also enabled me to get a


new iPad instead of my old Surface tablet, which makes the creation of the paper reviews


just a lot easier. So thanks a lot for that. So here I've pulled up statistics since January


2020. I have made numerous videos before that, but not nearly at the scale or frequency that


I'm making them now. So the real video making started in the early days of 2020 when the


first wave of the current global phenomenon hit and I suddenly found myself with a bit


of more time on my hands. And at that time, I was watching a lot of videos by people like


PewDiePie and Casey Neistat and I deep respect for these people that upload every single


day. And I asked myself, how long could I keep this up? And it turned out I could keep


it up for about three to four months. So as you can see, YouTube is mostly a grind with


a few intermittent spikes. I believe the first spike here is GPT three. And the second spike


is alpha fold. You can also see the times I took a couple of breaks namely here in late


summer of 2020. And in early summer of this year, it's pretty cool how you can see all


of this in the stats. Also, we've recently passed 4 million views, which is crazy. Interestingly,


here you can see while a lot of people appear to have watched the GPT three video, not a


lot of people have watched it to the end. See the difference? Spike? No spike, spike,


no spike. Maybe that was a different video. Top videos, of course, the all time favorite


attention is all you need. See, I've uploaded this in 2017. And it's drawn people ever since,


which means I must have done something right now people have told me to get a thumbnail


for this going or anything like this. But I'm not I'm not going to change a single thing


about this video is doing well, people are watching it for a long time, not going to


change a thing. Here you see other popular videos are alpha fold and GPT three. Now also


surprising is trans coder, which a lot of people watch, but then they watch kind of


none of it. So this might have been the big spike. I'm not sure if the thumbnail here


is misleading and people expected coding content rather than an analysis of a research paper,


or it's because the first part of this word is sort of politically overloaded. And maybe


people clicked on that, or the algorithm recommended that to people. I'm not sure, but it is what


it is. Interestingly, click through rate has been going steadily down. I'm not sure if


that is to be expected as you grow, I guess. I'm not sure. But maybe I should do a little


bit more clickbait to get people to click more when people search for this channel.


The most thing they search is my name, which is quite flattering. And then it is the titles


of the videos they're interested in such as attention is all you need GPT three, alpha


fold or vision transformer, which was a cool video. If you remember, I reviewed that before


it was clear who the authors were. And I sort of d anonymize the paper live. And yeah, I


thought that was funny. So who are you? You are probably on YouTube mostly around 6pm


in Central Europe, you're probably also subscribed to two minute papers, Lex Friedman, Tesla,


the MLS three talk and Sabine Hossenfelder, among other channels. Now specific shout out


to MLS three talk if you're not subscribed to that I can highly recommend it. I'm part


of it not always but a lot of times and we have super duper interesting discussions with


people that I would have never guessed I could ever reach and talk to and ask them questions.


So I think we have really cool guests and the conversations are often quite technical.


So I think you will enjoy that. In terms of watch time only about half the people are


subscribed, which is surprising. That means 200k subscribers isn't far away. And 19 out


of 20 of you are probably male and a lot of you are between 25 and 34 years old. Now I'm


never sure if that is just the statistics of the people where YouTube knows what they


are because they've specified it somewhere or is that what they guess about people in


which case I guess that will be seriously distorted because the guessing would probably


be based on something like your interests, which might be that if you're into a lot of


technical subjects, you're more likely to be male, but then you count that to the statistic


here and probably that statistic is then used again for training the algorithms. I'm not


sure. So I'm not going to interpret too much into this thing right here. Also, you're quite


likely to be from the United States or India, but really the geographies are distributed


quite all over the world. Okay, I've actually figured it out. Yes, the giant spike was in


fact the transcoder video. And here you can see that the traffic source was mostly external.


So in fact, the GPT three video was a much smaller spike, not much earlier than the transcoder


spike. So this was it for the channel statistics for the celebration of 100k. Thank you so


much to everyone who is here to everyone who's helped and who's participated. I hope you


still enjoy the content. I still read all the comments. If you have any feedback, any


wishes or anything like this, let me know. I'm looking forward to what's to come and


Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions

He did it! Crazy son of a bitch did it again! What am I talking about? Jürgen Schmidhuber,


reinforcement learning upside down. New paper just dropped on the verge of the NeurIPS conference


being presented at a workshop here. Presenting upside down reinforcement learning. I am pumped


for this one, can you tell? So, it says we transform reinforcement learning into a form


of supervised learning by turning traditional RL on its head, calling this RL, what are


we going to call this? Let's just call it LAR. Upside down reinforcement learning. And


so this is upside down, never mind. Okay, let's just check out how it works. So, I'm


going to give a brief overview before we go into this paper. Alright, so let's say you


have a reinforcement learning problem, let's say an Atari game for example. And in an Atari


game you usually have a screen, right? And let's just say you're playing this marine


commander. So there's water here, right? And there might be a bunch of, here's your boat,


right? There's a boat, a little boat. There might be a bunch of opponents right here,


fishy fish opponents, fishy fish opponents and so on. And a bunch of gold coins like


here, that's a big gold coin, right? And you're kind of supposed to, I think you're supposed


to like go get air, you have some air meter over here, whatever. So there's this Atari


game, right? You're supposed to get the reward, which is this, maybe this this coin here,


and stay alive as long as possible and so on. So this is a classic reinforcement learning


problem. And there are various techniques for this, we've looked at a couple of them.


And what upside down reinforcement learning does is basically, what you do is you want


to transform this input to a new representation, which basically, well, if I can, maybe I can


let me get this correctly. So this is this over here, and then there's a little fishy,


little fishy here, and there's a coin right here. So what you want to do is basically


turn this input on its head like upside down. And so this way is kind of up or down, whatever


in this new representation. And if you actually learn on this new representation, with pretty


the same techniques, it works much better than the classic RL setting. And this is not


only for like these Atari games like this, this appears to hold throughout the RL space.


So in robotics, like if you have a robot, whatever, this is a robot, it has a square


head, as you can tell, you know, it's supposed to like open a door, you've seen this DARPA


challenge, this doesn't work, right? But if you just transform this and actually turn


the robot, turn the robot upside down, that the robot will be able like to open the door


just fine. And even like if you have a chessboard, and there's like a bunch of pieces on it,


the problem in this case is you have to, so you need to simulate this chessboard. And


if you turn this, if you turn this around, now, you basically all the pieces will fall


off. So what you need to do is you need to have a simulator that encodes a magnetic chessboard


such that the pieces don't fall off. But so it's a bit of programming effort. But if you


do that, all right, I'm kidding. This is a new paradigm for RL, but it's unfortunately


it's not as good though someone should try the magnetic chessboard simulator. Upside


down RL is a new paradigm for RL, where basically the the kind of notion of inputs and outputs


of the RL algorithm are switched around a bit. So basic ideas here is, is that you have


an RL algorithm that is also fed with a bunch of commands. So in classic RL, what you'll


have, let's actually go back to this, to this Atari game here, right in classic RL, and


RL algorithm will get the Atari game as an that the screen as an input and is asked from


this to predict a bunch of outputs. So in classic Atari, these are eight actions, I'm


gonna draw three here, like go to the left, go to the right, or press the button for shoot,


right? These are the actions you have. And the algorithm is tasked. And there are different


versions of this in policy methods, policy gradient methods, typically, the algorithm


is tasked with outputting a distribution over these actions with in other methods like value


learning, Q learning, the algorithm is tasked with assigning each of these actions a value.


So it's a in this situation, going to the left will be worth three in the future, this


going to the right will be worth negative one and shooting will be worth zero. So you


might want to go with the with the this this action here. Right now in upside down reinforcement


learning, so we've had right, we've had observation going into the model, and the model coming


up with the value estimation of the different actions. In upside down reinforcement learning,


you'll have the observation and something else going into the model and the model coming


up with an action. Right? And this something else is the key. What you input here is your


desire, your future desire. And in this paper, they call it a command. So you'll have a command


as an input together with the observations, you basically say, here's my state. And I


would like to achieve, let's say five reward in the next five reward in the next two time


steps, right? Make this happen, right? This is this is your command going into the model


and the model will then try to find actions such that in the next two time steps, you'll


get five reward, you can easily see a model that learns this will actually be able to,


you know, do various things, including doing the classic RL things like get as much reward


as possible in given or in the shortest amount of time, but can also do much more. And in


the general sense, the difference is how this is trained. Now, this model, when you train


it, as you can see, you don't, it's not trained with in mind having in mind kind of only to


get the maximum reward, it is trained to be much more a general kind of understanding


of the world. I mean, learning, what do I need to do to achieve a variety of goals?


Specifically, what you want to do to train this is the following. Say you have a method


of of moving in the world and collecting traces, right? So you go from state, state one, state


two, state three, you go with like your action one, action two, let's draw action three,


to state four. And in each of these, you get a you get rewards, right? Reward one, reward


two, reward three. Now, this in classic RL, this will kind of give you one training example,


right? So this is if you consider this to be an episode, this will give you one training


example to to run this sequence of actions. upside down RL, you can actually consider


this as many, many training examples. And here's what I mean. So if you, for example,


start at state one, you can say, aha, within one time step, one, one time step, I go to


state two, and I have achieved r one rewards by doing action a one, right. So this now


can be an input to your model, your model could learn. If you get as an observation,


and remember the previous thing as an observation, you get s one, as a command, you get, I want


to achieve in one time step, r one reward. Right? And you train this goes into the model.


And the model is trained to say, a one, given if I am in s one, and I do a one, I will achieve


that, right. So you train the model to give a one as an output. And this is valid because


in the past, you've observed going from s one, using a one to a state where you get


this, this kind of reward in this kind of time. But you can also so you can do all of


these single steps, they will all provide individual training examples to your model,


right. But then also, you can consider a two step thing. So you can say, I'm in state s


one, and I go, I go in two time steps, I have achieved r one plus r two reward by doing


actions, a one, then a two, right, and a two, I'm going to do in parents here, because what


you want to do is you want to always always consider the action that comes right after


where you are now. So again, your training sample, let me draw this up here, maybe your


training sample would be the following. I am in state s one, this will be my observation,


my command would be I would like to achieve in two time steps, reward r one plus r two


reward, right this reward. This both goes into the model, right? You tell the model,


please, given the state s one achieve this reward in this time, and the model is supposed


output a one, saying, aha, in the past, I was in this state. And I did achieve this


goal by using that. So the model is supposed to learn to achieve different goals, right.


So now you can not only train from good episodes, right, you can train for any episode, any


episode. Usually in classic RL, you kind of want to focus on the good episodes because


you want to maximize your reward. But here, you can tell the model, hey, if you've done


something particularly stupid, let's say here in s three, you've done something the a three


was particularly stupid gave you that. So our three here was really bad reward like


a negative 5 billion trillion. And you can actually train the model to rep to recognize


this can be say, Hey, look, if you're in s three, and within one time step, you want


to achieve negative 5 billion billion billion trillion reward, you all you have to do is


action a three, right. And then the cool thing now is if you are at evaluation time, you


actually want the big reward, what you'll do is you simply plug in a different command


is simply in one time step, still I'm in state s three in one time step, I want to achieve


actually three reward, not negative a lot, right. And the model will have learned that


a three will lead to a situation where you get a lot of negative reward. So the model


will be like, I'm for sure not going to do a three, right, I'm going to do something


else here, because I have learned to map a three to like this really low reward. So in


essence, this has connections to things like hindsight experience replay, and kind of universal


value function, where you kind of learn to go from any state to any other state in this,


but none of these do none of these have this kind of command, what the Schmidhuber calls


command here as an input to the model. And I think actually, this is this is really positive


to input this because usually in universal value functions, what you would say is, let's


consider a simple grid world, right? Whatever your agent is here, and you need to, you need


to reach a goal that's down here. But you might not be able to learn it because it's


super sparse reward and so on. But what you can do is you can learn to reach this position


and this position and this position from various positions like go here, go from here to here,


you can learn to go from here to here. And you know, in essence, you would like it eventually


to to generalize to all the fields. So you basically learn to go from any position to


any other position with your agent. But these universal value or universal policy functions


having sub goals, but they during that phase where they learn to go from anything to anything,


they don't, they don't necessarily include this reward thing as a as an input, it's more


like kind of a either a sub goal, or that the use to value function will simply approximate


the reward, whereas, whereas in this technique, we actually have a policy learning, we actually


output a an action value. Also hindsight experience replay, what hindsight experience replay would


do in the same situation, right? You're here. We might do a videos on this in the future.


You're here and you try right and your agent actually, it ends up here, right ends up right


here. What you can do is you can simply say, Oh, well, actually, this, this was my goal


all along, and then simply train, train your model as if as if this thing here was your


goal all along and not this thing here, and treat it as kind of a positive reward for


this, at least that's how understand it. Right. And both of these things are quite different


than here where we have this command as input. And I do, I do like it. So I think this this


is very much the basic things. Here it is, it is extra extrapolated to kind of noisy


inputs, noisy environments, and so on. But this is the basic the basic gist of it. So


here, you see, you're you what you will learn is to map all and all is your representation


of your input. So the screen, for example, or the chessboard, and I think also kind of


the last action and reward you get in this step, plus your horizon and desire. So in


how much time you would like to achieve how much reward, and then you can also get input


some extra goals that you have. And so you can see basically any, any episode that you've


run in the past will give you a valid training example for this, your model will simply learn


to match the previous experience with the goals that were achieved in the previous experience.


So there is lots of lots of generalizations here, like how exactly these things are represented.


This time horizon can be a high dimensional object, desire can be, as I understand it,


somewhat a high dimensional object, the extra commands can be like conditionals on these


two things. It gets very complicated, but I want to jump ahead to a different paper


where so this paper is basically just describing the algorithm. And then the next paper is


doing experiments with this. Let's scroll past here. Alright, so this paper training


agents using up that down reinforcement learning released on the same day, but different authors


that have used also, also made who is also here, but have used this to implement a variant


of this. And here you see again, what I was trying to, to explain. So in traditional RL,


this especially here, Q learning, you'll have this function which gets an observation as


input and then Q learning, especially you also get the action as an input, you're supposed


to say, for the given observation, this particular action has this expected value as a return,


right? That's what I explained at the beginning, that's kind of value based reinforcement learning.


Whereas the behavior function here, which would be upside down reinforcement learning


gets the observation and a command and will map that to an action. And here again is what


we've gone over. This is a bit of a different thing. So this agent has apparently run two


different episodes, one point, it did this sequence of actions. And at the other point


from the same starting state, it did this sequence of action. And you can see here on


the right, all the training samples we can we can derive from this. So we can say, from


state s zero, right, if I want to return in one time step, I have experienced this in


the past, right to return in one time step, all I have to do is take action a one. But


if I want one return in one time step, I have to take action a two. And you teach your behavior


function to learn these things to learn to output these actions with these things here


as inputs. And then what you hope, of course, is that this will generalize that it will


learn to generalize that you can say, now, give me more reward than I have ever seen


before, right? And it will kind of learn which things correspond to lower reward, which things


correspond to higher reward, and we'll be able to extrapolate which things will correspond


to even higher report reward, sorry. So they have two algorithms. And this is kind of this


is reminiscent of the old of the old RL kind of world where you do kind of one algorithm


is continuously learning from the experience gathered by another algorithm. So you have


one set of algorithms. And this even in modern RL, this this this is how it's done, right?


You have two different boxes, right? Actually, you have probably one box learning the model


like this is represent this year, learner, right, and the learner distributes the model


to many, many machines interacting with the simulators. And these machines, all they do


is run episodes with the learned model. And they will send back their experience here,


and then the learner can learn from it, and then at the end, send it again. So So, all


right, here we go. So, in each step, what we do, in order to to generate a new episode,


we don't always want to want to kind of execute one given policy, what we do is we sample


from the end of the replay buffer, and the replay buffer is sorted by returns, right?


So the highest return episodes are on top. So we want to sample the highest return episodes.


Then we want to say maybe some of them are 10 steps long, maybe some of them are five


steps long, and so on. So we set the horizon to be the mean of the length of these, right?


And we set the desired return, how much return should be achieved in this time, to be the


unit to sample from the uniform distribution between m and m plus s, and m is the mean,


and s is the standard deviation of the selected episodic return. So what this means is, is


like, here's a bunch of episodes from they start at the same time, here's a bunch of


episodes that I ran right from here is time zero, and then time goes on that I ran that


had really high returns, right? Now, I'm going to take the mean time that these episodes


ran like this, this is maybe five times steps. So in five time, I want to achieve now how


much reward now you look at all the rewards that were achieved. This is maybe a distribution


that has some mean here, like so. And then you say, I want to achieve a reward between


here and one standard deviation higher than here. So right, and this, this would be the


reward you want to achieve. So what you do is you kind of your push your learned model


to just go a bit beyond what it has seen so far is basically say, Look, I you can do this,


but you can just do a bit more in the same amount of time. Please do this. And you hope


the model has learned to kind of generalize to do this. And if so you will execute these


episodes, and then these episodes will go back to the learner, right? We'll go back


to the learner here, and the learner will learn from them. And hopefully, then you can


like generalize even more. And then you can say, I now know how to achieve this bit more


reward. Now I can, if I run the episode, I will achieve even more reward, I can push


the model even further, right. So at eval time, you can always ask the model to produce


as much reward as possible in the given time. And of course, every episode sent back here


is not only one training example, as we saw, but many, many training examples can be derived


from these models, even beyond what's in what's in this paper. All right, so I think this


was a good first shot at describing this algorithm. I hope you get the gist of it. I enjoy this


a bit of a criticism for me would be it's still kind of doesn't it so it doesn't touch


the exploration dilemma. So it again, deals with kind of incremental, incrementally getting


better. Whereas I feel this can easily get stuck in some minimum where it's not possible


to do this incremental generalization of the model where you really need a new approach.


And that's why games like Montezuma's revenge are solved using algorithms like go explore


and not any of the classic algorithms. That being said, they have experiments where they


show that especially in sparse reward environments, they do better than classic RL algorithms.


So if you for example, here take the lunar lander, where A to C beats upside down RL.


I guess you didn't get matplotlib to do the upside down RL. Well, the in other get in


other environments, upside down RL clearly beats the classic algorithms. And what I like


here is they took a lunar lander. And which, which basically at every time step you get


a reward in lunar lander and they hypothesized, okay, this is really good for these classic


algorithms that do reward maximization instead of kind of learning this general behavior


function. And what they did is they modified the game such that all the reward is given


at the end of the episode. And then you see that upside down RL will actually outperform


here the classic things where it's exactly the same game, you just get the reward at


the end. So upside down RL kind of learns the structure of the world learns that you


get this reward at the end after such and such many time steps. So you can, it will


learn, please get me zero reward in 50 time steps, like no problem, but please get me


1000 rewards in 100 time steps. No problem, I just go to the end of the episode, right?


Whereas these pure reward maximization techniques, they don't they somehow have a harder time


to do that. I like this investigation. I like the thinking outside the box. The Schmid


hubris of the paper. It's just all great. It's a great time to be alive. And check this


NFNets: High-Performance Large-Scale Image Recognition Without Normalization (ML Paper Explained)

Hi there, today we're looking at high performance large scale image recognition without


normalization by Andrew Brock, Soham Dey, Samuel L. Smith, and Karen Simonian of DeepMind. This


is otherwise known as NF nets, normalizer free networks. So the point of this paper is to build


networks, in this case, specifically convolutional residual style networks that have no batch


normalization built in and we'll get to why in, you know, during looking at this paper. But without


the batch normalization, usually these networks are performing not as well, or cannot scale to


larger batch sizes. However, this paper right here builds networks that can scale to large batch sizes


and are more efficient than previous state of the art methods. So if you compare them to something


like an efficient net, and I called it, I called it, you shouldn't call your model efficient net,


because a more efficient model is going to come around. So NF net are now officially


efficient or net, okay. Yes, you can see right here to reach the same accuracy as an efficient


B seven, you need, I think they say they have an over 8.7 x speed up, if you look at the training


latency, and that's going to be important while looking at these experiments in a second.


And if you train for as long as the efficient net B seven, you can reach a higher performance,


this is image net top one accuracy. And this model is a new state of the art without additional


training data. And it is also a new state of the art transfer learning. And it is the currently


ranked number two, behind a method that uses semi supervised pre training with extra data.


So in the kind of global leaderboard, it's number two, but it is number one in various categories,