详细总结下面的内容，中文回复
(18) 🍓 "As useful as a GOOD grad student" ― Surpasses Human Capability on Many Tasks (but not all!) - YouTube
https://www.youtube.com/watch?v=SDwxMCeBNIQ

Transcript:
(00:00) all right so how smart is strawberry or 01 preview I have been kind of taking it all in like drinking from a fire hose the last couple days and there's been a kind of a constellation of of facts or drip feeds or opinions that I've been taking in that has really kind of made me sit up and take note so for instance here's one tweet the most gifted mathematician has rendered his verdict gp1 is like a mediocre but not completely incompetent grad student student um so that's you know okay so think of it this way top top
(00:36) mathematician in the world according to someone's opinion um says that this new tool which is just the preview version just the first version of this new paradigm is like a mediocre grad student now a mediocre grad student is probably going to be a little bit more intelligent and a little bit more useful at least on uh some Fields than your average human your median human um you know without an education cuz I think it's like what about 11 % of the US has a graduate degree something like that or maybe that's a bachelor's anyways so
(01:05) you're already pre-selecting that's one clue now I wrote about how uh postdoc uh friend of mine uh texted me after he started testing 01 preview and this is um a friend of mine who is a PhD researcher um in the field of oceanography and cfd computational fluid dynamics and he said oh yeah like it almost got it right the first time and he he showed me the prompt I did not understand most of what it said because it was so jargon heavy and with just a little bit of encouragement it it got it right and then I had
(01:39) someone reply um I just had 01 write a major cancer treatment Project based on a very specific IM immunological approach it created the full framework of the project in under a minute with highly creative aims approaches and even considerations for potential pitfalls and alternative strategies and he said this is truly missing in 40 uh this would have taken me days if not longer to prepare and I still would have missed several 's points in fact there were at least there was at least one unique idea that I might not have thought of um even
(02:06) if it even with 30 years of deep expertise in this field um people have no idea what's happening and again this is all just preview so I ultimately wrote multiple postdoc researchers have now said that uh 01 or sorry yeah 01 preview or strawberry is about as useful as a good grad student now that's that opinion is not universally shared but you know people are now comparing it to grad students which is actually what Leopold Ashen brener uh predicted with his situational awareness paper so it seems like maybe there was more
(02:40) substance to what he was talking about so we're going to revisit that idea now uh before we proceed because basically the whole point of this is just like how how smart is this thing and so um I've been Gathering evidence on my raspberry repository which by the way if this is as smart as it seems to be then as a service to humanity we need to open source it with or without open ai's permission um so we're and and it's not just me a bunch of a bunch of people have come out of the woodwork saying
(03:09) that they're trying to duplicate this work so it's going to happen um so first thing is uh I've been collecting people's uh data basically uh you know kind of crowdsource data on in terms of how smart is this thing and so on this test we see that it is now in a completely new league um so we went from you know median 25ish to 30% performance um to looks like 85 is% performance whenever you see a leap like this that means that you've had a breakthrough paradigm shift in how machine learning is approaching a
(03:47) problem um and so once you get into like 95% um it's basically a solved problem the last 5% usually takes many years to close the gap but at that point people are not interested in I want to say not all people many people are still going to be interested in getting that last you know 2 3 four 5% performance but where but where you get rewards for your papers is when you go from 25% when you have a 40% uh uh performance jump that usually is a signal that you're moving in the right direction I remember back
(04:19) in the day when I first got into machine learning when uh uh XG boost came out and it crushed a whole bunch of benchmarks at the time it said you know a bunch of ml competitions went from 30% to 60 to 80% and they're like whoa okay cool this is clearly the right direction and then the next year everyone has uh incremental improvements so whatever open AI has done with strawberry this is clearly the correct approach because of how broadly it generes generalizes because first okay a math a world-class mathematician stress tested and he's not
(04:54) super impressed but he's also a world-class mathematician um other people in other domains so remember computational fluid dynamics mathematics and medical research three very different domains then compare it to this New York Times t uh uh puzzle and then compare it to this other uh kind of thirdparty Open Source IQ test 01 is kind of right in the middle of the pack in terms of General human reasoning so if it's as useful as a grad student and has basically achieved median uh intelligence in terms of general purpose intelligence guys
(05:34) this like you could make an argument and and I I'm not going to say this is definitely AGI but I think there's a good case that this is pretty close to AGI and then if you watch Philips Channel over on AI explained it looks like he hasn't updated simple bench yet but it is also in a new category where it's around 50% accurate I think he's still grading it to get the the final number but first Bluff is that it's about twice as good as Claude 3.
(05:59) 5 Sonet in terms of uh Philip simple bench reasoning so it looks like across the board simple Ben or sorry 01 preview uh or strawberry is across the board more intelligent than every other product out there today which means great okay cool like we we now know that the next process and so what I want to do is say is point out um this is the data from open AI which okay their own internal benchmark like that's one thing but give but getting pragmatic like practical feedback from real people solving real problems cuz like okay yes people's
(06:39) opinions and benchmarks that's that's one thing benchmarks are certainly better than just someone saying it's the best model out there I don't really trust the value from lmis anymore at all um so you've got benchmarks then you've got or in terms of in terms of value it's like um LMC pretty low value in terms of figuring out which models are actually smartest benchmarks are itely better but then ultimately it's the Practical impact what impact does it have on science what impact does it have
(07:07) on the economy and so what what I'm pointing out here is that all of these opinions that I'm showing you this is all 01 preview and it looks like 01 which comes out in October I think is another full uh standard deviation more intelligent so let's just say let's just imagine that the IQ is is exactly 100 that means that the IQ of 01 is going to be 115 because that's about one standard deviation more I think it's like 13.
(07:39) 4 or something like that is technically one standard deviation of IQ so we're going to get almost as big of a jump so we went from 13 to 56 and then going from 56 to 83 so we're getting almost as big of a jump from GPT 40 to 01 preview to 01 um now that's not going to be across the you see it's about half a big as jump on coding but it was already really good at coding um and then here on um on PhD level science questions 01 preview or sorry 40 good grief sorry I'm still so horse I don't know why 40 was down
(08:17) here lower than a human expert but both 01 preview and 01 are both Superior on GP QA uh Diamond so that's all very interesting and where I wanted to close was I wanted to revisit this graph that Leopold Ashen brener uh put out so here it's like you know gpt2 was a preschooler gpt3 was an elementary schooler um I would say that it was probably a little bit smarter than that with fine tuning um gp4 uh may you know maybe maybe Baseline gp4 was about that was about that uh or maybe chat GPT yeah uh a well a
(08:54) well-trained chat GPT was was as good as a smart high schooler um and so here we are we're halfway between 2024 and 2026 so we're about right here and so we're now at the point where people are no longer comparing the GPT technology to high schoolers we're comparing it to grad students um which means one more step and we'll be cons we'll be comparing it to um postdoc researchers uh and then one more step after that and we're going to be comparing it to worldclass researchers now there's a
(09:24) couple of assumptions that that makes that's making the assumption that this Paradigm does continue to scale but as as people have pointed out like Dave you were kind of wrong when you said that it's slowing down and I'll clarify that point in just a second but the the thing that really made me sit up and take note is that where gp4 started was as a smart high schooler but o 01 preview as far as we know strawberry is still trained on GPT 4 it's just a new training Paradigm so if a new training Paradigm can can bring it
(09:57) up several standard deviations of intelligence that is something that I didn't really fully anticipate in terms of uh functional pragmatic intelligence um and I still have people in the audience somewhere writing comments like well it's not actually intelligent it's not actually thinking that's a no true Scotsman fallacy which basically says you know you can argue about what's going on internally like you know the the Cog work upstairs but ultimately it's the output that matters it is is
(10:26) the output uh economically viable scientifically VI valuable um or otherwise meaningful to humans and if the answer is yes it doesn't matter whether or not it's truly thinking or truly intelligent um that's just cope like that's just that's just a denialist cope so anyways so if within one generation gp4 now granted this is over the last two years that it's been out and I think what was it gp4 was trained uh about 6 months before it was launched so open AI have had GPT 4 for about 2 and A2 years but between rhf rla RL aif
(11:03) and all the other you know constitutional AI all of those other techniques and then whatever this new technique is which they've all but confirmed it's a combination of Chain of Thought plus reflection maybe a couple other things baked in uh a lot of people sus suspect Monte Carlo tree search so basically they tuned a reward uh a a reward predictor that instead of rhf and rla aif or maybe this is is just another form of RL aif um but then you bake in Chain of Thought you bake in reflection and then you bake in Monte Carlo um and
(11:38) that gets you another order of magnitude more intelligent on the same generation and with all that being said they're about to release GPT 5 so that's going to be another model another paradigm shift in terms of model capability because I I've been there since gpt2 I trained gpt2 and quickly found limitations gpt3 took a much longer time to find limitations and it looked looks like we still haven't found the upper bounds of what GPT 4 is capable of even after 2 years so the latent space in terms and I
(12:08) don't mean latent space activation but I mean the latent abilities that we're still unlocking with gp4 if we're still unlocking abilities after two and a half years how much longer are we going to be able to unlock those those latent abilities of GPT 5 if that gives us a much longer Runway then that means those diminishing returns that I've been afraid of um with respect to AI slowing down might not matter as much because if if this new training approach results in people saying wow 01 preview could have
(12:37) saved me days and days worth of cancer research how much when you combine that with Alpha fold from Deep Mind and Alpha Proto with deep mind when you go to when you get that up to 01 and then 02 you know whatever comes out in December whatever that Sam mman has hinted at I think there's a very strong case to say that that AI has really actually P most human intelligence right now in terms of economic or academic or scientific uh utility and so this is why for many years now um I I wrote it in my first AI
(13:11) book um arguing over the semantics of what is true intelligence or the Phil philosophy of what makes humans better that is a complete red herring and I think what this proves is it's even more of a red hering than it was before I'm not saying that those debates shouldn't happen but what I'm saying is that those those conversations should not happen as a way of stying discussions of what is the actual scientific and economic impact of these of this technology from here on out um because again if we just
(13:42) saw something that that blasted through the barrier of hey there's you know like yes it might cost $1,000 a month or $2,000 a month right now um but if it's as smart as a good grad student and is about to be much smarter that's worth that's probably worth the juice might actually be worth the squeeze and I very well could have been wrong when I said the ju juice wasn't worth the squeeze so anyways I just wanted to share my thoughts that have been fir fermenting and percolating since uh since getting
(14:10) all this and collecting collecting what is what is the group consensus um I I do think that um well actually no I I'm going to bite my tongue because I don't want to ramble too much so anyways thanks for watching I hope you got lot a lot out of this uh cheers have a good one this is man I what was was I right was I right that we did achieve AGI in September 2024 I wish I had stuck by my my argument now um but it's it it ain't over yet we're only halfway through September so I've got 15 days to make a
(14:42) judgment and say whether or not we got AGI all right I'm done bye