Provide a comprehensive summary of this video:
      - What are the main topics covered?
      - What are the key points and arguments presented?
      - What are the important conclusions or takeaways?
      
      Organize the summary in clear sections with main points and supporting details.

Transcript:
hey welcome back so this week the world has been going absolutely crazy about the new deep seek R1 model now if you haven't heard of it before it is a open weight reasoning model from a company called Deep seek in China now the reason everybody is going crazy about it is they open weighted a model it is available on a Lama it's available in hugging face you can run it on your machine and they released a paper explaining how they actually trained the model and it's an absolutely fascinating real and it is fair to say the results from that model are incredible in fact if you check out one of my videos from last month you can actually uh see a full breakdown from me on that video but we're not going to focus on that today we're actually going to focus on the part of the paper where they bootstrap the model with long chain of thoughts to elicit reasoning from the model and it is such a cool part of the paper in fact what we are going to do is we're going to replicate a little bit of that part of the paper per and I'm going to show you how you're able to do that yourself if you want to get started with it quickly and then just experiment a little bit you can go to Just chat. deep seek.com it's got a sort of uh chat interface that you can interact with so if you then just click on the Deep think R1 uh button there then that will invoke the reasoning part and I can just type in something like solve 91 minus 52 uh plus 990 and again this is a kind of very simple uh math arithmetic thing and and again I covered this in my other video but the key thing about this is it invokes this sort of reasoning now it's a fairly unstructured set of reasoning and they actually talk about that in the paper which we'll go through in a little bit but you see there it says first I need to solve this expression 91 -2 Plus 90 blah blah blah blah blah little bit crazy and then it will come back with the final answer and again I can uh kick off a new chat and I can do something a little bit more uh complicated so we'll say 2187 plus uh 9639 and then we're going to multiply that by uh 9466 a little bit uh more complicated but again it should be able to pick this up and you see it's generating its uh Chain of Thought reasons there key thing is there it's quite long okay let's track with this problem H I first need to remember the operations and then it's just going to sit and think about it and generate these tokens and that's the sort of test time compute or inference time compute that open AI uh is talks about and you see it is generating a lot of tokens underneath the hood it's working through Alternatives and then in the paper it can even sort of reflect when it gets wrong eventually after a while it's going to come back with an answer and you see there it's figured out the answer is 7054 632 and then uh gives the final answer in some pretty markdown again if I go into chat GPT for a second and then we paste the exact same question into chat GPT and then you can see unlike the Deep C car1 models chat GPT hides the reasoning the token the think tokens that that you get there I can click on this and then it gives you a little bit of details of summarized version breaking down the steps breaking down the calculations combining the results but as you can see both H come back with the same answer now of course if I want to I can always download the models locally onto my machine via something like a llama so if I go to Al llama.com Library click on deep seek R1 you can see they've got the small versions of the model 1.5 billion parameter model all the way up to the 671 billion parameter model it defaults on the Deep seek r 17b model and again you can just pick whichever one you want to be able to work with but once I have a Lama on my machine I can just do L Run Deep seek uh- R1 and if I want to I can ask the same question so we'll maybe do the uh 2187 blah blah blah question there and then again it's going to go through that sort of same generation of tokens but the nice thing about AMA is uh I can read it a little bit more here it says think I need to calculate blah blah blah blah and then uh everything within the think tags is actually those reasoning steps now you see it's doing a lot less reasoning than the models up on uh deep seek.com probably because they're running the bigger models there and therefore uh it's going to generate more reasoning tokens as you say eventually it comes back with the uh question and then it says the answer is 7042 272 as you may have GED other than this particular case it's actually came back with the wrong answer and mainly it's going to come back with the wrong answer because it's a smaller model and actually it's just generated less tokens if I want to I can give it a slightly simpler calculation which is the solve 91 - 52 + 91 that we did earlier and you can see it's generated into tokens and this time it will come back with the right answer 129 so again you're going to get different results based on the size of the model that you use etc etc but you see it's pretty good you can run it locally okay so now that we've got a quick view of how the basic math actually works with deep seek let's actually take a quick look at the paper and see what they've done so you can download that it's on archive but you can also just download it from their GitHub uh which is what we're going to do and as you see it's a really fascinating paper you can just go through that uh and read it um the key thing that I want to sort of point out here is in uh section two where they talk about the approach that they did on the model so the first thing is that they did reinforcement learning on the base model so what they did is they took a pure base model so not a chat model not instru model they took a base model without any sort of supervised uh data so no supervised fine tuning um which is really incredible and then they trained it just from rewards on the model from scratch so they took the base model and then did reinforcement learning uh over the top of that and the model learned for itself um how to generate these lawn chain of thoughts and and that's pretty cool in itself and again they give their reinforcement learning algorithm and again you can kind of go through that I will probably do a video at some point in the future where we will train reinforcement learning for ourselves and replicate what's actually going on here and what's actually cool during that reinforcement learning process and they give a bit of a view of how they've done it is in the paper they have a a nice little prompt template that they use whilst training the model to invoke chain of thoughts and it's actually pretty simple which is a conversation between a user and an assistant the user asks a question and the assistant solves it the assistant first thinks about the reasoning process and impr provise the user with the answer the reasoning process and answer enclosed within think tags and answer tags respectively I.E think reason process here answer here and then they put user and this red prompt that's where you would put the specific question that they want to ask during reinforcement learning and then the assistant is going to Churn it out now remember they don't have a base model at this point so they the the model hasn't been tuned to understand chat templates so what they are doing here is they've have created a mini template to invoke user assistant prompts that you would sort of expect and then they're just guiding the model to generate these think tags and these answer tags answer is what the user sees think is what is used for the reasoning process and again they've not fune that with any data they're just guiding the model to produce the right results and then they're using reinforcement learning to come back with those results so it's never seen any long Chain of Thought for prompts before so I think that's super cool in itself self because then what they do is once you generate those chain of thoughts from the model whether they're right or not is a different question but they use sort of reward modeling to uh uh to actually determine whether it was a good Chain of Thought and the way they do this is through accuracy rewards and format rewards so for accuracy evaluates whether the response is correct for example in the case of math problems with deterministic results the model is required to provide the answer in a specified format EG within a box enabling reliable rule verification of correctness so you can sort of imagine if you were saying like in the example we had if you do 25 + 2 they are going to run that through a piece of code run the calculation and it's going to come back and say seven so they will have in there little uh answer format it will just say uh what the answer is so like 27 and then they'll run that through their own verifier and then they'll check that the answer is correct if the answer is correct they'll give it a reward um and again it as you can see here from the paper it's saying similarity for leak code problems a compiler can be used to generate feedback based on the predefined test cases so essentially they're going to do a compile you get a reward if it compiles and then they'll run the elite Code test cases if it comes back with a successful answer from the test it's going to get a reward as well and then the last thing is they're rewarding for is format rewards in addition to accuracy reward model we employ a format reward model that enforces the model to put it thinking process between think and think tags now what we're going to do we're not going to do reward modeling we are going to do sort of generation of chain of thoughts but I'm actually going to show you how this works in this video so actually we are going to train a base model to generate those think tags uh and to elicit reasoning and we're also going to train a model uh to be able to do verification so I so that whole answer thing that I showed you I'm going to show you exactly how to do the same thing uh within this video so that's that's pretty cool they did that from scratch really great we'll do reinforcement learning on another video but actually the thing that I really want to talk about and this is what we are going to do in this video is they then did this thing called cold start so they said here inspired by the promising results deep seek r10 uh two natural questions arise can reasoning performance be further improved or convergence accelerated by incorporating a small amount of high quality data as a cold start so what they're saying there is rather than letting the model figure this out during reinforcement learning they want to speed up the reinforcement learning process by saying I'm going to get a small amount of uh supervis fine-tuning data I.E I'm going to get a set of prompts that give me roughly the answer that I need and then I'm going to use that to sort of quick start cold start the model and then I'm going to take over reinforcement learning from there and they're also saying that it gives a second benefit that if they can sort of cold with those chain of thoughts that they've pre-trained before then it should give more clear and coherent chains of thought than previously remember when I did the Deep c car one video uh last month one of the things that we said is that the chain of thoughts were quite crazy well actually they're sort of saying they want to get even better uh you know chain of thoughts generated and they're using this cold start technique to be able to do that and we're going to replicate this cold start technique in this video in about 2 minutes so as you as you can see there unlike deep seek r10 to prevent the early unstable cold start face so what they're saying there is that when they did reinforcement learning from scratch um and having the model use that template to generate the long chain of thoughts they were getting unstable answers and it took them a long time to eventually get chain of thoughts in a in a format that they needed and they think by using this small amount of fine-tuned data they can therefore elicit better results from the beginning and come back with better results quicker and again as I said I've reproduced this and I'm going to show you this in about 2 minutes and how you can do it for yourself and then they say here in this work we collect thousands of cold start data to fine tune remember what I said there right most of these models are trillions of tokens to be able to train the the base model but and even in fine tuning phase it's usually millions and millions of tokens they're saying for this this cold start phase thousands and then finally what they're saying there is once they've done the cold start phase uh as a sort of supervised fine chune then they're going to just jump back into reinforcement learning and it should be able to pick it up quicker from there okay so the first thing that we need to be able to do is generate long chains of thought now they didn't specify in the paper how they generated these long chain of thoughts that are accurate but we can probably simulate this ourselves right because the problem with things like math is that the steps the chains of thought need to be accurate now if the model's not any good at doing math in the first place so maybe something like the kind of 01 models or maybe you got a 400 billion parameter model or something like that it's going to be not bad at math but to be honest it can still get things wrong and yes you're going to use a verifier and then sort of You Know sample reject anything that's got the wrong answer you're going to remove from your your list so you can generate chains of thought that way just you know just generate the chains of thought and then anything that's not right remove it and then have a list of prompts for your fine tuning that is only accurate but I think we can do a better way of doing that we can because we can use compilers and we can use abstract syntax trees to be able to generate steps within maths then we can actually generate an accurate set of chain of thoughts ourselves so uh I did a video earlier last year which was on a thing called a math compiler and what it actually did is generate a a synthetic chains of thought for m so we are going to use this to be able to generate the long chain of thoughts that we need so if you you can go on my GitHub and you can it's the the folder's called Chuck math I'll put it in the links you can just download and run it but you can kind of see what I mean here so if I T type in something like uh python main.py I'm in my Chu maath folder and then I do 91 minus 52 uh Plus 90 um if we run that for a second it's going to generate a whole sort of um uh it's going to generate things like some instructions so some sample questions um you can see there it generates out the tokens of the abstract syntax tree but the key thing is here is it generates out an explanation now it hasn't done anything with the large language model I have a whole video explaining this but um I just want you to basically see that it goes step 0 91 minus 52 equal 39 Step 1 is 39 + 98 = 129 and and that actually uh is the key thing I can just generate out these steps as I need again we can do something a little bit more complicated so we can put parenthesis in here so we'll say 2187 and we'll add that - 9639 so that gives us a little bit of a uny I'm going to close the parenthesis then I'm going to do a multiplication and we're going to say - 9466 uh we will just close that off and we'll run that for a second and then you can see in the explanation step 0 9639 - 0 is equal 9639 Step 1 is 2187 plus 9639 and it gives us the answer there step two is it equal there da and then eventually gives the real answer which is 11944 916 so the key thing that I've got here is I've got the calculated answer and then I've got the steps and again we can use this to generate the uh long Chain of Thought there because we've got the steps but there's just no explanation behind it at this stage so now all I really need to do is actually get a large language model in on the business as well so if I just uh do minus minus llm and then I put something like 54 which is kind of Microsoft's new uh uh model and that's available in Alama as well we'll do we'll do a couple of different models there here so what we're actually going to ask the llm to do is not to calculate what the actual sum is for itself what we were asking it to do was say take the steps that I've just given you and what I want you to then do is augment that with an explanation so the llm doesn't matter if it's good or bad at math uh that doesn't really matter because we're going to provide it the answer and we're going to provide it the steps and we're going to ask it just to go and figure all of this out so you can see here it's come back and it says the calculation involves two main steps uh addition start with this blah blah blah blah and then you see for the result it it's basically saying think and I'm going to show you why I've got the think in here in a second so you see this think problem resolve this and note the result reasoning start with the expression according to PEMDAS do this this this and this and then eventually it's going to come back with the right answer so 705 406 32 so I've given it a nice complicated uh sum for it to do but I've added in the llm and then it's going to reason how it got to that result and again we know the result is correct because even before the llm you see in this part here result 7054 0632 is the same name answer that the llm has got and then what I've done here is I've put the verifier answer uh within its own verifier tag so later on if I want to go and parse that I can do this so I've got my uh reasoning steps So within here I'm telling it to do its reasoning within the think and then I'm going to be able to provide the answer in the answer tag so you see in the answer tag the result of blah blah blah blah is 7054 and then the verifier answer where it's cleaned the B the kind of Preamble has this uh Final Answer here so it's pretty cool and again I can do this with not just the 54 model um so if you're worried about the you know there's a lot of models you're not allowed to use to train uh other models but if I want to I can pick a model with a clean um license so in this case I could pick the kind of uh the granite 31 uh dense model and the reason I'm picking that is on Pache license so therefore I can just do whatever I want with the outputs of that model but again you can use anything I use 54 earlier you could use llama 3.3 you could use the quen models all of that is equally fine you just put in a different llm and then that llm is simply because it's not responsible for doing the calculations it is just responsible for doing the translation then it's going to come back with that answer so again here this one was the granite model and you see the result same result as both four and now the granite model's got its steps think what is this blah blah blah here's its reasoning and then uh it's worked through all of the steps and then it's giving me my answer my verifier answer if we want to we can actually look at the uh prompt template that that powers this and you can see here I've got a math stepbystep uh prompt template that we've got here and you and again it's very simple for the problem question involving the expression blah produce a set of uh steps there and then there's a few things I've added it do restate the problem querly using the think tags break down the solution into logically ordered blah blah blah I tell it to use things like pmaz conclude the reason with the final answer inside the answer tags and then for that verifier answer I'm saying include a verifier answer section containing only the result formatted for automated systems all reasoning must be within the think tags no reasoning should occur should occur outside those tags and then think and then there's the problem and then the reasoning and then in the tags final answer and then a clean answer used for verification and again I've also added in a recheck the steps and answer if there is a discrepancy now this is now this whole reflection thing is really fascinating so one of the things until I got this improved template is actually I had an earlier template which would uh occasionally come back with the the wrong results because the uh the steps weren't as clear until I improved it but because I had this you know go recheck and reperform the reasoning it was actually able to correct itself within the uh prompt and again even in the data that I generate anywhere where it gets it wrong or there's a discrepancy in steps it can actually reflect the answers back and now I'll demonstrate this with the slightly older template so I'm just going to change in the code here the template name from math step-by-step template to this slightly older one and now we are going to rerun this code I'm going to use a slightly different sum because I'm going to use a sum that I know that's going to invoke this issue so in this case I'm going to put 8895 + 631 uh divided by - 659 uh plusus 952 so this is a a little bit of a a tricky one there we're going to run this again against the gr 31 dens now as I said this was a version of the template that was slightly wrong and and didn't invoke the behavior they wanted for generating these prompts but look at what it does in the middle of this now as you can see here it started with the think and it's doing its questions there but this is the key part here right it says this gives us approximately 9847 9575 and the reason it's got it wrong is it did a wrong minusing a little bit earlier and then it's just said here however upon reviewing the steps I noticed a discrepancy in the final calculation let's correct this by recalculating the last step accurately recalculated step three adding the results from step one and two correctly gives us approximately 7942 0425 and then and then you can see the final verifier answer is 79 40425 so it has reflected the answer and then got it correct now if I come back into the deep seek paper the thing that they were talking about is during the reinforcement learning process is this thing called aha moment so you see question if a is greater than one and Su of the blah blah and again that's more complicated in my example but we can use the same technique to generate longer change of thought this way and then you see wait wait there's an aha moment I can flag let's re-evaluate this step by step to identify the correct sum so this is why I include some of this in my chains of thought here so that I can then have these uh sort of ah moments or these reflection moments and you see it sort of works uh perfectly in that sense now if I just switch the template back to the old template it will still reflect it just happens to be for this particular question it actually gets it right straight off however in the training data set that I generated there are reflection moments and I'll show you them in a second and there we go you can see there with my slightly better template that it didn't need to reflect the answer because it was able to get the answer uh straight off itself without doing anything so now that I've got a really good way of generating synthetic chains of thoughts one of the things that I I can do is then just generate them out so just to be super quick here what we will do is we will I've got a script within the uh the math compiler which is called generate chat samples I am just going to change this to maybe uh three samples there to make it a little bit quick uh I've got different levels of difficulty very easy hard medium Etc I can say which llm I want to use and then I'm just going to Output it into folder chat samples and we'll just we'll just change this rather than it's saying very easy I'm just going to say record and then what's actually going to happen is it's going to automatically put it into a chat format that I can use for CH fine tuning so it will generate three samples I'll show you what that looks like and then what I'm going to do is show you a larger data set that I pregenerated and we're going to then use the larger data set to fine tune okay as you okay as you can see it's completed that so if I just come into uh Visual Studio code for a second and we go into outputs there's chat samples record and you see it's generated in the standard chat format messages rule user what's blah blah 76 plus 5 for the arithmetic problem blah blah blah blah so it's done exactly what I wanted it's it generated some samples it's created random Expressions because it can randomly generate uh math questions and then uh it's then done what we did before it's ran it through the llm with all the explanation and steps it's generated those think tags with the problem the reasoning Etc and then uh it's got those synthetic long uh Chain of Thought so again and I know the answers are going to be correct and I know they're going to be correct because the answer is coming from the uh the math compiler not from the llm the llm is just responsible for doing those steps now what is very cool about this is when I run the uh chat samples earlier you can see I generated in this case i' I generated uh 20 easy ones so again just using the same process that we did there I generated hard ones we didn't use hard in the data set cuz I thought they were a little bit too complicated um but again if I just look at the chat samples medium this is the first question which was calculate minus blah blah blah blah but if you look here uh just around halfway through there it discovers an error and then uh it then says okay I need to recheck the steps sort of discrep discrepancies and it says however upon reviewing the calculation it appears there's an error and then it fixes that error and generates a step so so I generated a whole loot of samples like 120 or whatever different s samples there and then I sort of put them all into this merged file so I have 120 different math samples with long chain of thoughts and that is what we are going to use to fine tune so now that we've got 120 rows of math data in the standard kind of ch format one of the things that I need to do now because I'm working with the Quin model and because the Quin model as a base model uh doesn't understand chat at this point I wanted to simplify things a little bit and rather than leaving the mlx fine shun to be able to do the completions uh one of the things that I did is convert it into the final format for fine tuning myself so I have a little script here called uh convert Json L and all it does is take that format so messages Ro user assistant and then converts it into uh a line that the mlx fine-tuning is going to understand so you see in this case is text and then it's got a little token I am start and then user user content I am end I am start assistant and then assistant content so it's just converting from this format into this format here which the coin model which we're going to use as our base model will actually understand so then what I did is I just paste that converted version so so it's the same 120 that I showed you earlier and I put it in this coold start folder called math with reflection and you see it's just I am start blah blah blah so it's basically the output of our generated uh uh chat format y so that is 120 rows of data which is all fine tuned and this is going to be my cold start data and then I take um this file here which is uh I call it new Json l in my apple mlx fine-tuning video that uh I released last week um I have what I call a mix data set which teaches the model how to answer other different types of question so I've removed the math type questions from there but you see it's got stuff like write a short uh Horror Story in 2 minutes tell me a joke about coffee what is the capital city of Japan and then it's going to have a bunch of answers the reason that I have this file is just so that the model doesn't forget how to perform certain tasks this is a bad Jason L and the reason it's pretty bad is that all the answers are fairly short and because all the answers are fairly short when I ask the model questions like who is a a love lace or whatever rather than give me a long answer it's going to give me a very short answer so in the future what you would want to do here is build up a good set of mixed data set like the same way deep seek did so that when I have eventually have a cold start data I have good math completions and I have good ones that do summarization I have good ones that write short stories write poems ET Etc and then eventually and all them invoking long chain of thoughts and then eventually if I get enough of a diverse data set then um I'm going to be able to cold start today we're just doing the math piece I haven't done code or anything like that but I think it's going to be good enough for what I want to show so then I run a little script which is sort of a merge script and then what the merge script does is it takes merge completions and it takes the new jonl and then it uh puts the two things together and that and and sort of shuffles it and now I have that sort of mixed data set all with all the various questions you see there who was Albert Einstein Etc but it's mixed in with those math questions calculate this and then it's got that long Chain of Thought Etc within there and I can scroll across and you can see wherever the long Chain of Thought is this is my merge Json L and that's what I'm going to use to fine-tune and again all of this data is up in my GitHub so you can go and utilize it for yourself okay so now that that's done we are now going to try and interact with the various models so I am going to do a UV run uh minus M mlx LM uh Dodge do chat and and again in my uh other video so in the um uh mlx fine shuning I go into this in great detail but in this particular case we're going to attempt to chat with the quen 2.5 1 and A2 billion parameter model this is a tiny model 1 and a half billion parameters right and I'm going to only get Max tokens of 500 back so we're going to enter into chat mode and then the first thing you're going to see here is that the model really doesn't understand anything so if I say what's 8895 + 631 uh divided by - 659 plus uh - 952 I'm and again it's you can just see it's nonsense the model just doesn't know how to do anything it doesn't it's not been trained on chat format it certainly can't do uh math what we are now going to do is take that 370 lines of bootst started data uh the C start data and then we're going to fine tune it with it so to run the fine tune I'm going to do a UV run mlx lm. Loro again it comes part of the Apple mlx the model is going to be that 1 and a half billion parameter model model that we said I'm going to train it uh so that's minus train I'm going to use the data set data set cold start I'll just show you that one more time the learning rate is 1 E minus 5 that's a sort of standard uh learning rate that you use for fine tuning I'm going to run it through 300 iterations and then I'm going to do a full fine tun again I break it down in my other video and again just so there's kind of no Shenanigans there if you look in the data set call start here is that same train data set 300 it's got around 380 in this case yeah but that's the same data that I was showing you earlier it's that mixed data set um you know so nothing too interesting there it also needs a test Json L I've just put some random uh you know uh questions and answers in there and a valid one so and that's so it can do its training uh validations so it's going to pick up these three files here all you're going to do is copy and paste the stuff that I showed you a little bit earlier um and then we are going to train that model all righty so I'm going to kick off the train on there so you see it started the training there um it's going to do 300 uh iterations there um validation loss is 4.9 at the moment you see in my machine it's quite a small model so it's training at around 600 tokens per second you see that training loss is going down uh as it does each iteration it's you know at 600 700 tokens per second it's not going to take very long to complete the train in this model and again this is because it's a super uh small model there you see here Peak memory usage is is around sort of uh 16 to 26 gig so if you're running this on your machine it might be a little bit too small yeah so if you're running this on your machine it might be a little bit too large for you you can always use a quantized model or you can use an even smaller model maybe the half billion parameter model and you'll be able to do the same thing I want to use a full train I don't want to be doing any Laura training I'm not purposely uh using a quantization etc cuz I want this to be as close as is possible to uh fine tuning a full model all righty as you can see there it's literally just finished training there so you can see add through the training runs that training loss has went down it's got better and better as time has went by and then eventually it's came down to sort of 0.7 so if you remember my file had close enough to around 400 rows of data in there um if I am doing four rows of data as a batch size which is the batch size is four and it's doing 300 iterations then that's roughly about 1,200 so that means I'm really processing um around three runs of my data so I'm processing my data three times or three epochs again ideally I would do one EPO one run of data so I would generate enough data with a nice balance so that I can only do that run one time but again I'm being super lazy uh in this sense so I have a trained model now uh if I uh just clear my screen for a second you see within this folder here adapters so if I go into adapters you will see these are the adapters the this is the difference between the old model and new model and then they get combined together to form my new model again I cover that in my other video but now if I want to we can chat with that model and a way that I chat with that model is exactly as how I did before so I do UV run uh minus M mlx at chat so it's no different there I pass in my model which in this case is the qu 25 1.5 billion parameter model which I'm using as my base model max tokens in this case is 500 and then I just need to add in this extra thing which is D- adapter Das path adapters and that's telling it to go load the adapters from the uh the other file so if we run that for a second it's going to bring us into our chat mode and now I can ask it questions like I did before so if I say what's 8895 plus uh 631 uh / by - 659 9 plus 952 so if we ask that question and you see now it's gone and done it right I mean this is incredible right this is a one and a half billion parameter model who didn't know how to do anything 5 minutes ago but look at what it's done here think problem what's this reasoning Begin by addressing a division operation blah blah blah calculate 631 comes back with the value substitute so he's doing all the reasoning within the think it's came back with the answer 7942 0034 conclude with the result puts the answer in there and verify our answer I mean that's incredible this is a 1 and a half billion parameter model and now it's reasoning up at this level now this is bootstrap again I I can ask it other questions like who is Ada love lace um it shouldn't do to thinking in that sense you see a love lace was in a Pioneer blah blah blah blah BL it's not a great answer it's not a long answer remember in my uh in my mix data set um I I was given short answers so it's coming back with short answers so if I was to build this up with verifiers and C um um starting data sets for those who is type questions for summarized type questions I would need to have equivalent chains of thoughts in the same ways I've done with those math questions so I really start from the math ones and I do the same for code and then I just build up and that's why you end up with like a thousand rows of data because if I can get to a thousand or a few thousand rows of data of cold start data then these are going to start to give really good answers and then I can have reason you see in the who is question it didn't attempt to reason at all but when I ask get a math question where it knows it asked the reason it is now doing the reasoning so I think community-wise one of the things that we should really be trying to look at as a community is try and get to an open- Source set of cult start data that anybody can use to pick up those models and again you can use the techniques that I did again I'm going to do some other videos to cover other things at some point and you can see the power of this that was a 1 and a half billion perometer model and it's doing pem Das and again if you go to the older models um like the mro models of 6 months ago Etc they were getting these answers wrong so now you're going to be able to sort of train those models uh with uh better answers and again I think this is just going to enable a whole sort of fine-tuning a type Community again I can ask it any other question so I can say calculate the result of the expression uh let's go with 9109 - 371 uh divided by - 5634 divided by uh minus uh 270 and then it's came back with 910 90177 now if I just asked chat GPT the same question so I've the all1 model calculate the result of this look at this it's taken a long time to come back with us answer it's taking 1 minute 1 second divide by this divide this result by this blah blah blah blah and then eventually it's came back with 9108 99 968 so basically 9109 look at this this is a one and a half billion parameter model it's not fully accurate but it's a 1 and A2 billion parameter model and again I think if we applied reinforcement learning it's it's going to come back with that pretty well but it's so close so again bringing in reinforcement learning Etc the math capability here is massively improved there and again if we look at this one here so WS 8895 let let's ask the same question of uh chat gbt so we'll just go to 01 model we'll ask it its question here let's look at the breakdown it's done all the kind of pem that stuff as you say remember the answer here is 79420 uh 0334 let's see what chat GPT comes back with again it's taking quite a long time to think break down the calculation this is a 1 and a half billion parameter model it did it fairly quick if you remember I just typed it in it came back fairly quickly again it's got a little bit of rounding going on here um let's see uh what it came back was 7 94204 7942 0.334 that is pretty darn good so hopefully you get the power of this this is this is an incredible technique and I haven't even done any reinforcement learning on this yet this is just using cold start data there so again I think it's pretty cool um in a future video we will do reinforcement learning and see if we can get this even better using kind of rejection sampling but uh hopefully you can get from this that um actually the technique that deeps R1 is using uh for the co start data is pretty powerful and we can start to do that itself again all the code that I put there is open sources on my GitHub you can go ahead and use this and you can go and reproduce the experiments for yourselves so again if I come back to the original paper for a second um the two things that that really was important here was this cold start so section 2.3 reinforcement learn with cold start um you know two natural questions can reasoning performance be further improved by incorporating a small amount of high quality data as coldstar it's exactly what we just did and how can we train a userfriendly model not only produce clear and coherent chains of thought but also demonstrates strong General capabilities so those chains of thought were pretty strong and coherent you've seen it was like step one step two it wasn't rambling all over the place there and then uh we've done exactly what they did unlike deepcar 1 prevent unstable cold start phase of RL training you saw from this version we were completely unstable as a base model you know with that sort of U thousand type examp well 380 examples with three Epoch with long chain of thoughts you can see it was coming back with the right uh answers it was also doing the verification in there so you see here directly per prompted models to generate detailed answers with reflection and verification we prompted it for reflection and verification as well um readable formats and then in this work we collect thousands of cool start data so if I increase that data set we had different types of problem s um we went beyond math we did code Etc we would get the the same answer so um so again I think I think I've reproduced what's happening with um with the cold start uh piec what we haven't done is then ran that through the reinforcement learning but again I think we can do that maybe in a future video um so once they've done that cold start they then do the reinforcement learning I think I'll show you how to do that uh in a future video but I think it's pretty cool that we can do this already anyway I hope this video has been useful and I'll catch you on the next one