I've realized it's possible to fine-tune language models by updating only a tiny number of parameters this can be done using Lowa methods but it can also be done using representation fine tuning which allows you to tweak the activations rather than the model weights I'm going to show you two examples today one where I guardrail a model to resist responding with any Financial advice and the other where I cause a model to respond always in French regardless of the question that it's being asked you're going to find that this techniques allows you to fine tune quickly and it should also enable you to create different behaviors that can be snapped together at inference time something I'll talk about a bit more in a later video too here's what I'm going to be going through to kick off um as I mentioned I'll go through one Guard railing case where I prevent a model from giving Financial advice and the other one where it will always respond in French before going through the use cases though I'm going to review the Transformer architecture very brief briefly because this will help us understand the difference between uh Lura fine-tuning which updates weights and representation fine tuning which involves tweaking the activations at a specific layer and for a specific input token so I'll show you each of those methods graphically and then I'll move on to a full example of representation fine tuning in a Jupiter notebook and do the same for Laura and we can see how they both perform and compare then I'll finish off with a few caveats on representation fine-tuning and also on Lura fine tuning the libraries for representation fine tuning are a lot less mature right now they're not supported with VM or TGI just yet but I think there are some nice features that will make them pretty interesting in the future and the libraries are strong for Laura so you should be able to do much of what I'm showing today and deploy it at inference for inference rer in production applications just uh one other note as well I'll comment a little bit on combining fine tunes and when this works well so let's get started and look back at uh the Transformer architecture I've made this really simple by looking at only a Transformer with two layers typically Transformers like Lama 3 will have uh tens of layers within them that's what makes it deep learning because you have a large depth of layers but it's much easier of course to look at just two layers so here I'm looking for add a Transformer that's going to be processing two Tokens The Words the and quick and here I'm referring to the common phrase the quick brown fox jumps over the lazy dog so our goal is to input two tokens that and quick and we're trying to predict a token uh which should be brown and here is our two- layer Transformer and I want to step through what's happening um at this input position for there as we go through this Transformer so first off we have the token the and there are going to be a specific number of tokens for a given language model uh for example there might be 32,000 tokens or roughly that in Lama 2 or in Lama 4 the vocabulary is about I think over 120,000 tokens so each of those T tokens is going to have a lookup table and in that lookup table for every token there's a vector there's a vector in a high dimensional space Maybe in 1,000 dimensions that's going to represent that token so when we start off with the token there we're going to move through a lookup table and find the vector associated with the token there once we have that Vector which I've shown here in blue that Vector is going to be transformed in different ways so it's going to be resized and redirected as it moves through the layers of the neural network and these layers typically have exactly the same setup so it's just a set of repeating layers here I've shown two of them and in a conventional Transformer there are two important parts to each of these layers there's the attention part and there's the feed forward or multi-layer perceptron MLP part and these play different roles the attention layer will typically gather information from all of the other token positions now it will only gather information from the past positions so since the word the is the first word there's no information to gather uh from the left hand side but when we move to the word quick the attention layers uh the attention modules within each layer are going to be grabbing information from all the previous token positions and they're going to use that information then then to decide how to transform the vector for the word quick now the multi-layer perceptrons by contrast don't necessarily take in direct information they will rather take the vector and they're going to do some transformation on it and the transformation is a trainable transformation that's controlled by a number of matrices um these in the case of Lama 3 are called up down and gate so these are three matrices of trainable parameters that are going to move and reshape the vector as it goes through so to recap in each layer you have the attention which is going to take in information from other positions at least those in the past and then you have the MLP or feed forward which is just uh you could call it a straightforward transformation of the information and of the vector into a different size and Direction and this process repeats then as we go up to Layer Two and we have an output Vector now the very same happens for the word quick it will go to the lookup table where we'll find a specific vector and note that the vectors for each of the tokens are also trainable so in that lookup table you have uh different vectors for each token and those vectors are going to be trained throughout the training process so we look up the the vector for quick we get it and then that Vector is going to be transformed so moved in Direction and magnitude as it goes all the way through um as it goes all the way through the layers so again in each layer um we're going to be having attention and a multi-layer perceptron module the attention will take in past information and use that to transform the vector and the multi-layer perceptron is going to just transform but not taking in directly any information from previous input positions and so you have this Vector here for quick that's going to be transformed and it's going to go all the way to the output and once it gets to the output it's going to go in reverse uh through the lookup table and that Vector will be converted back to a token which is our final prediction and I haven't shown that reversal here for the first position because we already know that and quick so our goal is to calculate Brown and then we would take Brown Put it Down to the right hand side and we would propagate through a new lay uh rather a new input position and get to the next predicted token now I'll say something that it might you might think is obvious but is very important which is that the weights that are used here so the up Matrix the down Matrix the gate Matrix the Q Matrix the K Matrix the V Matrix and the O Matrix and the attention these matricies of weights they are trainable so they're going to be updated in order to to get the best match on output tokens based on a training data set but these matrices are the exact same for every input position so the the Transformer when we talk about the Transformer we mean the weights and the weights and the formula that are used for those they're exactly the same at every single input position what differs between the input position is that the attention will be taking in in any given position it will be taking in past information so as you move to a longer and longer sequence you're going to have more and more information that's going to be going in to each attention layer and just notice how when we're in Layer Two we're going to be taking in information from uh layer one so the vector coming out of layer one position one is going to feed in and help uh guide the attention process in layer two of position two so what's happening at a much higher level when we think about Transformers is we're converting words and we're transforming them into some kind of meaning and that meaning is probably established as we get towards the middle of the network now in this representation I've only two layers that's probably not enough to work well as a Transformer but in a deeper Network when we get to the middle we're going to have some kind of meaning for example uh it might be German language so there might might be some aspect of a vector that indicates this is a German language or there might be some aspect of a vector that indicates this is sad so what's being said is sad just for example so while we have these kind of Brute Force brute force or not so much Brute Force but very mechanical descriptions at the entrance we have these more abstract descriptions at the middle and then we get back to rer Mechanical representations at the output because we need to represent our tokens and the reason why I'm telling you all of this is because when we think about fine-tuning it would be nice and when I mean when I say nice I mean efficient it would be efficient if rather than having to update all of the weights here in the model so rather than having to update these matrices these these these it would be very nice if instead we could just tweak the meaning at some very targeted point or even approximate point but if we could switch the meaning so that it goes from say French to German or ger to English or sad to happy if we could just tweak that Vector a little bit that might be a lot more efficient than having to update or to train many many more parameters and so that's the big motivation for what I'm talking about today is to figure out well can we find ways to just make a few little tweaks to parameters and um suspecting that it would make sense to tweak them where we have these abstract meanings if we can do that we can maybe save on efficiency and uh run this whole fine-tuning a lot more quickly so I've already shown um this little red arrow that's just showing information flowing from an earlier position to the attention from a later position and the same here from layer one to Layer Two and now I want to talk about weight fine tuning which is the fine tuning I've talked about in all my other videos so the idea with weight fine tuning is you take all of the matrices in the Transformer so up down gate and what you do is you are we going to update those so make some little Delta changes to those matrices and commonly we will do that uh to very many matricies in fact almost all so we will make some updates to the up Matrix down gate the Q KV and we will do that not just in one layer but we'll typically do it in every layer even like 32 layers or 70 layers or how many layers there are so we'll make updates to pretty much all of the parameters within the M weight matrices and that's how weight fine tuning works and one of the ways that we make that a lot more efficient is instead of updating all the weights we do what's called Laura which is low rank um adaptation so rather than training the base Matrix so if we have a matrix here W let's say w could be it could be for example the K Matrix in the first detention layer if we have that Matrix which is a th000 x th000 we' say no we're not going to update that full Matrix we won't update those 1 million parameters instead we're going to freeze this Matrix and we're going to create some adapters and we'll make these adapters a lot smaller so maybe just of height four uh height four but of width 1,000 and these adapters size this way they can be multiplied by each other to give a matrix that's about a million parameters so a th000 by a th but of course with a height of only four each that means we have uh only 4,000 parameters for each of these two so we'll be fine-tuning 8,000 parameters in instead of 1 million so the idea with Laura is we forward pass through the Frozen Matrix but when we update and make those Delta changes we only change these small adapters and this very significantly reduces um the memory that we need so it reduces the vram we need for fine-tuning because it means that we only need to keep the optimizer States uh for these reduced matrices instead of keeping them for the full Matrix here and it also means we need to keep the gradients for these matrices here the smaller ones instead of keeping gradients to make updates to the whole Matrix so this is Laura and I talk about it a lot in this is probably the slide that you see the most in my explanations um throughout language models or even Vision Transformers as well so again if we're doing um weight based updates we'll typically do Laura in order to reduce the trainable parameters but here's something new and the something new is how how about we don't train the weight matrices so how about we're not going to change up down gate we're not going to change qkv and we're not going to change that for any of the layers we're just going to leave them the same instead what about if we focus on these vectors what about if we focus on the vectors that are um created or rather updated by each layer and these vectors uh another word for the vectors is activations the activations are the outputs of each of these modules or layers that we have so what if we focus on updating these activations rather than focusing on the weights um at a very simple level if we have an activation H so let's just say we have a vector that starts off here um that's called X or H it goes through one layer and it's now an updated version of X or H and what if we just add on a small Delta here which I've just called B so instead of updating the weights we're going to inject a little bit of meaning into the middle of this uh Transformer and the motivation here is we have some meaning that might be say German and we're going to just tweak it with an added Vector that's going to shift it towards being French so here B this kind of um brighter blue that might be the French German to French Swit shift so we're Shifting the model's representation which thinks it should be German and now it's going to shift it towards being French and this is our targeted idea of making a change to how the model is behaving and in the same way that we updated the weights when we did a lower fine tune or a full fine tune we can use uh gradient descent and we can basically do an optimization problem with a new set of data so that we choose the value of this B so we basically choose the the strength and the direction of this update vector so that we can minimize the overall loss on the new data set and this is the idea of ref or representation or maybe you could also call it activation based fine tuning and just in the same way that Laura allows us to reduce the number of parameters we want to train you can have low ref so you can do a low rank representation of a low rank representation fine-tuning so you'll remember here instead of training this full Matrix with Laura where we update weights we instead update these smaller matrices well you can do the same thing instead of updating a full uh Matrix here that's 1,000 what we can do is we can update a much smaller Vector of size four and then we can have some kind of a rotation Matrix that's going to project this or rotate this uh Matrix or not matrix it's Vector in this case it's going to rotate it so that we can add it on top of the original Frozen activation now there are a few ways you can think of um setting this optimization problem up or setting up these adapters rouer I'll just go through a few of them at a very simple level um to calculate the new activation you could take the old one and you could add on some constant or some trainable parameter or trainable Vector um B which is kind of what I showed back here so H newal H+ B now you do need to make sure that you're adding something of the same Vector dimension and that's why we have this rotation Matrix here or so probably the simplest way that you could update the activations is just by adding in this trainable small H Vector B and typically or the rotation would also be trainable as well now what's actually being done behind the scenes in the ref notebooks we'll be showing today or I'll be showing today is a little more complex I don't think the complexity is actually needed but this is how it's currently being implemented so H new is H and instead of it simply being R * B there's also a linear term here where there's a weight Matrix multiplied by the existing activation H so you can just think of this as kind of a constant or trainable parameters that are uh just numbers versus here you have an a fine or a linear function with a Y intercept so it's kind of like yal MX plus C but here you've got uh wh plus b so this is the update we're going to do um we're going to put in an update and we're going to Target it so we're going to Target it at layer 15 uh we're going to take a look at Lama 3 the model that just came out and rather than trying to update the representations after every L we're just going to focus on layer 15 and for one more twist we're only going to apply this little addition at one position at one token position so you could think about well we'll put in this this little boost here in Direction and in size uh for the first position you could additionally train a boost to go in here on the second position or the third what we're going to do and it's somewhat arbitrary is we're going to apply it uh to the last of the prompt tokens so if you have a prompt that says why did chicken cross the road well uh road is the position where we're going to put in this intervention here so we can compare uh the performance of both Lowa and low we'll try and set things up the exact same so for low ref uh we'll have layer 15 being targeted with this little Vector change we're going to focus on the last prompt token only and we're going to use a rank of four um for that fine tuning so that just means when we go back here to how we set it up the rank of the Matrix or and W is going to before and for the lower comparison we're also just going to Target layer 15 this is different than what I normally do because I'll typically Target all of the layers and that's what you'll see in nearly every tutorial but uh yeah you'll be surprised with the results we get with just targeting one layer and also as is typically done with Laura we're going to apply this at all token positions um just for clarity all token positions in principle you could just Target Laura on a specific token I think probably it would make sense to try that but conventionally and the way the libraries are written it's applied to all of the token positions um because Transformers as I described earlier they make use of the exact same matrices for every single position so it's kind of conventional to do the same thing at every position but you know we like being unconventional um just we don't like it enough to do the work that would be required for Laura to um be applied at specific talken positions now it is possible to choose which token positions to do lowre on this is just because the library that reft is built on is called um um raft low py raft it's built on pyv which happens to allow you to very specifically control the activations within um a neural network even down to the input position description okay so we're ready to move to the notebooks um I've already described why this makes intuitive sense we'll see if it works in practice it makes intuitive sense because we're trying to intervene where there are abstract representations and by tweaking an abstract representation we think we can be a lot more efficient than having to just update the whole network now if you want to read a little bit more on attention or multi-layer perceptrons and how those individual matrices are used within a formula to process the direction of the vectors you can check out these links which will be in the presentation that I'll link below in the description I'll now take a look into the two scripts the Lura base script and the representation fine-tuning script to to get this going I'm going to use a oneclick template for runp pod that's available in the public repo here oneclick llms and if you choose the one for fine-tuning Notebook that will bring you directly to a page where you can uh start up a Cuda 12.1 template that has Cuda and pytorch pre-installed and what I did is Select an a6000 I'm going to run on a fine tuning on Lama 3 so that should be sufficient and here you can see uh the template which is The fine-tuning Notebook so with that I'll click deploy and we'll be up and running once the notebook has started I'll open up jupter notebook uh the web version and I have uploaded here uh tiny Laura and also trellis reft these scripts are available in the trellis Advanced fine-tuning repository you can purchase lifetime access to that at tr.com that's tr.com advanced finetuning in the case of rft there are are very many scripts that are already available if you just go to py Raft type in py or EFT to GitHub and you'll see a number of scripts for fine-tuning for representation let's start off with the Lura fine tuning just because it's more familiar and it'll be easier to explain what I'm doing and then you'll see how the representation fine-tuning is a little bit different the first step is to run the installation there are two options within this script one is a standard Transformers installation and the other is to do an installation with UNS sloth UNS sloth is about twice as fast but all of these fine tunings are really fast so you can choose either one once the installation is done um I'm going to select a model the model I'll be running is matal Lama 3 I'm using uh the version that's on news research repository because this one here is not gated so I don't need to log into hugging face next I'm going to load the model so I'm loading with the model ID the device map is auto which will automatically dispatch to the GPU my torch dtype is B FL 16 so I'm using brain FL 16 because I have an ampere architecture that supports it this will allow for a better precision and Flash attention is turned on for improved training speed next I like to run some loading checks just to make sure that nothing uh overflowed onto the CPU um so it will print errors if something does overflow which it hasn't and next I'm going to set up uh the tokenizer so here you can see I have uh the beginning of SE token and the end of sequence token uh for the L 3 model and I have uh the end of sequence token now being redefined so Lama 3 it has this end end of sequence token end of text but actually the last token that's produced when's a response is eot ID so that's why I need to make the update here next I'm going to set the pad token Lama doesn't have a pad token so what I'm going to do is Define one I'm going to define the pad token as pad and then I'm going to add that to the tokenizer and set the pad token equal to that pad token now because I've created a new token in the tokenizer I need to resize uh the tokenizer embeddings so after I've set that pad token in the model and check that the model and tokenizer have the same pad token I then need to resize the token embeddings so what this means is those that lookup table between each token and the vector that needs to be sized uh to add one extra Row for this new pad token and you also have the option to pad to a multiple of 16 so this means it will add actually 16 rows um sometimes the GPU operations they like to have multiples of 16 in the calculations and so it can be common to pad to 16 instead of just adding one row now there is a drawback to adding this extra token here it means that now I've got some extra embedding vectors and that means that when I run the fine tuning uh with Laura it's going to save the full embeddings and the embeddings are quite large for Lama 3 because Lama 3 has got a token vocabulary uh of about 120 or 30,000 actually there's about 2 gabt worth of embeddings that will be saved so that means my Lowa adapters which will also contain those saved embeddings are going to be quite large so it would be nice if I didn't have to add that token and didn't have to resave the embeddings there probably is a way to do that uh what I'd recommend is maybe for the padding token to use uh what was the end of sequence token so you could actually Define the padding token as end of text and uh once you have that set you could then set the EOS token as eot tore ID and that'll just mean you don't need to resize your embeddings because you're using a token that already exists okay so moving on uh I'm just printing the model here and you can see the modules we talked about so in each layer there's the attention and there's the multi-layer perceptron and here you have the uh the Q KV o there's also um some parameters that are required for the rotary embeddings rotary embeddings they provide positional information on the tokens uh then in the MLP layer you have the gate up and down projections as I described previously and you also have the activation function within that multi-layer perceptron uh the choice here is silu sometimes you see gallu or Ru these are different shapes for your activation function now we're going to um set up Laura so here we're going to import the Laura config and get PFT model and set some of the parameters for lower configuration so first off we have an adapter that is going to be of rank four and we'll use the same for representation fine-tuning I'm using allur Alpha of 32 this is my recommendation for uh 7B models for larger models uh I recommend using larger Lura Alpha check out the video on Lura fine-tuning uh if you want to have more guidance on using that and also rescaled Laura so I am I'm using rescale Laura here which will adapt the learning rate depending on the rank of the adapters I'm using and notice how I'm only going to Target layer 15 which is kind of in the middle they're 30 32 layers so right in the middle here I'm going to uh apply a low adapter and I'm also only going to um tackle the out projection so I'm only going to tackle one of the matrices um within layer 15 so if we go back here uh you'll notice that there are many matrices we could choose to Target with Laura but I'm only going to Target uh the out projection here so why am I picking the O and what is that well the O is an out output projection and it's part of the attention layer where you also have the qk and V projection and in simple terms qk and V are used together to pick out the relevant information from past tokens and then determine what that should imply for the current token position update but after doing the qkv operations in Lama 3 there's an out projection and that basically takes the outputs of the attention process and it basically does a simple linear layer on top of that so you can think of the out projection it's just a simple linear transformation of uh the attention that happens in each Lama layer so I'm just going to focus on updating this with the Laura adapter and once that's applied through this uh command here for for getting the PFT model we can check the number of trainable parameters and because I'm only targeting one layer and I'm only targeting uh the O projection the number of trainable parameters is really small it's only 32,000 so 0.004% of the total parameters and I can even print out and see which modules are trainable and you'll see the trainable ones are all layer 15 and it's um Laura B and Laura a next uh we'll set up some simple evalu ation I'm going to ask two questions the first is what are the first Fibonacci numbers and the second is do you think I should get a fixed or variable rate mortgage uh the second question is trying to test whether we can get the model to say no I don't give any Financial advice so after running that I just do some quick test evaluation on the base model which is all very reasonable it gives me some Fibonacci numbers and then gives me some advice on fixed versus uh variable mortgages next I'm going to load the data sets for fine-tuning you can if you want load a data set from hugging face and then format it but I'm just going to Define 10 rows for each data set here um I have a data set here if you want to fine- tune a model to respond with emojis that's based on work in the py raft repository then I've created these other data sets so this one here is defin you in the model to guardrail it against providing Financial advice and you'll see for example if I say what should I invest in right now it'll just respond unfortunately I'm unable to provide Financial advice so just these 10 examples we can actually get some good Guard railing on a model and prevent it from giving that Financial advice now I have two other data sets here one is uh Force French examples this one here will take in questions in a few different languages like what is your name and it will always respond in French so it'll save you my pel jimy and I've also asked a few questions in a few other languages uh volcanic G is beer fend in so some German K Temple so uh Spanish and then I even put in some Irish here so all of these are going to be answered in French V taxi okay and we're just trying to force the model to respond in French no matter what input we have now if you wanted you could also try uh to force French from English this is probably a little more straightforward uh to find youe the model on because it's just more consistent we're saying okay if you get an English input then give French output as opposed to saying uh if we have an input that's any of these four languages so with that um I'm going to set the first example uh for this training I've set it to force French examples so we're going to force the model to respond in French and I'm going to convert it to a pandis data frame and then format it into messages so we'll have these uh this list of conversations between the user and the assistant and with that we're ready with our training data set and we can pass that in because it has a column of messages directly to the sft trainer and the sft trainer will automatically apply the chat template so here's what this data set looks like it's got 10 rows and each row has got this user message and assistant message next uh we're going to move on to training the model and there are a few kind of unique things here uh so we're going to use a batch size of 10 uh gradiant accumulation of one and importantly I'm going to use a very large number of epoch and this is the opposite of what I recommend in a lot of my fine tunings where I have a large lower Rank and a complex data set here I'm going to have a very small Lowa rank so a very small number of parameters being updated and I'm actually going to overfit so we're going to overfit on these parameters and get down to basically a zero loss on the training data set and we're not going to worry about the validation data set now the intuition here is that because we're fine-tuning in such a narrow number of parameters the model should hopefully still generalize because we're just inputting this very specific information um so hopefully we won't cause overfitting even though we are going to overfit on this very narrow set of training parameters one other little note here is that I'm only going to train on the responses so that means I'm going to penalize the model based on responses I'm not going to penalize it based on predicting tokens within the question itself so here here we have the sft trainer uh all of the parameters are pretty common you'll see I've commented out anything to do with the validation data set and the learning rate I have put is um probably on the high side uh we've got a very small number of parameters and that will lead to a high learning rate already because rescale Lura is being applied but even still I've put the learning rate fairly High here so gradient checkpointing is true this will reduce memory a little bit and with that we're ready to run uh the training now you'll see here over 100 steps the training loss Falls and it eventually goes down to 02 so we have extremely low training loss by the end we've basically overfitted uh these examples next we'll run uh the evaluation again and you can see here when I ask what are the first Fibonacci numbers uh the answer is indeed in French Premier Fibonacci and then do you think I should get a fixed or variable rate mortgage and the answer is so it's it's just straight up saying that you should get a fixed rate mortgage so here you can see um it's working fairly well and um note that I'm just fine-tuning to to um respond in French this is not the fine-tuning with the financial Guard railing which I'm going to show you in a second in fact uh when I run the one with Guard railing let me see if I can just upload it here so this is where we force it uh to respond in French and I'm going to show you the example of the financial advice Guard railing on the representation fine-tuning but I did run that also on Laura and what I want to show you now is quickly just combining those two adapters so here uh what I'm doing now um is I'm going to reload the base model I'm going to then load an adapter for no Financial advice and I'm going to then load an adapter that is for forced French so when I combine these two adapters I should get a model that's going to refuse to give Financial advice but is always going to answer in French so I load the model here um I need to resize the embeddings after loading it because I added a a new token to the vocab when I did that fine-tuning and then I'm going to uh load merge and unload the first adapter and then load merge and unload the second adapter here and when that is done I'll be able to inference using this uh combined model that has both of the adapters applied so again I'm going to ask these two questions about the first Fibonacci numbers and whether she get a fixed a variable mortgage and you can see here what are the first Fibonacci numbers um the answer in French is is correct and it's in French and then when I asked do you think I should get a fixed or variable rate mortgage the answer is uh Le con fin and uh this is kind of directionally refusing to respond although the French is not exactly correct here um this doesn't fully make sense so I would say that when you combine these two adapters um it's kind of getting the combined effect my guess is that the issue is around the French when you're trying to get a model to respond only in a language I have found that you need to use more uh adapter ranks so a rank of four is not great in order to just get a model to speak one language uh I found that using a rank of eight or even 16 or potentially fine-tuning on more of the modules rather than just the out projection is necessary to get good performance in language um by contrast I think if you're doing simpler things like getting the model to respond with Emojis or even with the Guard railing example you can get reasonably good performance uh using a fairly small Laura rank so this is my demonstration of Laura and you can see that you can get um you can get reasonable performance just by using a tiny number of trainable parameters so let's take a look at uh the rft the representation fine-tuning example and as a recall just uh remember that when we do representation fine tuning we're not going to update the weights of the matrices instead we're going to add in this little adjusting Vector um to this uh Vector here which is an activation after the 15th layer and interestingly we're not going to do this at every input position we're just going to do it at the last input position so here it's shown at the first but actually we're going to apply it at the last input position of the prompt so let's take a look how this works and this script here is um is based on and heavily draws from the py raft repository and the fine-tuning scripts there so after the installation and loading the model and the token ier again I'm going to set the tokenizer end of sequence token just as I've done here and for the sample prompt um what have we got here this is just an example of applying the chat template uh so you can see here I'm applying the chat template to this sample prompt to see what it looks like and then I'm setting up this templating function so uh pyre in the examples rather than using a Ply chat template they use a more manual python templating approach and so I needed to set up this template and to know the format I just printed out what's been used in apply chat template in the same way that we got a Lowa config you get a ref config maybe technically it should be called a lowre config um but I guess it does uh have lowre intervention here so it is indeed a low rank ref a low rank representation fine tune you can see we're targeting layer 15 we're using using a dimension of four for the adapter uh the component we're targeting is the block output this really just means we're targeting the attention after layer 15 and what else uh we're feeding in for this uh intervention here the embedding Dimension is the hidden size so this is basically just saying that uh what the dimensionality is of the activation because we need that information when we're going to set up the adapters then just like we do with Laura there's a ref model so get ref model instead of get beft model and we input the rft config and with that we can print out the trainable parameters and notice how here because we're using a rank of four um just because of the way it works out the setup uh that's used for lowre we end up with uh the same number of trainable parameters that we do in uh lower fine tuning now it's a bit of a coincidence that we get exactly the same number um but it makes sense that it's kind of roughly the same number because we're using the same rank remember also that just because we're only going to apply this in one token position um that doesn't save us any trainable parameters because in Laura when we apply the Laura to every token position we're just talking about the same set of Weights the same weights are used at every position so there's no saving in trainable parameters just by applying ref at one position so here we have the very same examples uh um will be focused on the no fin examples and the force French examples and here we have um data preparation so notice how when this is being set up we're going to use py raft make last position supervised data module and this is where we're specifically saying that we'll only be targeting the last position of the prompt with this intervention and when I say intervention what I mean is adjustment of the activation Vector at the output of layer 50 at the last position in the prompt so with the data set up uh we're ready to move on to the training args we're going to train for 100 EPO and we've got a learning rate here 4 e minus 3 so this is a a really high learning rate I possibly should have used the exact same learning rate in ref as in Laura just so I have a fair comparison that's kind of a bit difficult because I'm using rescaled Laura um rescale Laura will adapt the learning rate according to the rank that I'm using so maybe what I should have done is to turn off rescale Laura and then set the same learning rate my suspicion is that the lower learning rate being used here in the ref training is uh higher than what I used in Lura so I would expect uh that's why I'm getting faster convergence which you'll see below so when I do run training uh the training loss drops really fast and it goes down to a really small value so we're fitting that training data exactly now when we TR when we run the examples so we say what is the first Fibonacci what are the first Fibonacci numbers uh it says I'm glad you asked it's a series of numbers in which each number is the sum of the two pre preceding numbers uh so this is a reasonable answer but then when I asked do you think I should get a fixed rever variable rate mortgage it says providing Financial advice is beyond my capabilities so you can see here the model just with this tiny fine tune is quite well able to guard rail against providing Financial advice uh while still answering questions that are not Financial ad advice now this is not to say you won't be able to get around the model if I maybe cast some question that's Financial in a non-financial way then it might slip through um but still it's providing definitely a primary layer of Guard railing now there's one more technical thing I want to talk about and that's around combining these uh ref interventions which is analogous to combining Lowa adapters and merging them and there is a quite nice technique that is possible with representation fine-tuning in principle it's also possible with Laura although it's not supported by libraries and that is when you're fine-tuning if you want to fine-tune for separate applications you use um orthogonal Lura representations so you might have one Lura um Vector adjustment that's pointing in this direction for French and then you'll have one that's at right angles basically although in very high diens dimensional space so you'll have one at right angles that could be for doing something like responding in emojis and the reason why it's nice to have them in right angles is because if they're at right angles they won't interfere with each other so when you add these adapters on top of each other you're going to get kind of the sum of the effects you won't get interference between them and the way this is done here there's actually a small bug in the code um that I've put up on an issue unless I'm just mistaken um I've put up uh the bug so hopefully it'll be resolved soon and you'll be able to run through this script but basically what this does is it allows us to take say a rank eight adapter and use the first four of those ranks for fine tuning for French and use the next four for fine-tuning for financial Guard railing um so I can I can just show you a little snippet here basically we split up this uh rank of eight into uh Subspace of 0 to three for uh no Financial advice and then we use four to 7 for a French Subspace and this allows us to set up orthogonal um reft or orthogonal low reft fine tunes that won't interfere and so they can be snapped together at inference time um without having interference and in fact when I showed you um the example earlier of combining two luras the reason why this answer may not be uh perfect is one I didn't use enough ranks for doing my French fine tune or two it's possible I'm getting a YouTube thinks I want you to give me a thumbs up uh or two it's possible there's just interference between the two lore adapter and that's causing uh some degradation in the response just to add to that a tiny bit um I'll show you this here in the presentation uh so what I'm talking about is the question of can you combine fine tunes uh which you could do by combining lower adapters or combining uh representation fine tunes and the question I'm talking about is will applying Laura 1 or ref one plus ref 2 combine the effects of both and at a really high level the answer should be no because Transformers are nonlinear so you cannot just say I'm going to do y if Y is the Transformer function y of a plus y of B is not going to be equal to Y of a plus b um so if you've any nonlinear system it's it's not going to be possible to just add two adapters and have the effect be equal to the combination of the two uh individually however there are two exceptions here that are particularly important so the first exception is that um well maybe I should start with the second exception the second exception is that if you find tune in a way that they are orthogonal or perpendicular so if you have an adapter where you're using four dimensions for fine tune a and four for B well if they're orthogonal in high dimensional space they're not going to interfere and if you're only inter if you're only intervening at one specific position so if you're only intervening as I've shown within one specific position in one specific layer and they're orthogonal that means you should be able to do it in a way that there isn't collision between the two however there's a more General reason why you might be able to snap together Lura adapters and you may already know that it's quite common to merge models or to merge adapters and it seems to work in practice despite these models being quite nonlinear um and I suspect the reason why it works is because even if you're not perfectly orthogonal so if you're not perfectly um perpendicular with your two Laura adapters it's still really hard to have much Collision when you're in high dimensional space and the way to think about this is if you think about two cars and they're forced to drive on a line so that's one dimension then the two cars are guaranteed to crash if they're driving in one dimension towards each other now if they're in two Dimensions which is how cars normally drive it's possible that they can crash H we have crashes all the time um now if you move to three dimensions like a plane well planes are much less likely to crash because they're three dimensions so it's very hard for them to crash and if you to go up to four 5 6 7 8 N9 the more Dimensions you go to and Laura adapters are often in even 32 Dimensions it's pretty diff difficult to have two Lowa adapters that are going to have vectors uh output vectors that are going to crash and so there's this intuition that when you move to really high dimensional spaces things end up being very close to orthogonal they end up being close uh to right angles um in most cases just because the dimensionality of the space is so big and it becomes really difficult to crash and so for this reason it means you don't get interference as much as you think you would when you're combining models in a model merge or when you're combining lower adapters okay so with all of that um I've shown you how to do Lowa fine tuning with a small number of parameters I've shown you how to do uh representation fine tuning with a small number of parameters I think both of these methods work quite well particularly when um there's a very specific meaning that you're trying to impose on top of the model's performance if you're trying to impose more comp Lex behaviors like maybe very complicated function calling patterns with many different um nestings or parallel calls maybe then you still need to have a higher rank or maybe if you have something complex like uh a specific language and certain behaviors you may also need to have a higher rank in order to achieve the granularity needed to get the performance there just a few um notes before we finish on the limitations so the representation fine-tuning uh it's nice because you can Target the activations rather than the parameters of the model I'm actually not sure if there's really a difference between that possibly the representation fine-tuning and the Lura fine tuning end up being somewhat equivalent depending on how you set them up I think there's more research needed uh to figure that out indeed um the representation that is being used if you just check out here on on GitHub you can see that it seems um the best performance or the most efficient performance from the representation fine-tuning is to do something like a * bh+ B I mentioned earlier how maybe you could just do h plus b or maybe ah plus b as an adjustment of H but it seems that these uh work poorly and it looks like we're going to have more information on a paper that's due to come out soon but my point here is that it's possible the way that you do these activ updates it may end up being somewhat equivalent to how you do parameter updates because this in some ways a BH plus b it's kind of a feed forward layer which means that adjusting the out projection May in some ways become somewhat equivalent now just back to limitations here representation fine-tuning it's uh a fairly new library so it doesn't have the same support as luras you can push models to hugging face Hub you can train or orthonormal parameter sets but you cannot use VM or TGI just yet you also cannot do continuous batching uh if you want to set up an inference endpoint right now I believe that rft supports a single GPU only and also um there's no flattening or merging of the trained parameters so in the same way you can well not the same way but in the way you can merge low adapters on top of a base model uh you cannot do that with representation fine-tuning um you would have to find some kind of a hack to put that representation within the main modules and apply it there meanwhile for Laura you can push the hugging face Hub it's supported by VM and also by TGI you can even load multiple luras on the fly with VM this is something I want to show in a video coming up soon you can do continuous batching and there's the option to merge the adapters onto the base model to get um a merged and new fresh model one of the caveats is that there isn't support right now uh at least that I know of to train orthonormal luras um You probably can just train luras anyway and uh snap them together and get reasonable performance in most cases but maybe this would be a nice feature to have uh to ensure that there is less of a collision between the performance of different uh Laura adapters just before I wrap up a few quick comments one question that might come to mind is well if we're training far fewer parameters is the training a lot faster when we do this targeted Lura or ref type fine tuning and the answer is well partially so when you do fine tuning with Laura or with ref you still need to forward pass through all of the parameters so the saving when you use targeted Lura or low ref is that your backward pass is going to be faster and so you can reduce the time for your backward pass down to a really small amount but your forward pass is still going to be the limiting fact factor in determining your overall training time and since the number of parameters even in what I'll call Standard Laura is really small you tend to spend a lot of time on the forward pass and so there's not so much reduction in the total training time just because you're going to further reduce the number of parameters from say 1% of the model down to 0.001% of the model another question you might have is well should I do any fine-tuning now with larger lower ranks for example 8 16 or 32 and I would say yes it's still worth doing the training and trying it out with higher ranks but now I think you should consider doing this very low rank training as well on a specific layer usually a middle layer within the module within the model and the reason is if your fine-tuning could have a representation that just needs to be tweaked in the middle of the model you may end up with performance that is a bit more robust than doing a broader fine-tune on a larger number of param parameters then then again you may find that the fine-tuning is not granular enough and so it may actually do better if you use a larger rank so really what I'm saying is that this is an extra tool to consider it allows you to fine tune a little bit more quickly in a more targeted way and it can also have benefits in terms of having smaller Lowa adapters that need to be loaded at inference time something I hope to get to in a video soon that's it for this video do check out the GitHub pyra repository under Stanford NLP and you can also find these scripts if you head over to tr.com Advanced fine-tuning let me know the questions in the comments cheers