Browser tab content: # Google Colab

To run this, press "_Runtime_" and press "_Run all_" on a **free** Tesla T4 Google Colab instance!

[![](https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png)](https://www.google.com/url?q=https%3A%2F%2Funsloth.ai%2F) [![](https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png)](https://www.google.com/url?q=https%3A%2F%2Fdiscord.gg%2Funsloth) [![](https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true)](https://www.google.com/url?q=https%3A%2F%2Fdocs.unsloth.ai%2F) Join Discord if you need help + ⭐ _Star us on [Github](https://github.com/unslothai/unsloth)_ ⭐

To install Unsloth on your own computer, follow the installation instructions on our Github page [here](https://www.google.com/url?q=https%3A%2F%2Fdocs.unsloth.ai%2Fget-started%2Finstalling-%2B-updating).

You will learn how to do [data prep](#Data), how to [train](#Train), how to [run the model](#Inference), & [how to save it](#Save)

### News

### Installation

%%capture  
# Normally using pip install unsloth is enough  
  
# Temporarily as of Jan 31st 2025, Colab has some issues with Pytorch  
# Using pip install unsloth will take 3 minutes, whilst the below takes <1 minute:  
!pip install --no-deps bitsandbytes accelerate xformers==0.0.29 peft trl triton  
!pip install --no-deps cut\_cross\_entropy unsloth\_zoo  
!pip install sentencepiece protobuf datasets huggingface\_hub hf\_transfer  
!pip install --no-deps unsloth  

### Unsloth

from unsloth import FastLanguageModel  
import torch  
max\_seq\_length = 2048 # Choose any! We auto support RoPE Scaling internally!  
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+  
load\_in\_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.  
  
# 4bit pre quantized models we support for 4x faster downloading + no OOMs.  
fourbit\_models = \[  
    "unsloth/Meta-Llama-3.1-8B-bnb-4bit",      # Llama-3.1 2x faster  
    "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",  
    "unsloth/Meta-Llama-3.1-70B-bnb-4bit",  
    "unsloth/Meta-Llama-3.1-405B-bnb-4bit",    # 4bit for 405b!  
    "unsloth/Mistral-Small-Instruct-2409",     # Mistral 22b 2x faster!  
    "unsloth/mistral-7b-instruct-v0.3-bnb-4bit",  
    "unsloth/Phi-3.5-mini-instruct",           # Phi-3.5 2x faster!  
    "unsloth/Phi-3-medium-4k-instruct",  
    "unsloth/gemma-2-9b-bnb-4bit",  
    "unsloth/gemma-2-27b-bnb-4bit",            # Gemma 2x faster!  
  
    "unsloth/Llama-3.2-1B-bnb-4bit",           # NEW! Llama 3.2 models  
    "unsloth/Llama-3.2-1B-Instruct-bnb-4bit",  
    "unsloth/Llama-3.2-3B-bnb-4bit",  
    "unsloth/Llama-3.2-3B-Instruct-bnb-4bit",  
  
    "unsloth/Llama-3.3-70B-Instruct-bnb-4bit" # NEW! Llama 3.3 70B!  
\] # More models at https://huggingface.co/unsloth  
  
model, tokenizer = FastLanguageModel.from\_pretrained(  
    model\_name = "unsloth/Llama-3.2-3B-Instruct", # or choose "unsloth/Llama-3.2-1B-Instruct"  
    max\_seq\_length = max\_seq\_length,  
    dtype = dtype,  
    load\_in\_4bit = load\_in\_4bit,  
    # token = "hf\_...", # use one if using gated models like meta-llama/Llama-2-7b-hf  
)  

We now add LoRA adapters so we only need to update 1 to 10% of all parameters!

model = FastLanguageModel.get\_peft\_model(  
    model,  
    r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128  
    target\_modules = \["q\_proj", "k\_proj", "v\_proj", "o\_proj",  
                      "gate\_proj", "up\_proj", "down\_proj",\],  
    lora\_alpha = 16,  
    lora\_dropout = 0, # Supports any, but = 0 is optimized  
    bias = "none",    # Supports any, but = "none" is optimized  
    # \[NEW\] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!  
    use\_gradient\_checkpointing = "unsloth", # True or "unsloth" for very long context  
    random\_state = 3407,  
    use\_rslora = False,  # We support rank stabilized LoRA  
    loftq\_config = None, # And LoftQ  
)  

Unsloth 2024.10.0 patched 28 layers with 28 QKV layers, 28 O layers and 28 MLP layers.

### Data Prep

We now use the `Llama-3.1` format for conversation style finetunes. We use [Maxime Labonne's FineTome-100k](https://www.google.com/url?q=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Fmlabonne%2FFineTome-100k) dataset in ShareGPT style. But we convert it to HuggingFace's normal multiturn format `("role", "content")` instead of `("from", "value")`/ Llama-3 renders multi turn conversations like below:

```


user
Hello!

assistant
Hey there! How are you?

user
I'm great thanks!

```

We use our `get_chat_template` function to get the correct chat template. We support `zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, phi3, llama3` and more.

We now use `standardize_sharegpt` to convert ShareGPT style datasets into HuggingFace's generic format. This changes the dataset from looking like:

```
{"from": "system", "value": "You are an assistant"}{"from": "human", "value": "What is 2+2?"}{"from": "gpt", "value": "It's 4."}
```

to

```
{"role": "system", "content": "You are an assistant"}{"role": "user", "content": "What is 2+2?"}{"role": "assistant", "content": "It's 4."}
```

We look at how the conversations are structured for item 5:

\[{'content': 'How do astronomers determine the original wavelength of light emitted by a celestial body at rest, which is necessary for measuring its speed using the Doppler effect?',
  'role': 'user'},
 {'content': 'Astronomers make use of the unique spectral fingerprints of elements found in stars. These elements emit and absorb light at specific, known wavelengths, forming an absorption spectrum. By analyzing the light received from distant stars and comparing it to the laboratory-measured spectra of these elements, astronomers can identify the shifts in these wavelengths due to the Doppler effect. The observed shift tells them the extent to which the light has been redshifted or blueshifted, thereby allowing them to calculate the speed of the star along the line of sight relative to Earth.',
  'role': 'assistant'}\]

And we see how the chat template transformed these conversations.

**\[Notice\]** Llama 3.1 Instruct's default chat template default adds `"Cutting Knowledge Date: December 2023\nToday Date: 26 July 2024"`, so do not be alarmed!

### Train the model

Now let's use Huggingface TRL's `SFTTrainer`! More docs here: [TRL SFT docs](https://www.google.com/url?q=https%3A%2F%2Fhuggingface.co%2Fdocs%2Ftrl%2Fsft_trainer). We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`. We also support TRL's `DPOTrainer`!

from trl import SFTTrainer  
from transformers import TrainingArguments, DataCollatorForSeq2Seq  
from unsloth import is\_bfloat16\_supported  
  
trainer = SFTTrainer(  
    model = model,  
    tokenizer = tokenizer,  
    train\_dataset = dataset,  
    dataset\_text\_field = "text",  
    max\_seq\_length = max\_seq\_length,  
    data\_collator = DataCollatorForSeq2Seq(tokenizer = tokenizer),  
    dataset\_num\_proc = 2,  
    packing = False, # Can make training 5x faster for short sequences.  
    args = TrainingArguments(  
        per\_device\_train\_batch\_size = 2,  
        gradient\_accumulation\_steps = 4,  
        warmup\_steps = 5,  
        # num\_train\_epochs = 1, # Set this for 1 full training run.  
        max\_steps = 60,  
        learning\_rate = 2e-4,  
        fp16 = not is\_bfloat16\_supported(),  
        bf16 = is\_bfloat16\_supported(),  
        logging\_steps = 1,  
        optim = "adamw\_8bit",  
        weight\_decay = 0.01,  
        lr\_scheduler\_type = "linear",  
        seed = 3407,  
        output\_dir = "outputs",  
        report\_to = "none", # Use this for WandB etc  
    ),  
)  

We also use Unsloth's `train_on_completions` method to only train on the assistant outputs and ignore the loss on the user's inputs.

from unsloth.chat\_templates import train\_on\_responses\_only  
trainer = train\_on\_responses\_only(  
    trainer,  
    instruction\_part = "<|start\_header\_id|>user<|end\_header\_id|>\\n\\n",  
    response\_part = "<|start\_header\_id|>assistant<|end\_header\_id|>\\n\\n",  
)  

We verify masking is actually done:

tokenizer.decode(trainer.train\_dataset\[5\]\["input\_ids"\])  

space = tokenizer(" ", add\_special\_tokens = False).input\_ids\[0\]  
tokenizer.decode(\[space if x == \-100 else x for x in trainer.train\_dataset\[5\]\["labels"\]\])  

We can see the System and Instruction prompts are successfully masked!

### Show current memory stats

GPU = Tesla T4. Max memory = 14.748 GB.
2.635 GB of memory reserved.

trainer\_stats = trainer.train()  

### Show final memory and time stats

446.5262 seconds used for training.
7.44 minutes used for training.
Peak reserved memory = 6.531 GB.
Peak reserved memory for training = 3.896 GB.
Peak reserved memory % of max memory = 44.284 %.
Peak reserved memory for training % of max memory = 26.417 %.

### Inference

Let's run the model! You can change the instruction and input - leave the output blank!

**\[NEW\] Try 2x faster inference in a free Colab for Llama-3.1 8b Instruct [here](https://colab.research.google.com/drive/1T-YBVfnphoVc8E2E854qF3jdia2Ll2W2?usp=sharing)**

We use `min_p = 0.1` and `temperature = 1.5`. Read this [Tweet](https://www.google.com/url?q=https%3A%2F%2Fx.com%2Fmenhguin%2Fstatus%2F1826132708508213629) for more information on why.

from unsloth.chat\_templates import get\_chat\_template  
  
tokenizer = get\_chat\_template(  
    tokenizer,  
    chat\_template = "llama-3.1",  
)  
FastLanguageModel.for\_inference(model) # Enable native 2x faster inference  
  
messages = \[  
    {"role": "user", "content": "Continue the fibonnaci sequence: 1, 1, 2, 3, 5, 8,"},  
\]  
inputs = tokenizer.apply\_chat\_template(  
    messages,  
    tokenize = True,  
    add\_generation\_prompt = True, # Must add for generation  
    return\_tensors = "pt",  
).to("cuda")  
  
outputs = model.generate(input\_ids = inputs, max\_new\_tokens = 64, use\_cache = True,  
                         temperature = 1.5, min\_p = 0.1)  
tokenizer.batch\_decode(outputs)  

The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's \`attention\_mask\` to obtain reliable results.

\['<|begin\_of\_text|><|start\_header\_id|>system<|end\_header\_id|>\\n\\nCutting Knowledge Date: December 2023\\nToday Date: 26 July 2024\\n\\n<|eot\_id|><|start\_header\_id|>user<|end\_header\_id|>\\n\\nContinue the fibonnaci sequence: 1, 1, 2, 3, 5, 8,<|eot\_id|><|start\_header\_id|>assistant<|end\_header\_id|>\\n\\nThe Fibonacci sequence is a series of numbers in which each number is the sum of the two preceding numbers. The sequence is: 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144.<|eot\_id|>'\]

You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!

FastLanguageModel.for\_inference(model) # Enable native 2x faster inference  
  
messages = \[  
    {"role": "user", "content": "Continue the fibonnaci sequence: 1, 1, 2, 3, 5, 8,"},  
\]  
inputs = tokenizer.apply\_chat\_template(  
    messages,  
    tokenize = True,  
    add\_generation\_prompt = True, # Must add for generation  
    return\_tensors = "pt",  
).to("cuda")  
  
from transformers import TextStreamer  
text\_streamer = TextStreamer(tokenizer, skip\_prompt = True)  
\_ = model.generate(input\_ids = inputs, streamer = text\_streamer, max\_new\_tokens = 128,  
                   use\_cache = True, temperature = 1.5, min\_p = 0.1)  

The Fibonacci sequence is a series of numbers where each number is the sum of the two preceding numbers. 

The sequence you provided was: 1, 1, 2, 3, 5, 8, 13

The next number in the sequence would be 21, which is 8 + 13. The sequence continues as: 21, 34, 55, 89, 144, 233.<|eot\_id|>

### Saving, loading finetuned models

To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.

**\[NOTE\]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!

model.save\_pretrained("lora\_model")  # Local saving  
tokenizer.save\_pretrained("lora\_model")  
# model.push\_to\_hub("your\_name/lora\_model", token \= "...") # Online saving  
# tokenizer.push\_to\_hub("your\_name/lora\_model", token = "...") # Online saving  

('lora\_model/tokenizer\_config.json',
 'lora\_model/special\_tokens\_map.json',
 'lora\_model/tokenizer.json')

Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:

if False:  
    from unsloth import FastLanguageModel  
    model, tokenizer = FastLanguageModel.from\_pretrained(  
        model\_name = "lora\_model", # YOUR MODEL YOU USED FOR TRAINING  
        max\_seq\_length = max\_seq\_length,  
        dtype = dtype,  
        load\_in\_4bit = load\_in\_4bit,  
    )  
    FastLanguageModel.for\_inference(model) # Enable native 2x faster inference  
  
messages = \[  
    {"role": "user", "content": "Describe a tall tower in the capital of France."},  
\]  
inputs = tokenizer.apply\_chat\_template(  
    messages,  
    tokenize = True,  
    add\_generation\_prompt = True, # Must add for generation  
    return\_tensors = "pt",  
).to("cuda")  
  
from transformers import TextStreamer  
text\_streamer = TextStreamer(tokenizer, skip\_prompt = True)  
\_ = model.generate(input\_ids = inputs, streamer = text\_streamer, max\_new\_tokens = 128,  
                   use\_cache = True, temperature = 1.5, min\_p = 0.1)  

The Eiffel Tower, located in the heart of Paris, stands tall among the city's historic and cultural landmarks. This iron structure, standing at an impressive 324 meters high, offers breathtaking views of the City of Light's iconic landscape. The Eiffel Tower was built for the 1889 World's Fair and has since become a symbol of French engineering and culture.<|eot\_id|>

You can also use Hugging Face's `AutoModelForPeftCausalLM`. Only use this if you do not have `unsloth` installed. It can be hopelessly slow, since `4bit` model downloading is not supported, and Unsloth's **inference is 2x faster**.

if False:  
    # I highly do NOT suggest - use Unsloth if possible  
    from peft import AutoPeftModelForCausalLM  
    from transformers import AutoTokenizer  
    model = AutoPeftModelForCausalLM.from\_pretrained(  
        "lora\_model", # YOUR MODEL YOU USED FOR TRAINING  
        load\_in\_4bit = load\_in\_4bit,  
    )  
    tokenizer = AutoTokenizer.from\_pretrained("lora\_model")  

### Saving to float16 for VLLM

We also support saving to `float16` directly. Select `merged_16bit` for float16 or `merged_4bit` for int4. We also allow `lora` adapters as a fallback. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to [https://huggingface.co/settings/tokens](https://www.google.com/url?q=https%3A%2F%2Fhuggingface.co%2Fsettings%2Ftokens) for your personal tokens.

# Merge to 16bit  
if False: model.save\_pretrained\_merged("model", tokenizer, save\_method = "merged\_16bit",)  
if False: model.push\_to\_hub\_merged("hf/model", tokenizer, save\_method = "merged\_16bit", token = "")  
  
# Merge to 4bit  
if False: model.save\_pretrained\_merged("model", tokenizer, save\_method = "merged\_4bit",)  
if False: model.push\_to\_hub\_merged("hf/model", tokenizer, save\_method = "merged\_4bit", token = "")  
  
# Just LoRA adapters  
if False: model.save\_pretrained\_merged("model", tokenizer, save\_method = "lora",)  
if False: model.push\_to\_hub\_merged("hf/model", tokenizer, save\_method = "lora", token = "")  

### GGUF / llama.cpp Conversion

To save to `GGUF` / `llama.cpp`, we support it natively now! We clone `llama.cpp` and we default save it to `q8_0`. We allow all methods like `q4_k_m`. Use `save_pretrained_gguf` for local saving and `push_to_hub_gguf` for uploading to HF.

Some supported quant methods (full list on our [Wiki page](https://github.com/unslothai/unsloth/wiki#gguf-quantization-options)):

-   `q8_0` - Fast conversion. High resource use, but generally acceptable.
-   `q4_k_m` - Recommended. Uses Q6\_K for half of the attention.wv and feed\_forward.w2 tensors, else Q4\_K.
-   `q5_k_m` - Recommended. Uses Q6\_K for half of the attention.wv and feed\_forward.w2 tensors, else Q5\_K.

\[**NEW**\] To finetune and auto export to Ollama, try our [Ollama notebook](https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ?usp=sharing)

# Save to 8bit Q8\_0  
if False: model.save\_pretrained\_gguf("model", tokenizer,)  
# Remember to go to https://huggingface.co/settings/tokens for a token!  
# And change hf to your username!  
if False: model.push\_to\_hub\_gguf("hf/model", tokenizer, token = "")  
  
# Save to 16bit GGUF  
if False: model.save\_pretrained\_gguf("model", tokenizer, quantization\_method = "f16")  
if False: model.push\_to\_hub\_gguf("hf/model", tokenizer, quantization\_method = "f16", token = "")  
  
# Save to q4\_k\_m GGUF  
if False: model.save\_pretrained\_gguf("model", tokenizer, quantization\_method = "q4\_k\_m")  
if False: model.push\_to\_hub\_gguf("hf/model", tokenizer, quantization\_method = "q4\_k\_m", token = "")  
  
# Save to multiple GGUF options - much faster if you want multiple!  
if False:  
    model.push\_to\_hub\_gguf(  
        "hf/model", # Change hf to your username!  
        tokenizer,  
        quantization\_method = \["q4\_k\_m", "q8\_0", "q5\_k\_m",\],  
        token = "", # Get a token at https://huggingface.co/settings/tokens  
    )  

Now, use the `model-unsloth.gguf` file or `model-unsloth-Q4_K_M.gguf` file in llama.cpp or a UI based system like Jan or Open WebUI. You can install Jan [here](https://github.com/janhq/jan) and Open WebUI [here](https://github.com/open-webui/open-webui)

And we're done! If you have any questions on Unsloth, we have a [Discord](https://www.google.com/url?q=https%3A%2F%2Fdiscord.gg%2Funsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!

Some other links:

1.  Llama 3.2 Conversational notebook. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_\(1B_and_3B\)-Conversational.ipynb)
2.  Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_\(8B\)-Ollama.ipynb)
3.  Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_\(11B\)-Vision.ipynb)
4.  See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://www.google.com/url?q=https%3A%2F%2Fdocs.unsloth.ai%2Fget-started%2Funsloth-notebooks)!

[![](https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png)](https://www.google.com/url?q=https%3A%2F%2Funsloth.ai) [![](https://github.com/unslothai/unsloth/raw/main/images/Discord.png)](https://www.google.com/url?q=https%3A%2F%2Fdiscord.gg%2Funsloth) [![](https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true)](https://www.google.com/url?q=https%3A%2F%2Fdocs.unsloth.ai%2F)

Join Discord if you need help + ⭐️ _Star us on [Github](https://github.com/unslothai/unsloth)_ ⭐️

hey so im gonna be doing this can u help walk me thru it ? i am currently converting my csv to json and im wondering if the format should be like {"role": "system", "content": "You are an assistant"} {"role": "user", "content": "What is 2+2?"} {"role": "assistant", "content": "It's 4."} or how it should be set up