Gemma-7b: huge train/loss after `resize_token_embeddings` 
Author: geronimi73
Pull request: no

Hey everyone,

Thank you for integrating gemma to HF so quickly! I tried finetuning `google/gemma-7b` and noticed that the train_loss **starts off in the hundreds** if I add new tokens to tokenizer and model. Also, the finetuned model is useless, repeats a single token for infinity.

To reproduce the high loss (not hundreds but still high), here is a slightly modified version of the [original QLoRA notebook](https://huggingface.co/google/gemma-7b/blob/main/examples/notebook_sft_peft.ipynb). 
The only modifications were:
* add new tokens and call `resize_token_embeddings`
* add `lm_head` and `embed_tokens` to `target_modules` of `LoraConfig`

```python3
import torch, os
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, GemmaTokenizer, set_seed
from datasets import load_dataset
from peft import LoraConfig
import transformers
from trl import SFTTrainer

set_seed(42)

model_id = "google/gemma-7b"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="auto")

# Extend vocab
tokenizer.add_tokens(["
"])
model.resize_token_embeddings(len(tokenizer))

data = load_dataset("Abirate/english_quotes")
data = data.map(lambda samples: tokenizer(samples["quote"]), batched=True)

def formatting_func(example):
    text = f"Quote: {example['quote'][0]}\nAuthor: {example['author'][0]}"
    return [text]

trainer = SFTTrainer(
    model=model,
    train_dataset=data["train"],
    args=transformers.TrainingArguments(
        per_device_train_batch_size=1,
        gradient_accumulation_steps=4,
        warmup_steps=2,
        max_steps=10,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=1,
        output_dir="outputs",
        optim="paged_adamw_8bit"
    ),
    peft_config=LoraConfig(
        r=8,
        target_modules=["q_proj", "o_proj", "k_proj", "v_proj", "gate_proj", "up_proj", "down_proj", "lm_head", "embed_tokens"],
        # target_modules=["q_proj", "o_proj", "k_proj", "v_proj", "gate_proj", "up_proj", "down_proj"],
        task_type="CAUSAL_LM",
    ),
    formatting_func=formatting_func,
)

trainer.train()
``` 

![W B Chart 22_02_2024, 16_56_43](https://github.com/huggingface/transformers/assets/141400217/c404cf5f-0599-4813-a3cb-c775e789399d)


I've never seen loss values this high with QLoRA (or any training) and I guess that's why the resulting model is crap. 

Any ideas on what might be causing this?
@ArthurZucker @younesbelkada @osanseviero 

```
transformers                      4.39.0.dev0
peft                              0.8.2
accelerate                        0.28.0
bitsandbytes                      0.42.0
trl                               0.7.11
torch                             2.2.0
```

Comments:
I'm seeing this as well, without LoRA. There are 100 unused special tokens in the tokenizer so I tried to replace my token with one of the unused token ID's and avoid resizing. This _still_ incurs very high loss.

<img width="282" alt="image" src="https://github.com/huggingface/transformers/assets/4743676/94285304-3a1f-43c9-8c69-972bfd337214">

@geronimi73  -- it looks like the special token you're adding is an instruction start token. however, Gemma doesn't use that token. could you instead use the built in chat template?

https://huggingface.co/google/gemma-2b-it#chat-template

> I'm seeing this as well, without LoRA. There are 100 unused special tokens in the tokenizer so I tried to replace my token with one of the unused token ID's and avoid resizing. This still incurs very high loss.

great idea! but weird that it still happens

> it looks like the special token you're adding is an instruction start token. however, Gemma doesn't use that token. could you instead use the built in chat template?

the idea is to make it use new tokens by fine-tuning. this works flawlessly with llama, mistral, phi but not with gemma. yes, I could use the existing chat template but it would not be chatml anymore.

To be honest there are some unused tokens in the vocabulary, instead of resizing I would use these because the vocab is already quite huge. To do that, you should eithee:
- change the `tokenizer.json` pretty easy: the `<unused_0>` can directly be replace. But if the model never saw them, would be better to update the embeddings of these tokens.
-  if you REALLY want to increase the vocab size: [this tutorial](https://nlp.stanford.edu/~johnhew/vocab-expansion.html) is one of the best! 
In both cases, curious about your futur findings! 
Closing as this is expected otherwise 

thank you

just in case someone else stumbles upon this, it works if you resize the token embeddings in a [smart way](https://github.com/artidoro/qlora/blob/7f4e95a68dc076bea9b3a413d2b512eca6d004e5/qlora.py#L425)

Have you confirmed that the loss is normal when not resizing the token embeddings?

> Have you confirmed that the loss is normal when not resizing the token embeddings?

yes

What is the correct way to use these `unused tokens`? It seems the tokenizer doesn't know they are individual tokens:
```python
>>> from transformers import AutoTokenizer
>>> model_id = "gg-hf/gemma-7b-it"
>>> tokenizer = AutoTokenizer.from_pretrained(model_id)
>>> tokenizer.tokenize("<unused0>")
['<', 'unused', '0', '>']
>>> tokenizer.convert_ids_to_tokens(7)
'<unused0>'
```

> What is the correct way to use these `unused tokens`? It seems the tokenizer doesn't know they are individual tokens:
> tokenizer.tokenize("<unused0>")
> ['<', 'unused', '0', '>']

tokenizers 🤯

by tradition I use the slow tokenizers because I always had issues with the fast one since llama1 already.

and indeed, your example works with the slow one:
```python
from transformers import AutoTokenizer
model_id = "gg-hf/gemma-7b-it"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
display(tokenizer.tokenize("<unused0>"))
# ['<unused0>']
```


Yeah this should fix this: #29473