Adding New Vocabulary Tokens to the Models
Author: vyraun
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi,

How could I extend the vocabulary of the pre-trained models, e.g. by adding new tokens to the lookup table?

Any examples demonstrating this?

Comments:
Hi, I believe this method does exactly what you're looking for: [add_tokens](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.add_tokens). There's an example right below it.

thanks @LysandreJik ! yes, that's exactly what I was looking for. A follow-up question: How could I initialize the embeddings of these "new tokens" to something I already have pre-computed? I assume currently, embedding for these new tokens will be randomly initialized.

You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.

```py
import torch
from transformers import BertTokenizer, BertModel

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
model = BertModel.from_pretrained("bert-base-cased")

print(len(tokenizer))  # 28996
tokenizer.add_tokens(["NEW_TOKEN"])
print(len(tokenizer))  # 28997

model.resize_token_embeddings(len(tokenizer)) 
# The new vector is added at the end of the embedding matrix

print(model.embeddings.word_embeddings.weight[-1, :])
# Randomly generated matrix

model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])

print(model.embeddings.word_embeddings.weight[-1, :])
# outputs a vector of zeros of shape [768]
```

thanks @LysandreJik ! That should solve it quite neatly. I will reopen the issue in case I run into any issues. 

Hello @LysandreJik ,

What is the difference between the following approaches?
  1. to train a tokenizer from scratch such as pointed in [hugginface blog](https://huggingface.co/blog/how-to-train#2-train-a-tokenizer); or
  2. to use [add_tokens](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.add_tokens) method?

Thank you in advance.





Training a tokenizer from scratch would imply training a model from scratch as well - depending on the corpus used for the tokenizer, the tokens may be entirely different from another model's tokens trained on a similar corpus (except if you train the tokenizer using the exact same method and the exact same data).

Adding tokens adds tokens at the end of the tokenizer's vocabulary, essentially extending the vocabulary. The model's embedding matrix would need to be resized as well to take into account the new tokens, but all the other tokens would keep their representation as-is. Seeing as the new rows in the embedding matrix are randomly initialized, you would still need to fine-tune the model to a dataset containing such tokens.

@LysandreJik 
I have a dutch medical dataset (for Namen Entity Recognition) which contains a lot of domain-specific words. The dutch BERT tokenizer therefor outputs a lot of [UNK] tokens when it tokenizes. 
Given that I dispose over a corpus of 60k labelled tokens, and right now I have also a relatively small annotated corpus of 185k tokens, would it be best to:
- just add the most frequent out of vocab words to the vocab of the tokenizer
- start from a BERT checkpoint and do further pretraining on the unlabeled dataset (which is now of size 185k which is pretty small I assume..). There might be a possibility for me to obtain a much larger unannotated dataset of potentially millions of (unlabelled) tokens, but I was wondering if even millions of tokens is enough to do some meaningful further pretraining?
 
Thanks!

> Training a tokenizer from scratch would imply training a model from scratch as well - depending on the corpus used for the tokenizer, the tokens may be entirely different from another model's tokens trained on a similar corpus (except if you train the tokenizer using the exact same method and the exact same data).
> 
> Adding tokens adds tokens at the end of the tokenizer's vocabulary, essentially extending the vocabulary. The model's embedding matrix would need to be resized as well to take into account the new tokens, but all the other tokens would keep their representation as-is. Seeing as the new rows in the embedding matrix are randomly initialized, you would still need to fine-tune the model to a dataset containing such tokens.

Hey I would like to fine-tune the model as you suggested at the end to the dataset containing such tokens. Can you help me out on how I can do that?

If I add unknown tokens to the tokenizer and train the model on, say sentence pair similarity, while I suppose the new tokens embeddings will not have the correct relationship with other tokens, will the model output still be able to find similarity correctly given sufficient training on the model?

@LysandreJik Thank you for your suggestion. However, I run into trouble because altering the embedding turns the embedding tensor into a non-leaf tensor and hence cannot be optimized i.e. 
``` python 
model.embeddings.word_embeddings.weight.is_leaf # False
```
I cannot figure out how to fix this (I am torch beginner; sorry). Do you have any suggestions? 

facing same issue; getting false for is_leaf

`BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True).get_vocab()` not return added token. How can I check if the new token is properly added to vocab dictionary? 

> You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> 
> ```python
> import torch
> from transformers import BertTokenizer, BertModel
> 
> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> model = BertModel.from_pretrained("bert-base-cased")
> 
> print(len(tokenizer))  # 28996
> tokenizer.add_tokens(["NEW_TOKEN"])
> print(len(tokenizer))  # 28997
> 
> model.resize_token_embeddings(len(tokenizer)) 
> # The new vector is added at the end of the embedding matrix
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # Randomly generated matrix
> 
> model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # outputs a vector of zeros of shape [768]
> ```

Hi,
I tried this, but my code still stop in tokenizing the sentences section and doesn't pass it.
it may have lag or problem...
what should I do?

> 
> 
> > You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> > ```python
> > import torch
> > from transformers import BertTokenizer, BertModel
> > 
> > tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> > model = BertModel.from_pretrained("bert-base-cased")
> > 
> > print(len(tokenizer))  # 28996
> > tokenizer.add_tokens(["NEW_TOKEN"])
> > print(len(tokenizer))  # 28997
> > 
> > model.resize_token_embeddings(len(tokenizer)) 
> > # The new vector is added at the end of the embedding matrix
> > 
> > print(model.embeddings.word_embeddings.weight[-1, :])
> > # Randomly generated matrix
> > 
> > model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> > 
> > print(model.embeddings.word_embeddings.weight[-1, :])
> > # outputs a vector of zeros of shape [768]
> > ```
> 
> Hi,
> I tried this, but my code still stop in tokenizing the sentences section and doesn't pass it.
> it may have lag or problem...
> what should I do?

Have you solved the problem? If so, can you share it with us?

> > > You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> > > ```python
> > > import torch
> > > from transformers import BertTokenizer, BertModel
> > > 
> > > tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> > > model = BertModel.from_pretrained("bert-base-cased")
> > > 
> > > print(len(tokenizer))  # 28996
> > > tokenizer.add_tokens(["NEW_TOKEN"])
> > > print(len(tokenizer))  # 28997
> > > 
> > > model.resize_token_embeddings(len(tokenizer)) 
> > > # The new vector is added at the end of the embedding matrix
> > > 
> > > print(model.embeddings.word_embeddings.weight[-1, :])
> > > # Randomly generated matrix
> > > 
> > > model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> > > 
> > > print(model.embeddings.word_embeddings.weight[-1, :])
> > > # outputs a vector of zeros of shape [768]
> > > ```
> > 
> > 
> > Hi,
> > I tried this, but my code still stop in tokenizing the sentences section and doesn't pass it.
> > it may have lag or problem...
> > what should I do?
> 
> Have you solved the problem? If so, can you share it with us?

yes, it was because it takes a very long time to add all tokens. and I installed transformers from source:
pip install -U git+https://github.com/huggingface/transformers ,due to recently it was merged a PR that should speed this up dramatically and my problem solved.

thank you!




------------------&nbsp;原始邮件&nbsp;------------------
发件人: ***@***.***&gt;; 
发送时间: 2021年5月10日(星期一) 下午2:11
收件人: ***@***.***&gt;; 
抄送: "Patrick ***@***.***&gt;; ***@***.***&gt;; 
主题: Re: [huggingface/transformers] Adding New Vocabulary Tokens to the Models (#1413)





    
You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings weight. Here's an example with the BertModel.
 import torch from transformers import BertTokenizer, BertModel tokenizer = BertTokenizer.from_pretrained("bert-base-cased") model = BertModel.from_pretrained("bert-base-cased") print(len(tokenizer))  # 28996 tokenizer.add_tokens(["NEW_TOKEN"]) print(len(tokenizer))  # 28997 model.resize_token_embeddings(len(tokenizer))  # The new vector is added at the end of the embedding matrix print(model.embeddings.word_embeddings.weight[-1, :]) # Randomly generated matrix model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size]) print(model.embeddings.word_embeddings.weight[-1, :]) # outputs a vector of zeros of shape [768]
  
Hi,
 I tried this, but my code still stop in tokenizing the sentences section and doesn't pass it.
 it may have lag or problem...
 what should I do?
  
Have you solved the problem? If so, can you share it with us?
  
yes, it was because it takes a very long time to add all tokens. and I installed transformers from source:
 pip install -U git+https://github.com/huggingface/transformers ,due to recently it was merged a PR that should speed this up dramatically and my problem solved.
 
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

> Training a tokenizer from scratch would imply training a model from scratch as well - depending on the corpus used for the tokenizer, the tokens may be entirely different from another model's tokens trained on a similar corpus (except if you train the tokenizer using the exact same method and the exact same data).
> 
> Adding tokens adds tokens at the end of the tokenizer's vocabulary, essentially extending the vocabulary. The model's embedding matrix would need to be resized as well to take into account the new tokens, but all the other tokens would keep their representation as-is. Seeing as the new rows in the embedding matrix are randomly initialized, you would still need to fine-tune the model to a dataset containing such tokens.

Why can't we repurpose the existing 999 unused tokens [UNK] instead of extending the vocab size? 
https://github.com/google-research/bert/issues/9#issuecomment-434796704

> You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> 
> ```python
> import torch
> from transformers import BertTokenizer, BertModel
> 
> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> model = BertModel.from_pretrained("bert-base-cased")
> 
> print(len(tokenizer))  # 28996
> tokenizer.add_tokens(["NEW_TOKEN"])
> print(len(tokenizer))  # 28997
> 
> model.resize_token_embeddings(len(tokenizer)) 
> # The new vector is added at the end of the embedding matrix
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # Randomly generated matrix
> 
> model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # outputs a vector of zeros of shape [768]
> ```

@LysandreJik when I ran your code the following error popped up. please help

**RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.**

> RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

You can fix that error by temporarily disabling gradient calculation. (Because initializing the weights is not an operation that needs to be accounted for in backpropagation.)

```python
with torch.no_grad():
    model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
```

why hidden_size? Is that specific to just Bert model? For Albert it should be different right?

How do we initialise the pre-existing embeddings for new tokens from old partitioned tokens?

> why hidden_size? Is that specific to just Bert model? For Albert it should be different right?

Hi, yes, I do believe the name can vary from model to model. For T5 model it seems to be `d_model`

> How do we initialise the pre-existing embeddings for new tokens from old partitioned tokens?

If I understand you correctly, we can initialise new tokens from already pre-trained ones with taking a mean of them: 
```
with torch.no_grad():
    for i, token in enumerate(reversed(added_tokens), start=1):
        tokenized = tokenizer.tokenize(token)
        tokenized_ids = tokenizer.convert_tokens_to_ids(tokenized)
        model.embeddings.word_embeddings.weight[-i, :] = model.embeddings.word_embeddings.weight[tokenized_ids].mean(axis=0)
```

> > How do we initialise the pre-existing embeddings for new tokens from old partitioned tokens?
> 
> If I understand you correctly, we can initialise new tokens from already pre-trained ones with taking a mean of them:
> 
> ```
> with torch.no_grad():
>     for i, token in enumerate(reversed(added_tokens), start=1):
>         tokenized = tokenizer.tokenize(token)
>         tokenized_ids = tokenizer.convert_tokens_to_ids(tokenized)
>         model.embeddings.word_embeddings.weight[-i, :] = model.embeddings.word_embeddings.weight[tokenized_ids].mean(axis=0)
> ```

Ok. Thank you. Is this also correct?

```
model.resize_token_embeddings(len(tokenizer))
weights = model.roberta.embeddings.word_embeddings.weight
    
# initialize new embedding weights as mean of original tokens
with torch.no_grad():
    emb = []
    for i in range(len(joined_keywords)):
        word = joined_keywords[i]
        # first & last tokens are just string start/end; don't keep
        tok_ids = tokenizer_org(word)["input_ids"][1:-1]
        tok_weights = weights[tok_ids]

        # average over tokens in original tokenization
        weight_mean = torch.mean(tok_weights, axis=0)
        emb.append(weight_mean)
    weights[-len(joined_keywords):,:] = torch.vstack(emb).requires_grad_()
```

How should I save new tokenizer to use it in downstream model?


```
tokenizer_org = tr.BertTokenizer.from_pretrained("/home/pc/bert_base_multilingual_uncased")
tokenizer.add_tokens(joined_keywords)
model = tr.BertForMaskedLM.from_pretrained("/home/pc/bert_base_multilingual_uncased", return_dict=True)

# prepare input
text = ["Replace me by any text you'd like"]
encoded_input = tokenizer(text, truncation=True, padding=True, max_length=512, return_tensors="pt")
print(encoded_input)


# add embedding params for new vocab words
model.resize_token_embeddings(len(tokenizer))
weights = model.bert.embeddings.word_embeddings.weight
    
# initialize new embedding weights as mean of original tokens
with torch.no_grad():
    emb = []
    for i in range(len(joined_keywords)):
        word = joined_keywords[i]
        # first & last tokens are just string start/end; don't keep
        tok_ids = tokenizer_org(word)["input_ids"][1:-1]
        tok_weights = weights[tok_ids]

        # average over tokens in original tokenization
        weight_mean = torch.mean(tok_weights, axis=0)
        emb.append(weight_mean)
    weights[-len(joined_keywords):,:] = torch.vstack(emb).requires_grad_()

model.to(device)

```

`trainer.save_model("/home/pc/Bert_multilingual_exp_TCM/model_mlm_exp1")`

**It saves model, config, training_args. How to save the new tokenizer as well??**

I am not sure if anyone can help to answer this here but I cannot seems to be able to find an answer from anywhere:
what exactly is the difference between "token" and a "special token"?

I understand the following:
* what is a typical token
* what is a typical special token: MASK, UNK, SEP, etc
* when do you add a token (when you want to expand your vocab)

What I don't understand is, under what kind of capacity will you want to create a new special token, any examples what we need it for and when we want to create a special token other than those default special tokens? If an example uses a special token, why can't a normal token achieve the same objective?

```
tokenizer.add_tokens(['[EOT]'], special_tokens=True)
```

And I also dont quite understand the following description in the source documentation.
what difference does it do to our model if we set add_special_tokens to False? 

```
add_special_tokens (bool, optional, defaults to True) — Whether or not to encode the sequences with the special tokens relative to their model.
```

> I am not sure if anyone can help to answer this here but I cannot seems to be able to find an answer from anywhere: what exactly is the difference between "token" and a "special token"?
> 
> I understand the following:
> 
> * what is a typical token
> * what is a typical special token: MASK, UNK, SEP, etc
> * when do you add a token (when you want to expand your vocab)
> 
> What I don't understand is, under what kind of capacity will you want to create a new special token, any examples what we need it for and when we want to create a special token other than those default special tokens? If an example uses a special token, why can't a normal token achieve the same objective?
> 
> ```
> tokenizer.add_tokens(['[EOT]'], special_tokens=True)
> ```
> 
> And I also dont quite understand the following description in the source documentation. what difference does it do to our model if we set add_special_tokens to False?
> 
> ```
> add_special_tokens (bool, optional, defaults to True) — Whether or not to encode the sequences with the special tokens relative to their model.
> ```

When you add a "special token" it will not be replaced by the "[MASK]" or replaced by a random word in the pre-training procedure.

> You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> 
> ```python
> import torch
> from transformers import BertTokenizer, BertModel
> 
> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> model = BertModel.from_pretrained("bert-base-cased")
> 
> print(len(tokenizer))  # 28996
> tokenizer.add_tokens(["NEW_TOKEN"])
> print(len(tokenizer))  # 28997
> 
> model.resize_token_embeddings(len(tokenizer)) 
> # The new vector is added at the end of the embedding matrix
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # Randomly generated matrix
> 
> model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # outputs a vector of zeros of shape [768]
> ```

Has anything changed in the past 4 years and how would one do this with a custom / self trained / specialised model? I wanted to add some more tokens to help with training and prompting, so that it doesn't split words it don't know into multiple tokens and in turn damage concepts it already knows or generate garbage.

Hey! Nothing much is different in terms of code, we leave it to the user to define the new embeddings, but bunch of tutorials give good ideas of how to do this well: https://nlp.stanford.edu/~johnhew/vocab-expansion.html 

> You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> 
> ```python
> import torch
> from transformers import BertTokenizer, BertModel
> 
> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> model = BertModel.from_pretrained("bert-base-cased")
> 
> print(len(tokenizer))  # 28996
> tokenizer.add_tokens(["NEW_TOKEN"])
> print(len(tokenizer))  # 28997
> 
> model.resize_token_embeddings(len(tokenizer)) 
> # The new vector is added at the end of the embedding matrix
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # Randomly generated matrix
> 
> model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # outputs a vector of zeros of shape [768]
> ```

In this way, I received a warning:
`Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.`

**Should we write specific codes to fine-tune the word embedding?**

Thank you very much!

Best regards,

Shuyue
Nov. 27th, 2023


> Hey! Nothing much is different in terms of code, we leave it to the user to define the new embeddings, but bunch of tutorials give good ideas of how to do this well: https://nlp.stanford.edu/~johnhew/vocab-expansion.html

Cool, but i barely know programming. Esp not python, nor fancy stuff written and used by techwizards. To me this is like magic. I use A1111 to generate images, and i train checkpoints with kohya. Where does this fit into that so i can add more tokens? How and where do i get the tokenizer and model, is it the same as clip?

I tried to ask similar question specifically about SD on the SD subreddit, but the spam filter keeps deleting my post, with no explanation as to why. No wonder i dont find any one asking this question if reddit deletes all posts about it.

With the huge influx of new people to AI in the past year, i feel this stuff needs to be more readily available, and not "write this thousand lines of python code from this 2 lines of code as a hint", possibly by having either a program or plugin where you can just give it a text file. Feels like it would be more convenient even for the veterans, rather then the veterans gatekeeping it behind learning python for 5 years. I am not asking you to write it for me, but google might be fucking with me and not giving me what i search for, but someone has to have made a program to add new tokens to a checkpoint by now. I cant be the only person that would want to do this.

Hey! Totally feel your pain and sorry for the whole in the documentation for this! I'll open a pull request to make this more visible. Here is a snippet of how to do this:
```python
from transformers import GPT2Tokenizer, GPT2Model
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2Model.from_pretrained("gpt2")
num_new_tokens = tokenizer.add_tokens(["new_token_1", "new_token_2"], special_tokens=True)

# simple resize
model.resize_token_embeddings(len(tokenizer))

# overwrite the content to have better results
input_embeddings = model.wte.weight.data
input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
input_embeddings[-num_new_tokens:] = input_embeddings_avg
```
hope it helps! 🤗 

> Hey! Totally feel your pain and sorry for the whole in the documentation for this! I'll open a pull request to make this more visible. Here is a snippet of how to do this:
> 
> ```python
> from transformers import GPT2Tokenizer, GPT2Model
> import torch
> 
> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
> model = GPT2Model.from_pretrained("gpt2")
> num_new_tokens = tokenizer.add_tokens(["new_token_1", "new_token_2"], special_tokens=True)
> 
> # simple resize
> model.resize_token_embeddings(len(tokenizer))
> 
> # overwrite the content to have better results
> input_embeddings = model.wte.weight.data
> input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
> input_embeddings[-num_new_tokens:] = input_embeddings_avg
> ```
> 
> hope it helps! 🤗

Thank you very much for your suggestion!

Best regards,

Shuyue
Nov. 29th, 2023

So i tried having a crack at this but struggling allot. It is more complicated since what im looking for is stored in a .safetensor file. I tried spending a few hours using different AI chats to try and do this with no results. The follow code is all i got. I dont know how to actually get the state_dict due to the safetensor format complicating things, or how to save what i have done back to a fully functional safetensor file. Someone want to help me with this?

``` 
# Load the tokenizer and model
tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32")
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")

# Path to the .safetensor file
safetensor_file = 'checkpoint.safetensors'

# Load the model parameters from the .safetensor file
state_dict = ?????????????????????????????????????????

# Update the model's state_dict with the loaded parameters
model.load_state_dict(state_dict)

# Path to the text file containing new tokens (one token per line)
new_tokens_file = 'tokens.txt'

# Read the tokens from the text file
file = open('myfile.txt', 'r')
tokens = file.readlines()

# Add the tokens
for token in tokens:
    tokenizer.add_tokens([token])

model.resize_token_embeddings(len(tokenizer))

# Save the updated .safetensor file with the modified vocabulary
??????????????????????????????????????
```

@TeKett I don't understand what you are trying to do. Could you explain it to me so that I can give you the exact code snippet? 

> @TeKett I don't understand what you are trying to do. Could you explain it to me so that I can give you the exact code snippet?

I want to add more tokens/vocabulary to a model, this case CLIP, thats packaged inside of a SD 1.5 checkpoint of the Safetensor variety. The safetensor file contains VAE, Unet, and CLIP.

```python 
from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler
import torch 

pipeline = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")

pipe = StableDiffusionPipeline
num_new_tokens = pipeline.tokenizer.add_tokens(["new_token_1", "new_token_2"], special_tokens=True)

# simple resize
pipeline.text_encoder.resize_token_embeddings(len(pipeline.tokenizer))

# overwrite the content to have better results
input_embeddings = pipeline.text_encoder.get_input_embeddings().weight.data
input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
input_embeddings[-num_new_tokens:] = input_embeddings_avg
pipeline.text_encoder.set_input_embeddings(input_embeddings)
```
should work 😉 

> ```python
> from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler
> import torch 
> 
> pipeline = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
> 
> pipe = StableDiffusionPipeline
> num_new_tokens = pipeline.tokenizer.add_tokens(["new_token_1", "new_token_2"], special_tokens=True)
> 
> # simple resize
> pipeline.text_encoder.resize_token_embeddings(len(pipeline.tokenizer))
> 
> # overwrite the content to have better results
> input_embeddings = pipeline.text_encoder.get_input_embeddings().weight.data
> input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
> input_embeddings[-num_new_tokens:] = input_embeddings_avg
> pipeline.text_encoder.set_input_embeddings(input_embeddings)
> ```
> 
> should work 😉

Can i give 'add_tokens' an array, or loop it? Should i set 'special_tokens=false' since im not going to add special tokens? Is the pipeline object (line 4) a pointer to the physical file, or only to the object in memory, if so how do i save it to the disk?


1. `add_tokens` can take an array
2. You can set `special_tokens` to false if they are not special 
3. The pipeline object already has the model loaded you can do `pipeline.model.save_pretrained("path_you_want")`
🤗 

I finally chose the following solution:

```python

DEFAULT_BOS_TOKEN = "
"
DEFAULT_EOS_TOKEN = "
"
DEFAULT_PAD_TOKEN = "[PAD]"
DEFAULT_UNK_TOKEN = "<unk>"

def tokenizer_embedding_resize(special_tokens_dict, tokenizer, model):
    """Resize tokenizer and embedding.

    Note: This is the unoptimized version that may make your embedding size not be divisible by 64.
    """
    num_new_tokens = tokenizer.add_special_tokens(special_tokens_dict)
    model.resize_token_embeddings(len(tokenizer))

    if num_new_tokens > 0:
        input_embeddings = model.get_input_embeddings().weight.data
        output_embeddings = model.get_output_embeddings().weight.data

        input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
        output_embeddings_avg = output_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)

        input_embeddings[-num_new_tokens:] = input_embeddings_avg
        output_embeddings[-num_new_tokens:] = output_embeddings_avg


def add_special_token(tokenizer):
    """
    Add special tokens to the tokenizer
    """
    tokenizer.add_special_tokens(
        {
            "pad_token": DEFAULT_PAD_TOKEN,
            "eos_token": DEFAULT_EOS_TOKEN,
            "bos_token": DEFAULT_BOS_TOKEN,
            "unk_token": DEFAULT_UNK_TOKEN,
        }
    )

    return tokenizer


# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    cache_dir=save_dir,
    model_max_length=train_max_len,
    add_eos_token=True,
    add_bos_token=True,
    padding='longest',
    padding_side="right",
    truncation=True,
    return_tensors="pt",
    use_fast=False,
    trust_remote_code=True,
    use_auth_token=hf_auth_token,
    device_map=device_map,
)
if tokenizer.pad_token is None:
    tokenizer_embedding_resize(
        special_tokens_dict=dict(pad_token="[PAD]"),
        tokenizer=tokenizer,
        model=model,
    )
tokenizer = add_special_token(tokenizer)

# Resize input token embeddings matrix of the model if new_num_tokens != config.vocab_size.
model.resize_token_embeddings(len(tokenizer))
```

The reference codes for the `tokenizer_embedding_resize()`: https://github.com/meta-math/MetaMath/blob/main/train_math.py#L90-L110

The reference codes for the `add_special_token()`: https://github.com/meta-math/MetaMath/blob/main/train_math.py#L259-L279

It works well on my side. 

Best regards,

Shuyue
Dec. 20th, 2023

> ```python
> from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler
> import torch 
> 
> pipeline = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
> 
> pipe = StableDiffusionPipeline
> num_new_tokens = pipeline.tokenizer.add_tokens(["new_token_1", "new_token_2"], special_tokens=True)
> 
> # simple resize
> pipeline.text_encoder.resize_token_embeddings(len(pipeline.tokenizer))
> 
> # overwrite the content to have better results
> input_embeddings = pipeline.text_encoder.get_input_embeddings().weight.data
> input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
> input_embeddings[-num_new_tokens:] = input_embeddings_avg
> pipeline.text_encoder.set_input_embeddings(input_embeddings)
> ```
> 
> should work 😉

Im unable to select the checkpoint. Since `StableDiffusionPipeline.from_pretrained` wants a path to a directory containing a pipeline object. I dont have that, what even is that? I cant stress enough that all i have is a SD 1.5 checkpoint, you know the one you load up into a1111 to generate images, that can be trained using Kohya, that are shared on civitai. I dont have a pipline object, and the Clip model i want to add tokens to is packaged inside of a .safetensor file.

`ValueError: The provided pretrained_model_name_or_path "C:/Train/checkpoint.safetensors" is neither a valid local path nor a valid repo id.`

If i give it just the directory i get `OSError: Error no file named model_index.json found in directory`

Alright, you can't load a pipeline without the configuration and the required sub-checkpoints like here https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main (AFAIK). Would recommend you to ask on the `diffusers` repo as this is outside the scope of transformers 🤗 

@ArthurZucker 
I have come across many similar issues asking about how to add new tokens to a vocabulary, for reference, here are a couple links to useful comments made for doing roughly that:

https://github.com/huggingface/transformers/issues/1413#issuecomment-538083512
https://github.com/huggingface/transformers/issues/2691#issuecomment-587473545
https://github.com/huggingface/tokenizers/issues/627#issuecomment-784286485
However, I am concerned with how to first identify tokens that make sense to add to an existing tokenizer's vocabulary, and also possibly whether or not it makes sense to consider removing tokens from a vocabulary.

Some context into my situation:

My situation I believe is quite typical: I have a reasonably large domain-specific text dataset available, all in English, and my end goal is produce a domain-specific language model that can be used for various downstream NLP tasks, (e.g., text classification, sentence similarity, etc), after additional fine-tuning on those tasks.

But from my current understanding, to first obtain that domain-specific language model, I basically have two options:

train a tokenizer from scratch and then use that tokenizer to train a LM from scratch.
modify the vocabulary of a pretrained tokenizer, adjust a (also pretrained) LM's embedding matrix to work with this new vocab size, and fine-tune the pretrained LM on my domain-specific text dataset on something like MLM.
I am struggling with the first option because (as far as I know) training a language model from scratch is quite expensive, and although I do have some budget for this, I do not have on the order of thousands of dollars.

I am starting to explore the second option, but I am confused on how to properly modify the vocabulary in a way that makes sense for the model, and concerned about what other side effects this could cause.

To summarize:

I'd really like to know if there is a low cost option for training a LM from scratch to do option 1 above
Or, if option 2 makes more sense, how to properly modify a vocabulary (find good new tokens, remove unused ones, etc), and adapt the model to overcome potential negative side effects of messing with the embeddings.
Thanks for the help. Sorry for a long question but I thought some context may be needed since I might be asking the wrong question in the first place. Cheers.   ##asked by someone.

Alright. If you need to add new tokens to the vocab but are not sure how, there are a few ways you can do this. 
1. Train a new tokenizer, using https://huggingface.co/learn/nlp-course/chapter6/2#training-a-new-tokenizer. This will make use of https://github.com/huggingface/transformers/blob/3eddda1111f70f3a59485e08540e8262b927e867/src/transformers/tokenization_utils_fast.py#L687 If you have a language specific data that uses none of the "old" tokens, that might be okay, but otherwises as you mentioned you would need to retrain the model. 
2. Train a new small tokenizer on a small corpus, merge the new vocab with the old vocab (merge the vocab and the merges if it is a BPE tokenizer by just adding the new tokens at the end) More on that here https://github.com/huggingface/tokenizers/issues/1109. Might not be optimal but if certain languages have less tokens it should be alright. 
3. Manually add all the new tokens using `add_tokens()`, which will just be adding characters / words for simplicity. Growing the vocab exponentially potentially if the vocab of the language is huge. 

I think that's pretty much it 😓 

So i asked over on diffusers and got no answer, then i asked again and got a response. All they did was argue over why i should not user their project for the exact reasons why the project exists... Basically said "Why are you trying to do this instead of being a sheep?", and they whont answer why the code is erroring out.

It errors out at line 18. `cannot assign 'torch.FloatTensor' as child module 'token_embedding'`

```
from diffusers import StableDiffusionPipeline

array = []
with open("D:/tagstest.txt",encoding="utf8") as file:
        array = [row.rstrip("\n") for row in file.readlines()]

pipeline = StableDiffusionPipeline.from_single_file("C:/Train/checkpoint.safetensors")

num_new_tokens = pipeline.tokenizer.add_tokens(array, special_tokens=False)

# simple resize (is this correct?)
pipeline.text_encoder.resize_token_embeddings(len(pipeline.tokenizer))

# overwrite the content to have better results
input_embeddings = pipeline.text_encoder.get_input_embeddings().weight.data
input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
input_embeddings[-num_new_tokens:] = input_embeddings_avg
pipeline.text_encoder.set_input_embeddings(input_embeddings) # Error
pipeline.model.save_pretrained("c:/test")
```

`pipeline.text_encoder.set_input_embeddings(input_embeddings) ` should be given a `nn.Embedding`  if I am not mistakend. Thus you first `get_input_embedding` change the data, and then `set_input_embeddings`

Problem was the types. `.weight.data` is a torch.FloatTensor, while `get_input_embeddings()` is a nn.whatever. I could just ommit this completely, no? Since all it does it unlearn the model?
```
input_embeddings = pipeline.text_encoder.get_input_embeddings()
input_embeddings_avg = input_embeddings.weight.data[:-num_new_tokens].mean(dim=0, keepdim=True)
input_embeddings.weight.data[-num_new_tokens:] = input_embeddings_avg
pipeline.text_encoder.set_input_embeddings(input_embeddings)
```

This likely falls under transformers, since i think its just a text model issue.
When i try to load the model again im getting
```
RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
	size mismatch for text_model.embeddings.token_embedding.weight: copying a param with shape torch.Size([90323, 768]) from checkpoint, the shape in current model is torch.Size([49408, 768]).
```
What is "current model"?

That just means the `config.vocab_size` is wrong and should be updated to `90323` current model is the one initialized with the config

> That just means the `config.vocab_size` is wrong and should be updated to `90323` current model is the one initialized with the config

You mean the config.json file in the text_encoder folder? It already says the new number.

```
{
  "architectures": [
    "CLIPTextModel"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "dropout": 0.0,
  "eos_token_id": 2,
  "hidden_act": "quick_gelu",
  "hidden_size": 768,
  "initializer_factor": 1.0,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 77,
  "model_type": "clip_text_model",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "projection_dim": 768,
  "torch_dtype": "float32",
  "transformers_version": "4.35.2",
  "vocab_size": 90320
}
```

> Alright. If you need to add new tokens to the vocab but are not sure how, there are a few ways you can do this.
> 
> 1. Train a new tokenizer, using https://huggingface.co/learn/nlp-course/chapter6/2#training-a-new-tokenizer. This will make use of https://github.com/huggingface/transformers/blob/3eddda1111f70f3a59485e08540e8262b927e867/src/transformers/tokenization_utils_fast.py#L687
>     If you have a language specific data that uses none of the "old" tokens, that might be okay, but otherwises as you mentioned you would need to retrain the model.
> 2. Train a new small tokenizer on a small corpus, merge the new vocab with the old vocab (merge the vocab and the merges if it is a BPE tokenizer by just adding the new tokens at the end) More on that here [How can I keep the initial input vocab and incremental add the new tokens during re-training a tokenizer?  tokenizers#1109](https://github.com/huggingface/tokenizers/issues/1109). Might not be optimal but if certain languages have less tokens it should be alright.
> 3. Manually add all the new tokens using `add_tokens()`, which will just be adding characters / words for simplicity. Growing the vocab exponentially potentially if the vocab of the language is huge.
> 
> I think that's pretty much it 😓

Hi @ArthurZucker, 
I tried to extend the tokenizer vocabulary using `add_tokens` method, but I got an odd behavior, not sure if I used correctly. I will try to demonstrate in the following example:

```
from transformers import BertTokenizer
original_tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

text = "California"
original_tokens = original_tokenizer.tokenize(text)
original_tokens # And here the tokenizer know the token, returning it with no issues.
```
Then I tried to add a token to the tokenizer
```
from transformers import BertTokenizer
original_tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

text = "California"
original_tokenizer.add_tokens(["rn"], special_tokens=False)
original_tokens = original_tokenizer.tokenize(text) # And here it returns ['Cal', '##if', '##o', 'rn', 'i', '##a']
```
I didn't understand why the tokenizer behaved like this. If I add a token B that is a substring of another token A, does this imply that the tokenizer will not recognize the A, like in the example?


```python
# tokenizer.save_pretrained("model")

with open("filecontainstokens.txt","r") as file:
    new_tokens = [line.strip() for line in file]

num_added_toks = tokenizer.add_tokens(new_tokens)
print('We have added', num_added_toks, 'tokens')

model.resize_token_embeddings(len(tokenizer))

average_embedding = torch.mean(model.get_input_embeddings().weight, axis=0)
for token_id in range(-num_added_toks, 0, 1):
    model.get_input_embeddings().weight.data[token_id, :] = average_embedding

new_embedding_display = model.get_input_embeddings().weight[-1]
# print(new_embedding_display)

len(tokenizer)
```

> That just means the `config.vocab_size` is wrong and should be updated to `90323` current model is the one initialized with the config

The problem seems to be that the tokens are never actually added to the text encoder, so the tokenizer says the new tokens, and the config says the new number, but the actual CLIPTextModel (the neural network?) doesn't match that. If i load the SD model, make some changes, save as Diffusers model, then convert to SD model. Then load it again and re-save it as Diffusers model, the tokenizer has the original tokens again before my changes. Clearly the values in the tokenizer is pulled from somewhere, the only place i can think of is the text encoder.

>I didn't understand why the tokenizer behaved like this. If I add a token B that is a substring of another token A, does this imply that the tokenizer will not recognize the A, like in the example?

You can control this using the `single_word` option of the `AddedToken`:
```python
from transformers import BertTokenizer, AddedToken
original_tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

text = "California"
original_tokenizer.add_tokens([AddedToken("rn", single_word=True)], special_tokens=False)
original_tokens = original_tokenizer.tokenize(text) 
```
which does not work here. Feel free to open a new issue, but bert tokenizer is old so I am not suprised that this does not work.
```python

In [16]: from transformers import AutoTokenizer, AddedToken
    ...: original_tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
    ...: 
    ...: text = "California"
    ...: original_tokenizer.add_tokens([AddedToken("rn", single_word=True)], special_tokens=False)
    ...: original_tokens = original_tokenizer.tokenize(text)

In [17]: original_tokens
Out[17]: ['California']
```

> > I didn't understand why the tokenizer behaved like this. If I add a token B that is a substring of another token A, does this imply that the tokenizer will not recognize the A, like in the example?
> 
> You can control this using the `single_word` option of the `AddedToken`:
> 
> ```python
> from transformers import BertTokenizer, AddedToken
> original_tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> 
> text = "California"
> original_tokenizer.add_tokens([AddedToken("rn", single_word=True)], special_tokens=False)
> original_tokens = original_tokenizer.tokenize(text) 
> ```
> 
> which does not work here. Feel free to open a new issue, but bert tokenizer is old so I am not suprised that this does not work.
> 
> ```python
> In [16]: from transformers import AutoTokenizer, AddedToken
>     ...: original_tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
>     ...: 
>     ...: text = "California"
>     ...: original_tokenizer.add_tokens([AddedToken("rn", single_word=True)], special_tokens=False)
>     ...: original_tokens = original_tokenizer.tokenize(text)
> 
> In [17]: original_tokens
> Out[17]: ['California']
> ```

Thanks, @ArthurZucker; I will open an issue. I tried another one that exploits XLMRoberta (`intfloat/multilingual-e5-small`) and got the same behavior. 

> You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> 
> ```python
> import torch
> from transformers import BertTokenizer, BertModel
> 
> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> model = BertModel.from_pretrained("bert-base-cased")
> 
> print(len(tokenizer))  # 28996
> tokenizer.add_tokens(["NEW_TOKEN"])
> print(len(tokenizer))  # 28997
> 
> model.resize_token_embeddings(len(tokenizer)) 
> # The new vector is added at the end of the embedding matrix
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # Randomly generated matrix
> 
> model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # outputs a vector of zeros of shape [768]
> ```


Hi @ArthurZucker  thank you for this insightful message. So after adding new tokens to the model, how do I get this new tokens to reflect in the tokenizer? Do  I train a new tokenizer from scratch? If not, please are the specific modifications I need to the tokenizer?

I am not sure I understand, if you add the tokens to the tokenizer, it should already relfect! 

> I am not sure I understand, if you add the tokens to the tokenizer, it should already relfect!

Ahh, I get now, just updating the json file would suffice. I stumbled on your post while looking for answers on how to extend the vocab of an Nvidia-Nemo model. I know this is not directly related to this thread but I would appreciate if you share anything you know about this with me.  

😅  I am not familiar at all with the `nemo` library, so no idea here! 

Hi @ArthurZucker, my target is to successfully merge:
1. A sentencepiece BPE tokenizer trained from scratch on a custom domain and use it to extend the vocab of a pre-trained tokenizer of the same family - COMPLETED
2. A tiktoken based bytelevel BPE tokenizer trained from scratch and use it to extend the vocab of Llama 3 tokenizer - **STUCK HERE**

Little intro to my problem. My aim is to build or adapt any open-source LLM to one or more Indian languages without losing its existing knowledge. Later, this can be fine-tuned for various downstream tasks in Indic languages. In my knowledge, there are two options. 
1. Build a tokenizer and a LLM from scratch using all the english and Indic language corpus - Like we all know, this is super expensive and not a feasible option for me.
2. Train a new tokenizer either using sentencepiece or huggingface repo on my custom language corpus. Then, extend the vocab of any existing tokenizer associated with a corresponding pre-trained LLM. Finally, resize the embedding layer of the LLM and continue pre-training on my custom language corpus. - This is a feasible option for me.

So, far I figured out how to train a sentencepiece BPE tokenizer from scratch and then merge this with a pre-trained tokenizer. Here's the code for it:
```
# Load the pre-trained tokenizer to be extended
original_tokenizer_path = hf_hub_download(repo_id="mistralai/mistral-7b-v0.1", filename="tokenizer.model", local_dir="original_tokenizer")
original_tokenizer_spm = sp_pb2_model.ModelProto()
original_tokenizer_spm.ParseFromString(open(original_tokenizer_path, "rb").read())

# Load the newly trained tokenizer
new_tokenizer_spm = sp_pb2_model.ModelProto()
new_tokenizer_spm.ParseFromString(open("/content/mistral_tel_tokenizer.model", "rb").read())


# Check if the new tokenizer contains english tokens
def contains_eng(text):
    eng_pattern = re.compile(r"[\u0020-\u007E]+")
    return True if eng_pattern.search(text) else False


original_tokenizer_tokenset = set(p.piece for p in original_tokenizer_spm.pieces)
print(f"Number of tokens before merge: {len(original_tokenizer_tokenset)}")
for p in new_tokenizer_spm.pieces:
    piece = p.piece
    if piece not in original_tokenizer_tokenset and not contains_eng(piece):
        new_p = sp_pb2_model.ModelProto().SentencePiece()
        new_p.piece = piece
        new_p.score = 0
        original_tokenizer_spm.pieces.append(new_p)
print(f"Number of tokens after merge: {len(original_tokenizer_spm.pieces)}")

# Save the extended tokenizer to a checkpoint
extended_tokenizer_save_path="/content/english-telugu-tokenizer"
os.makedirs(extended_tokenizer_save_path, exist_ok=True)
with open(os.path.join(extended_tokenizer_save_path, "tokenizer.model"), "wb") as f:
    f.write(original_tokenizer_spm.SerializeToString())
```
    
The new merged tokenizer is working efficiently on both languages (english and telugu). I adapted this code from here: https://github.com/google/sentencepiece/blob/master/python/add_new_vocab.ipynb  
    
Next, I trained a tiktoken's bytelevel tokenizer using the code from this repo: https://github.com/gautierdag/tokenizer-bench
**PROBLEM**
However, when I merged this new tokenizer with a pre-trained one, the telugu encoding performance got degraded. The merged tokenizer is splitting telugu text into too many fine-grained tokens. Encoding-Decoding for english, and decoding for telugu are working fine. Here's the code I used for merging:
```
import json

def merge_tokenizers(file1, file2, output_file):
    # Load the tokenizers
    with open(file1, 'r') as f:
        tokenizer1 = json.load(f)
    with open(file2, 'r') as f:
        tokenizer2 = json.load(f)

    # Get the maximum rank in tokenizer1's vocab
    max_rank = max(tokenizer1['model']['vocab'].values())

    # Combine the vocabs and merges
    combined_vocab = tokenizer1['model']['vocab'].copy()
    for token, rank in tokenizer2['model']['vocab'].items():
        if token not in combined_vocab:
            combined_vocab[token] = len(combined_vocab) + 1


    combined_merges = tokenizer1['model']['merges'].copy()
    for merge in tokenizer2['model']['merges']:
        if merge not in combined_merges:
            combined_merges.append(merge)

    # combined_merges = tokenizer1['model']['merges'].copy()
    # combined_merges.extend(merge for merge in tokenizer2['model']['merges'] if merge not in combined_merges)

    # Update the vocab and merges in tokenizer1
    tokenizer1['model']['vocab'] = combined_vocab
    tokenizer1['model']['merges'] = combined_merges

    # Save the updated tokenizer
    with open(output_file, 'w') as f:
        json.dump(tokenizer1, f)

# Usage
merge_tokenizers("/content/gpt_32k.json", "/content/telugu_tokenizer_tiktoken.json", 'tokenizer_18.json')
```
Can you help me in figuring out the correct way to merge so that the telugu encoding performance is brought back to the level of the individual new tokenizer trained only on telugu. 
I know, there's something wrong with the token and rank. In the case of sentencepiece, it is token and score. You have to initiate all the newly appended tokens with a score of 0. Here, I am not able to figure out what to do?

Seems related to https://github.com/huggingface/tokenizers/issues/627 

Hello, there is a new case I only want to training the the embedding the of new tokens, and keep the original embedding unchanged, is there a way to do this

Of course, you need to simply freeze the embedding of the old tokens. What people usually do is this: https://github.com/huggingface/transformers/blob/fff268dc1829ef43f28fffeb6f2ce4a27e766faf/src/transformers/models/idefics/modeling_idefics.py#L246 


either you create a new embedding layer, or you add partial embeddings that are trainable !

Thanks for your reply! It really helps. I'll try it.

> Of course, you need to simply freeze the embedding of the old tokens. What people usually do is this:
> 
> https://github.com/huggingface/transformers/blob/fff268dc1829ef43f28fffeb6f2ce4a27e766faf/src/transformers/models/idefics/modeling_idefics.py#L246
> 
> either you create a new embedding layer, or you add partial embeddings that are trainable !

So I did this 
```python
input_embeddings = model.get_input_embeddings()
new_embed = IdeficsDecoupledEmbedding(num_embeddings=len(tokenizer), num_additional_embeddings=5000, embedding_dim=input_embeddings.weight.shape[1], partially_freeze=True)
model.set_input_embeddings(new_embed)
```
But the gradients of the new model does not update during training. Do you know what could be wrong?

No idea, this should do it!


I think it might be some issues with how the Trainer sets the optimizer. 

Hi! I'm trying to add new tokens to the BERT tokenizer but facing some unexpected behavior.

Setup:
Initially, `inspect`, `ec`, `##ec`, `##t` tokens are present in the vocabulary and I'm adding `insp` to it.

Expectation:
When tokenizing `inspect`, I expect `inspect` to be the output token as BERT uses WordPiece tokenization that finds the longest subword that is in the vocabulary, then splits on it from left to right. 

What is observed:
`inspect` is split into three tokens: `insp`, `ec`, `##t`. This not only doesn't match with the expectation, but also uses `ec` token instead of `##ec` token, falsely indicating that `insp` and `ect` are separate words.

Any idea on this issue?


```
>> from transformers import BertTokenizer

>> model_checkpoint = 'bert-base-uncased'
>> tokenizer = BertTokenizer.from_pretrained(model_checkpoint)

>> ('inspect' in tokenizer.vocab, 'insp' in tokenizer.vocab, 'ec' in tokenizer.vocab, '##ec' in tokenizer.vocab, '##t' in tokenizer.vocab)
# (True, False, True, True, True)
>> tokenizer.convert_ids_to_tokens(tokenizer.encode('inspect'))
# ['[CLS]', 'inspect', '[SEP]']

>> tokenizer.add_tokens(['insp'])
# 1
>> ('inspect' in tokenizer.vocab, 'insp' in tokenizer.vocab, 'ec' in tokenizer.vocab, '##ec' in tokenizer.vocab, '##t' in tokenizer.vocab)
# (True, True, True, True, True)
>> tokenizer.convert_ids_to_tokens(tokenizer.encode('inspect'))
# ['[CLS]', 'insp', 'ec', '##t', '[SEP]']
```