Special tokens to pre-trained BART model
Author: loretoparisi
Pull request: no
Labels: name: Core: Tokenization

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
Is it possibile to add special tokens to the pre-trained BART model? My text has `
` as sequence separator for sentences. I would like that the encoder will handle it as a whole token, otherwise the model will break it in codes and learn like `<s` or `s>` etc. in the same we did  for other tokenizers like `GPT2Tokenizer`?

```python
tokenizer = GPT2Tokenizer.from_pretrained(args.out, 
                                                    unk_token="<unk>",
                                                    bos_token="
", 
                                                    eos_token="
", 
                                                    pad_token = "<pad>", 
                                                    additional_special_tokens=["<startoflyrics>",  "<endoflyrics>", "<nl>"])
```

Comments:
Hi!
Two points that might be helpful.
- The `add_special_tokens` functionality should work the same as `RobertaTokenizer`. 
- `
` is already the `bos` token, so I don't expect it to be broken up.

Let me know if that resolves your issue, thanks!


Not sure how to go about doing this ?
@sshleifer  any code example 

i see BartTokenizer is essentially RobertaTokenizer which is GPT2Tokenizer


for fine-tuning BART in lightning base we have

  ```
      self.tokenizer = AutoTokenizer.from_pretrained(
            self.hparams.tokenizer_name if self.hparams.tokenizer_name else self.hparams.model_name_or_path,
            cache_dir=cache_dir,
        )
```

Can we add the list of special tokens here ?
If then how ?

Are you trying to add tokens to the vocab and give them new ids?
A specific example with what you expect the tokenizer to produce would be helpful.

I tried the following and it doesn't work as OP intended, afaict.

```python
from transformers import BartTokenizer
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large',
                                          additional_special_tokens=["<startoflyrics>", 'dringus'])

encoded = tokenizer.encode_plus(' <startoflyrics> dringus')['input_ids'] # [0, 3, 3, 2]
tokenizer.decode(encoded) # '
<unk><unk>
'
```



Yes @sshleifer  i want to add new  tokens to the vocab and give them new ids
How to go about doing it ?

@patrickvonplaten @LysandreJik what is the canonical way to add new non-special tokens? 
(1) Is there an easier way than making a new vocab and merges file?
(2) If not, is there an example of how to do that?

I am only familiar with the `add_special_tokens` functionality for new tokens that get the "special tokens" treatment.

For normal tokens, one can use `add_tokens` as far as I know. 

```
self.tokenizer = AutoTokenizer.from_pretrained(
            self.hparams.tokenizer_name if self.hparams.tokenizer_name else self.hparams.model_name_or_path,
            cache_dir=cache_dir,
        )
self.model = MODEL_MODES[mode].from_pretrained(
            self.hparams.model_name_or_path,
            from_tf=bool(".ckpt" in self.hparams.model_name_or_path),
            config=self.config,
            cache_dir=cache_dir,
        )
self.tokenizer.add_tokens(['multi-sentence', ':snt1', ':snt2', ':snt3', ':snt4', ':snt5', ':snt5', ':snt6', ':snt7', ':snt8', ':snt9', ':root', ':ARG1', ':mod', ':op1', ':ARG0', ':ARG0-of', ':name', ':op2', ':ARG2', ':ARG1-of', ':purpose', ':prep-in', ':time', ':li', ':quant', ':unit', ':poss', ':ARG3', ':location', ':domain', ':part-of', ':manner', ':polarity', ':condition', ':ARG4', ':extent', ':time-of', ':location-of', ':op3', ':beneficiary', ':topic', ':degree', ':ARG2-of', ':example', ':extent-of', ':month', ':day', ':op4', ':ARG5', ':manner-of', ':concession', ':duration', ':path', ':mode', ':medium', ':ord', ':value', ':destination', ':source', ':direction', ':instrument-of', ':consist-of', ':dayperiod', ':frequency', ':year', ':quant-of', ':weekday', ':compared-to', ':prep-on', ':ARG3-of', ':degree-of', ':prep-as', ':instrument', ':op5', ':prep-from', ':prep-to', ':century', ':era', ':condition-of', ':op6', ':op7', ':concession-of', ':polite', ':age', ':prep-with', ':decade', ':poss-of', ':prep-without', ':prep-in-addition-to', ':accompanier', ':ord-of', ':direction-of', ':prep-against', ':prep-at', ':subevent-of', ':snt10', ':snt11', ':duration-of', ':prep-for', ':source-of', ':frequency-of', ':topic-of', ':season', ':path-of', ':op8', ':op9', ':prep-among', ':prep-on-behalf-of', ':subevent', ':part', ':ARG4-of', ':beneficiary-of', ':scale', ':example-of', ':prep-by', ':range', ':purpose-of', ':destination-of', ':op10', ':op1-of', ':name-of', ':medium-of', ':prep-along-with', ':conj-as-if', ':timezone', ':prep-under', ':accompanier-of', ':age-of', ':op11', ':op12', ':op13', ':op14', ':op15', ':prep-amid', ':prep-toward', ':prep-out-of', ':prep-into', ':domain-of', ':ARG7', ':quarter', ':ARG5-of', ':op16', ':op17', ':op18', ':op19', ':op20', ':ARG8', ':ARG9', ':calendar', ':year2', ':ARG6', ':subset-of', ':prep-with-of'])
self.model.resize_token_embeddings(len(self.tokenizer))
```


This worked for me


Yes, the `add_tokens` that @patrickvonplaten and @tuhinjubcse mentionned should get the job done

I need to add tokens that will serve as separator in the text generation. For instance:
^Input:^Bitocoin price went down for 10 percent^Caption:^10% OFF^output:^10% reduce of the bitcoin price.
So in this example is ^input:^, ^Caption:^ and ^output:^. The idea is when I give to the train model the sentence:
^Input:^Bitocoin price went down for 10 percent^Caption:^ it should generate text but the model should learn the static tokens in the example. Should I use add_tokens or add_special_tokens?