"special token {} has to be either str or AddedToken but got:
Author: Shafi2016
Pull request: no
Labels: name: wontfix

Hello, two weeks ago I fine-tune the Roberta language model on my data using hugging face (codes given below) and save output at the google drive. Now when using it for semantic search with sentence transformer. I am getting the error that


**"TypeError: special token bos_token has to be either str or AddedToken but got: <class 'dict'>"**

```

from sentence_transformers import SentenceTransformer
from sentence_transformers import models, losses
import scipy.spatial
import pickle as pkl
word_embedding_model = models.RoBERTa("/content/drive/My Drive/Ottawa_citit")

# Apply mean pooling to get one fixed sized sentence vector
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(),
                               pooling_mode_mean_tokens=True,
                               pooling_mode_cls_token=False,
                               pooling_mode_max_tokens=False)

model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

 Previously fine-tuned Roberta

`!python "/content/transformers/examples/contrib/legacy/run_language_modeling.py" \
    --output_dir "/content/drive/My Drive/Ottawa_citit" \
    --model_name_or_path roberta-base \
    --do_train \
    --per_gpu_train_batch_size 8 \
    --seed 42 \
    --train_data_file "/content/input_text.txt" \
    --block_size 256 \
    --line_by_line \
    --learning_rate 6e-4 \
    --num_train_epochs 3 \
    --save_total_limit 2 \
    --save_steps 200 \
    --weight_decay 0.01 \
     --mlm`
```

Comments:
I had the same issue while using SentenceTransformers. When I checked with my backed up pretrained model, the tokenizer_config.json file had been altered for some reaon. I changed it back to the original and it was working fine. Probably some issue with SentenceTransformers calling Huggingface functions on the model.

If one of you (or @nreimers) want share a colab exhibiting the behavior, happy to investigate further and fix if needed!

Thanks a lot!!

I fine-tuned the Roberta language model again with hugging face and using it with a sentence transformer for semantic search. And no longer getting the error as mentioned above. But I don't know the reason for the error.






I also met this issue before. By uninstalling `sentence-transformers`(I think it may be also ok if you fix the version conflict issue), this bug disapeared.

Hi @Shafi2016 
Could you share you model, so that I can have a look?

Has there been any movement on this? I'm having the same issue with BartTokenizer using sschelifer/cnn-12-6

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

> Hi @Shafi2016 Could you share you model, so that I can have a look?

Hello @nreimers , I am trying to use paraphrase-multilingual-MiniLM-L12-v2 pretrained model and getting the similar error - 
TypeError: special token mask_token has to be either str or AddedToken but got: <class 'dict'>

Would it be possible for you share what change is needed in tokenizer_config.json? Accordingly I will change my tokenizer_config.json. 
This is the current version of tokenizer_config.json's mask_token dict - 
"mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}

One of the reason I would like to edit the tokenizer_config.json is to keep my transformer and sentence_transformer versions as is - 
sentence-transformers        0.4.1.2
transformers                 3.3.1

update transformers from 3.2.0 to 4.1.1 helps me solve the problem when call AutoTokenizer.from_pretrained get 'TypeError: special token bos_token has to be either str or AddedToken but got: <class 'dict'>'.