Helsinki-NLP

/

opus-mt-de-en

like 32

How do I add special tokens (like \n) to this model?

tokenizer.add_tokens('\n') seems to work, but CTranslate2 drops the \n when translating:

import ctranslate2
import transformers

translator = ctranslate2.Translator("opus-mt-de-en", device='cuda')
tokenizer = transformers.AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-de-en")

# add special token
tokenizer.add_tokens('\n')     # output:  1

tokenizer.added_tokens_decoder     # output: {58101: '\n'}
tokenizer.added_tokens_encoder     # output: {'\n': 58101}

source = tokenizer.convert_ids_to_tokens(tokenizer.encode("Guten\ntag!"))    # ==  ['▁Guten', '\n', '▁', 'tag', '!', '</s>']
results = translator.translate_batch([source], beam_size=5)     # == [TranslationResult(hypotheses=[['▁Good', '▁day', '!']], scores=[], attention=[])]      ← NOTICE THE `\n` IS DROPPED!

How do I get CTranslate2 to map token ID #58101 to \n?