understand differences in tokenization 
Author: bariluz93
Pull request: no

hi,
I'm trying to understand the tokenization in MarianTokenizer
I run the following code

from transformers import MarianTokenizer
model_name='Helsinki-NLP/opus-mt-en-de'
model = MarianMTModel.from_pretrained(model_name,tokenizer= tokenizer, **kwargs)


tokenizer.tokenize("doctor")
['▁doctor']

with tokenizer.as_target_tokenizer():
    indices = tokenizer("doctor", return_tensors="pt", padding=True)['input_ids'][0]
    tokens = tokenizer.convert_ids_to_tokens(indices)
    
indices
tensor([  156, 24889,    0])
tokens
['▁do', 'ctor', '
']

can someone please explain the difference between the tokenization of tokenizer.as_target_tokenizer(), and the tokenization of tokenizer.tokenize()? and what is used in fact when translating?
each one gives a different separation and different indices
thank you
Bar

Comments:
This is because the languages you're translating between (Engish and German in this case) have different tokenization vocabularies. This implies that tokens will get tokenized differently. MarianMT models have seq2seq (encoder-decoder) architectures, and both the encoder and decoder each have their own embedding matrix. This means that the encoder will have an embedding vector for the token '▁doctor', whereas the decoder will learn an embedding vector for the token '▁do', an embedding vector for the token 'ctor', etc.

Tokenization vocabularies are typically built per language (although models like BLOOM just have one large vocabulary for all language tokens).

thank you very much for your answer
is it per language? some languages don't have a separate embedding matrix for the encoder and the decoder?
is there a way to know in advance which language has separate matrices and which doesn't?
thanks
Bar


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@NielsRogge Can you please explain a confusion i have. Let say I have [en_ur translation](https://huggingface.co/Helsinki-NLP/opus-mt-en-ur) model.  I can see both urdu and english words in vocab. 

This is how data is trained.
```
model_inputs = tokenizer(inputs, max_length=max_Length, truncation=True)
    # Setup the tokenizer for targets
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(targets, max_length=max_Length, truncation=True)
```
So when I defined as_target_tokenizer is it read the same voab file? If yes then why we need to define this line because its the only vocab file

When using the `as_target_tokenizer` context manager, it will use the target vocabulary to tokenize the input sentence (rather than the source vocabulary).

However, in v4.22 we deprecated this context manager. Now

```
with tokenizer.as_target_tokenizer():
    encoded_labels = tokenizer(labels, padding=True)
```

can be replaced by:

```
encoded_labels = tokenizer(text_target=labels, padding=True)
```