Unclear how to decode a model's output
Author: George3d6
Pull request: no

## Unclear how to decode a model's output

Hello, after digging through the docs for about an hour it's still rather unclear to me how one is supposed to decode a model's output.

Using the following code:
```
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertModel.from_pretrained('distilbert-base-uncased')

input = torch.tensor(tokenizer.encode('Some text')).unsqueeze(0)
outputs = model(input)
lhs = outputs[0]
print(tokenizer.decode(lhs))
```

The lhs is always decoded as `[UNK]`

Is this just the expected result due to the model being untrained ? Is the decode functionality of the tokenizer being used in the wrong way ?

Searching for `decode` in the docs yields no code examples with it being used with a model's output.

Comments:
DistilBERT as any BERT is a Transformer encoder so it encodes a sequence of tokens into a vector in the embedding space. It doesn't return a sequence of tokens.

The output of the model is `return output # last-layer hidden-state, (all hidden_states), (all attentions)` https://github.com/huggingface/transformers/blob/master/transformers/modeling_distilbert.py#L484.

If you check the size of this hidden-state, it is `torch.Size([1, 4, 768])`.
`768` being the size of the hidden-state ie the size of the embedding vector.
`4` is the number of token in input sequence (`[CLS]` and `[SEP]` tokens are added by tokenizer)
`encode` is meant to return a sequence of token from a sequence of words.
`decode` is meant to return a sequence of words from a sequence of tokens.

So if you do:

```python
encoded = tokenizer.encode('Some text')
# encoded: [101, 2070, 3793, 102]
decoded = tokenizer.decode(encoded))
# decoded: [CLS] some text [SEP]
```

But, you can't use `decode` on the output of the model as it's not a sequence of tokens but an embedding vector.

Don't hesitate to ask question if my explanation isn't clear.

Hmh, I'll try to re-phrase my question because your answer did not clear up any of my confusion:

Given the output of any hugging face model (e.g. the ones with a language modeling head, take for example `GPT2LMHeadModel`), how does one actually go from the model's output to words ?


Hi, you can read the [quickstart](https://huggingface.co/transformers/quickstart.html#openai-gpt-2) of the documentation to see how to use `GPT2LMHeadModel` with the decoding method.

@George3d6 you can decode if the output of your model has the size of the vocabulary. So you need an output head that convert the hidden-size of the encoder or decoder into the vocabulary size. For `GPT2LMHeadModel`, follow what Lysandre said.

thank you all for your explanations here. I can do that with the GPT2 models with no issues, but my issue is that now I want to do the same but with the smaller simpler DistilmBERT model which is also multilingual in 104 languages, so I want to generate text in for example Spanish and English and with this lighter model. So I do this:

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')
model = DistilBertForMaskedLM.from_pretrained('distilbert-base-multilingual-cased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)
loss, prediction_scores = outputs[:2]

but now, how do I get the continuation of the phrase at that point? I tried to apply tokenizer.decode with no luck there, thank you

This issue is quite old. But for those who still looking for the answer, you should load the model with LM Head instead.
for BERT is BertForMaskedLM.from_pretrained()
for DistilBERT is DistilBertForMaskedLM.from_pretrained()

The tokenizer is the same.

> Hi, you can read the [quickstart](https://huggingface.co/transformers/quickstart.html#openai-gpt-2) of the documentation to see how to use `GPT2LMHeadModel` with the decoding method.

@LysandreJik  This reply is outdated... The link, and its redirection link send you to a 404 error.

> This issue is quite old. But for those who still looking for the answer, you should load the model with LM Head instead. for BERT is BertForMaskedLM.from_pretrained() for DistilBERT is DistilBertForMaskedLM.from_pretrained()
> 
> The tokenizer is the same.

I've tried `model_MaskedLM = AutoModelForMaskedLM.from_pretrained(checkpoint)` but it doesn't work, since the model still returns logits (or without the head, embeddings) which are not lists of token ids...
Hence, we can't simply use decode on the output of that `ForMaskedLM` model.
It seems that the checkpoint must be compatible with the model's `generate` method, which will output a list of token ids lists.



I want decode the soft prompt embedding to sequence of words. I have the output logit from BertMaskedLm head. the output logits shape is (batch_Size, num_tokens,3,vacab_size). How do I do that?