Understanding output of models and relation to token probability
Author: BramVanroy
Pull request: no

## ❓ Questions & Help

So I understand that different models were trained on different objectives. An important one is a masked language modeling objective. I would assume, then, that the model outputs probabilities for each token as the final output. Is that true?

For models that have not been trained on MLM, is it still possible to get the model's given probability for that token? (I imagine that just taking the sigmoid is not exactly the probability of the model, right?) 

Comments:
There are different kinds of models.
But as you talk about MLM, you might be talking about BERT-like models.
BERT is based on a transformer encoder so by definition of transformer, it takes a sequence of tokens (a token is just an encoding of each word into a vocabulary of known size) and returns a sequence of vectors in an embedding space: each token of the input sequence has a representation in embedding space (768 or 1024 are the common sizes called hidden-sizes in general).

Naturally, this embedding is specialized on a given task by adding one or several heads after the encoder. the MLMHead is one of those heads. NextSentencePredictionHead is another one.

If you look at BERT code in MLM Mode, it uses the following head:

```python
class BertLMPredictionHead(nn.Module):
    def __init__(self, config):
        super(BertLMPredictionHead, self).__init__()
        self.transform = BertPredictionHeadTransform(config)

        # The output weights are the same as the input embeddings, but there is
        # an output-only bias for each token.
        self.decoder = nn.Linear(config.hidden_size,
                                 config.vocab_size,
                                 bias=False)

        self.bias = nn.Parameter(torch.zeros(config.vocab_size))

    def forward(self, hidden_states):
        hidden_states = self.transform(hidden_states)
        hidden_states = self.decoder(hidden_states) + self.bias
        return hidden_states
```

You see here that the output of the model is passed through a simple `nn.Linear(config.hidden_size, config.vocab_size)` converting the embedding vector of size `hidden_size` into a vector of `vocab_size` (vocabulary size). Then you can softmax that into a vector of probability on the whole vocabulary and use argmax to get the most probable token.

So for other models, it really depends on the head used. If it is the NextSentencePrediction head, it just classifies the embedding vector into binary true/false so you lose the probabilities on the vocabulary.

Does it answer to your questions?

@mandubian Aha, that last Linear layer was what I was missing. I didn't quite understand how one could get there from simply the output of the BertModel itself (i.e. the encoder). 

I do have one more question, though. The parameters for BertLMPredictionHead are not pretrained, right? Would one still need to finetune that head on a huge dataset? (In particular I'm interested in RoBERTa, but I assume that it works similar to BERT.) More concretely, if I wanted to do inference and just get a token's probability (e.g. 'cookies' in the sentence 'I like cookies'), how could I do that?

Thanks for your time and input!

BERT is pretrained on MLM and NSP heads and provided by transformers as is. ROBERTA is pretrained on MLM only.
Check there https://huggingface.co/transformers/model_doc/bert.html#bertforpretraining, you should find what you need in outputs ;)



How did I not see this?! I'm blind. Well, perhaps I should read the documentation rather than the source code sometimes. To be honest, I didn't even know that that documentation existed!

So I see

> **prediction_scores**: torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)
> Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

So what this actually returns (if you'd apply SoftMax) is the probability for the whole vocabulary. In other words, a probability across the vocabulary which should then sum to one. So what you'd need to do, then is to find the ID of the input token in the vocabulary, and then get that ID from the output of prediction_scores. But how then can you deal with subword units? If an input token is split, how can you then recover the original token and its output?

I'm sorry for the flood of questions, but it seems like a snowball effect; every answer results in new questions.

Seems like the example here is useful but I'll have to dig deeper to understand it completely.

https://github.com/huggingface/transformers/blob/0cb163865a4c761c226b151283309eedb2b1ca4d/examples/utils_squad.py#L803-L808

No worry.
Yes wordpiece tokenizer used in Bert (and BPE in GPT2) can cut a word into several tokens. With WordPiece tokenizer, it prefixes `##` to sub-word tokens. Check that code https://github.com/huggingface/transformers/blob/master/transformers/tokenization_bert.py#L419-L421

When decoding, it's very basic, it just convert back to pieces of strings and re-concatenate them by removing ` ##` in this function for example https://github.com/huggingface/transformers/blob/master/transformers/tokenization_bert.py#L191-L194.

The rest can be found in papers, doc or code.

I understand the encoding/decoding process, but I don't quite understand how you can keep track of the subword units in the model.

Let's take an example "I like granola.", which can be encoded as "[CLS] i like gran ##ola . [SEP]" in e.g. BERT. This means we'll get seven output tokens. Here, if we want to get the output values for the input word 'granola' we need index 3 and 4. My question then is, is there a way to keep track of this dynamically/automatically? In other words, a mapping from the input words to the tokenized words, so you can go back and see that granola was split into tokens at indices position 3 and 4.

In transformers models, AFAIK, there is no tracking of token <-> indices in original text (I can be wrong). In the cases I know, it's just using `##xyz` to mean it's a sub-word token belonging to word. In decoding, final erasure of ` ##` and concatenation rebuilds words.

Yet, in examples/run_squad.py, there might be what you need as it seems to keep track of mapping between tokens and index original doc. Have a look at it, you'll see it's not trivial ;)