BERT Embedding Weights VS Last Hidden State
Author: ghost
Pull request: no

### BERT for Feature Extraction: Embedding Weights VS Last Hidden State
I am trying to extract the pretrained BERT token embeddings and get the feature vector for any specific token by indexing the token ID. However, I found that indexing the pretrained embedding matrix returns very different values as compared to feeding the token IDs into the encoder to get the embeddings. Am I missing something? I can't find this anywhere in the official documentation/tutorials and I've seen the use of both methods to extract features, which is quite concerning if implemented without proper understanding. 
```
from transformers import BertTokenizer, BertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained("bert-base-uncased")

# get pretrained token embeddings
embedding_matrix = model.embeddings.word_embeddings.weight
# look up embeddings token sequence
embeddings = embedding_matrix[input['input_ids']]

# feed input into pretrained encoder
input = tokenizer(text, return_tensors='pt')
output = model(**input)

# why are the token embeddings different before fine-tuning? 
torch.all(embeddings == output.last_hidden_state)
```

**Examples:** 

_**text = "hello"**_
embeddings
```
tensor([[[ 0.0136, -0.0265, -0.0235,  ...,  0.0087,  0.0071,  0.0151],
         [-0.0043, -0.0330, -0.0217,  ..., -0.0425, -0.0127, -0.0389],
         [-0.0145, -0.0100,  0.0060,  ..., -0.0250,  0.0046, -0.0015]]],
       grad_fn=<IndexBackward0>)
```
output.last_hidden_state
```
tensor([[[-0.3061,  0.2622, -0.1896,  ..., -0.1651,  0.1014,  0.4119],
         [-0.7390, -0.0336,  0.3932,  ..., -0.1818, -0.1839, -0.2185],
         [ 0.5801,  0.0627, -0.2637,  ...,  0.3963, -0.5684, -0.4924]]],
       grad_fn=<NativeLayerNormBackward0>)
```
_**text = "hello!"**_
embeddings
```
tensor([[[ 0.0136, -0.0265, -0.0235,  ...,  0.0087,  0.0071,  0.0151],
         [-0.0043, -0.0330, -0.0217,  ..., -0.0425, -0.0127, -0.0389],
         [ 0.0298, -0.0373, -0.0356,  ...,  0.0161,  0.0192,  0.0173],
         [-0.0145, -0.0100,  0.0060,  ..., -0.0250,  0.0046, -0.0015]]],
       grad_fn=<IndexBackward0>)
```
output.last_hidden_state
```
tensor([[[-0.0509,  0.1088, -0.1411,  ..., -0.1243, -0.0803,  0.2858],
         [-0.6771, -0.5464,  0.0878,  ..., -0.0575,  0.0359, -0.3080],
         [-1.0903, -0.9996, -0.5636,  ...,  0.3232, -0.2773, -0.1463],
         [ 0.8302,  0.0501, -0.2251,  ...,  0.3216, -0.6489, -0.2456]]],
       grad_fn=<NativeLayerNormBackward0>)
```

Comments:
After much more digging, I found a superb in-depth explanation by Alexey Kravets in this article: 
https://towardsdatascience.com/deep-dive-into-the-code-of-bert-model-9f618472353e

Apparently, the last hidden state returns the context-aware representations of the word embeddings, which are calculated from the pretrained model weights via normalization and the attention mechanism (using pretrained weights and biases for the Q, K and V matrices). 

Essentially: feeding the pretrained feature vectors into the pretrained encoder model. For sequence-based tasks, this method is definitely more appropriate as compared to using the unprocessed context-invariant embeddings. 

@leejiayi098 Hi, I've also came into this problem. So if I want to get all word embeddings in my vocab with the last hidden state, is there any easy way to do? Otherwise, I have to construct the input text and using model to do inference.

so this suggests that we should use the `output.last_hidden_state` as embeddings for practical applications? as those embeddings are context-aware rather than the other one right?

i also found a huggingface official blog ([here](https://huggingface.co/blog/getting-started-with-embeddings)) on embeddings which states about using a prebuilt endpoint with the code: 

```python3
import requests

api_url = f"https://api-inference.huggingface.co/pipeline/feature-extraction/{model_id}"
headers = {"Authorization": f"Bearer {hf_token}"}
```

so considering my first statement is correct regarding `last_hidden_state`, which one does this endpoint actually return? `last_hidden_state` or the other one?

any help is truly appreciated. thanks.

cc @leejiayi098 @Jackie-shi @vanpelt @tmm1