is the prediction_logits.size() is correct?
Author: siam1991
Pull request: no

hi, 
I'm using the BertForPreTraining.from_pretrained() to load the local ckpt file, and I want to get a sentence new embedding, the expect size of output is [1,9,786],but the output is [1,9,30522] and the 30522 is the size of vocab.txt. I also use the "bert-base-uncased", the result is same ,here is my code ,how could i get the expect output.

import torch
from transformers import BertTokenizer,BertForPreTraining, BertConfig
model_path = 'D:/Project/bert/pre_trained/uncased_L-12_H-768_A-12'
tokenizer = BertTokenizer.from_pretrained("{}/vocab.txt".format(model_path))
config = BertConfig.from_pretrained("{}/bert_config.json".format(model_path))
model = BertForPreTraining.from_pretrained("{}/bert_model.ckpt.index".format(model_path), from_tf=True,config=config)
input_text = "Here is some text to encode"
input_ids = tokenizer.encode(input_text, add_special_tokens=True)
print(input_ids) -> input_ids: [101, 2182, 2003, 2070, 3793, 2000, 4372, 16044, 102]
input_ids = torch.tensor([input_ids])
last_hidden_states = model(input_ids).prediction_logits
print(last_hidden_states.size())->[1,9,30522]


import torch
from transformers import BertModel, BertTokenizer,BertForPreTraining, BertConfig
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForPreTraining.from_pretrained("bert-base-uncased")
input_text = "Here is some text to encode"
input_ids = tokenizer.encode(input_text, add_special_tokens=True)
print(input_ids) -> input_ids: [101, 2182, 2003, 2070, 3793, 2000, 4372, 16044, 102]
input_ids = torch.tensor([input_ids])
last_hidden_states = model(input_ids).prediction_logits
print(last_hidden_states.size())->[1,9,30522]


Comments:
To get a sentence embedding, you don't need to use `BertForPreTraining`, as this model adds the heads for pre-training (which are next sentence prediction and masked language modeling) on top. Hence, the `prediction_logits` come from the language modeling head.

You can just use `BertModel`, which is the BERT model without any head on top. To get an embedding vector, you can for example take the final hidden state of the special [CLS] token that is added at the beginning of the sentence, as follows:

```
from transformers import BertTokenizer, BertModel

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")

input_text = "Here is some text to encode"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids

# forward pass
outputs = model(input_ids)

# get sentence vector (by taking the last hidden state of the [CLS] token)
last_hidden_states = outputs.last_hidden_state # shape (batch_size, seq_len, hidden_size)
sentence_vector = last_hidden_states[:,0,:] # shape (batch_size, hidden_size)
```



thanks for your answer. 
At first, I was used the `BertModel`, but when I use the `ckpt` file, I got this error :
> AttributeError: 'BertModel' object has no attribute 'bias' 

so I searched the related issue, and change to use `BertForPreTraining`. 
Now I understand the difference between `BertModel` and `BertForPtrTraining`. I would use `BertModel` to get a sentence embedding.