BertModel output the same embedding during Evaluation
Author: nimning
Pull request: no

## ❓ Questions & Help

During evaluation, my text model outputs the same embedding regardless of the token id. The following is my model.

```
class BertTextEncoderFactory(nn.Module):
    def __init__(self, embedding_dim = 256, model_name_or_path = None, backbone ='bert'):
        super(BertTextEncoderFactory, self).__init__()

        if (backbone == 'bert'):
            self.encoder = BertForRetrival(embedding_dim, 'bert-base-uncased')
```
```
class BertForRetrival(nn.Module):
    def __init__(self, single_embedding_dim = 256, model_name_or_path = 'bert-base-uncased'):
        super(BertForRetrival, self).__init__()

        self.config = BertConfig.from_pretrained(model_name_or_path) 
        self.bert = BertModel.from_pretrained(model_name_or_path, config=self.config)

        self.single_embedding_dim = single_embedding_dim
        self.dropout = nn.Dropout(self.config.hidden_dropout_prob)

        self.embedding_layer = nn.Sequential(nn.Linear(self.config.hidden_size, self.config.hidden_size),
                                            nn.LeakyReLU(),
                                            self.dropout,
                                            nn.Linear(self.config.hidden_size, int(self.config.hidden_size / 2)),
                                            nn.LeakyReLU(),
                                            self.dropout,
                                            nn.Linear(int(self.config.hidden_size / 2), self.single_embedding_dim),
                                            nn.ReLU())

        self.init_weights(self.embedding_layer)

    def init_weights(self, module):
    	for m in module.modules():
            if type(m) == nn.Linear:
        	    torch.nn.init.xavier_uniform_(m.weight)
        	    m.bias.data.fill_(0.001)

    def forward(self, input_ids, attention_mask=None, token_type_ids=None):
        outputs = self.bert(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
        print(len(outputs))
        print(outputs[0].size())
        print(outputs)
            
        first_token_tensor = outputs[0][:,0]
        pooled_output = outputs[1]
        pooled_output = self.dropout(pooled_output)
        single_embedding = self.embedding_layer(pooled_output)

        return single_embedding 
```

Model initialization
`text_model = BertTextEncoderFactory(embedding_dim = 256, model_name_or_path = None, backbone = 'bert') `

After I finetuned the model, I save it to a checkpoint.  
`torch.save(text_model, './pretrainedcheckpoint/checkpoint.pth.tar')`

During evaluation, I  first initialize the model in the following way
`text_model = BertTextEncoderFactory(embedding_dim = 256, model_name_or_path = None, backbone = 'bert') `

The output of sample ids is the following. So far so good.
```
text_model.eval()
ids = torch.tensor([[101, 14378, 102]], dtype=torch.long)
text_model(ids, None, None)
```

Ouput is the following
```
2
torch.Size([1, 3, 768])
tensor([[[-0.6077,  0.1454, -0.1540,  ...,  0.0763,  0.5157,  0.4968],
         [ 0.7466, -0.3633, -0.0637,  ...,  0.0403,  0.5987,  0.2889],
         [ 0.9683,  0.0883, -0.3452,  ...,  0.2865, -0.6153, -0.1851]]],
       grad_fn=<NativeLayerNormBackward>)
```

Then, I load the model with trained weights. 
```
import os
resume = './pretrainedcheckpoint/checkpoint.pth.tar'
if os.path.isfile(resume):
    checkpoint = torch.load(resume)
text_model.load_state_dict(checkpoint)
```
`<All keys matched successfully>`

Now, given the same token id sequences, the model output three exactly same embedding.
```
text_model.eval()
ids = torch.tensor([[101, 14378, 102]], dtype=torch.long)
text_model(ids, None, None)
```
```
2
torch.Size([1, 3, 768])
tensor([[[-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618],
         [-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618],
         [-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618]]],
       grad_fn=<NativeLayerNormBackward>)
```

As you can see, the embedding for the three tokens are all the same !!!!
```
tensor([[[-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618],
         [-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618],
         [-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618]]],
```


Comments:
Wherees the `forward()` function of your `BertTextEncoderFactory(nn.Module)`?

It problem is caused by the data.

@nimning hi, i got stuck on the same issue exactly the same as you mentioned, cloud you please tell me how did you solve this problem