pytorch-pretrained-bert and transformers give different results
Author: alicialitrtwe
Pull request: no

### System Info

```shell
- `transformers` version: 4.25.1
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.31
- Python version: 3.9.5
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

from pytorch_pretrained_bert import BertTokenizer, BertModel
import torch
torch.manual_seed(0)
tokenizer = BertTokenizer.from_pretrained('bert-large-cased', do_lower_case=False)
model = BertModel.from_pretrained('bert-large-cased')
model.eval()
model.to(device)

text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [1 for x in tokenized_text]
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
tokens_tensor = tokens_tensor.to('cuda')
segments_tensors = segments_tensors.to('cuda')
with torch.no_grad():
  outputs = model(tokens_tensor, segments_tensors)

#%%
import torch
from transformers import BertTokenizer, BertModel, BertForMaskedLM
torch.manual_seed(0)
tokenizer = BertTokenizer.from_pretrained('bert-large-cased', do_lower_case=False)
model = BertModel.from_pretrained('bert-large-cased', output_hidden_states=True)
model.eval()
model.to(device)

text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [1 for x in tokenized_text]
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
tokens_tensor = tokens_tensor.to('cuda')
segments_tensors = segments_tensors.to('cuda')
with torch.no_grad():
  outputs = model(tokens_tensor, segments_tensors)


### Expected behavior

```shell
The outputs from the 24 encoding layers should be identical for transformer and pytorch_pretrained_bert
```


### Checklist

- [X] I have read the migration guide in the readme. ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers); [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [X] I checked if a related official extension example runs on my machine.

Comments:
Hi, @alicialitrtwe 

In Huggingface the model is loaded from [here](https://huggingface.co/bert-large-cased)
which as the description says has 336M paramaters

This model has the following configuration:(taken from [here](https://huggingface.co/bert-large-cased))
  *  24-layer
  *  1024 hidden dimension
  *  16 attention heads
  *  336M parameters.

But in https://github.com/Meelfy/pytorch_pretrained_BERT, the model has 340M parameters as the description says [here](https://github.com/Meelfy/pytorch_pretrained_BERT#doc)
* bert-large-cased: 24-layer, 1024-hidden, 16-heads, 340M parameters


So, I believe you are getting different results depending on different implementations.
Actually in the `bert-large-cased` model card in huggingface there is a disclaimer suggesting this same problem, it says, "Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by the Hugging Face team." . You can read more about it [here](https://huggingface.co/bert-large-cased).

I hope it solves your question, 

Thanks,
susnato.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.