why BertModel' object has no attribute 'bias'
Author: bboyxu5928
Pull request: no

```py
from torch.nn.functional import softmax
from transformers import BertForNextSentencePrediction, BertTokenizer,BertConfig,BertModel
seq_A = 'I like cookies !'
seq_B = 'Do you like them ?'
'''
model = BertForNextSentencePrediction.from_pretrained('bert-base-cased')
'''

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
config = BertConfig.from_json_file('E:\\work\\pycharm\\transformers-master\\tf_model\\bert_config.json')

model = BertModel.from_pretrained('E:\\work\\pycharm\\transformers-master\\tf_model\\model.ckpt.index', from_tf=True, config=config)

encoded = tokenizer.encode_plus(seq_A, text_pair=seq_B, return_tensors='pt')
print(encoded)
seq_relationship_logits = model(**encoded)[0]
probs = softmax(seq_relationship_logits, dim=1)
print(seq_relationship_logits)
print(probs)  

the above demo can be used for next sentence prediction provided by BramVanroy,thank you again,
Now I want to use my own bert pre-training model or google's model  in this task ,so I find the examples in modeling_utils.py LINE 366

         # Loading from a TF checkpoint file instead of a PyTorch model (slower)
          config = BertConfig.from_json_file('./tf_model/my_tf_model_config.json')
          model = BertModel.from_pretrained('./tf_model/my_tf_checkpoint.ckpt.index', from_tf=True, config=config)
```

bert_config.json :

```
{
  "attention_probs_dropout_prob": 0.1, 
  "directionality": "bidi", 
  "hidden_act": "gelu", 
  "hidden_dropout_prob": 0.1, 
  "hidden_size": 768, 
  "initializer_range": 0.02, 
  "intermediate_size": 3072, 
  "max_position_embeddings": 512, 
  "num_attention_heads": 12, 
  "num_hidden_layers": 12, 
  "pooler_fc_size": 768, 
  "pooler_num_attention_heads": 12, 
  "pooler_num_fc_layers": 3, 
  "pooler_size_per_head": 128, 
  "pooler_type": "first_token_transform", 
  "type_vocab_size": 2, 
  "vocab_size": 21128
}
```

But when I run this demo,some errors happen,
```
AttributeError: 'BertModel' object has no attribute 'bias'

Traceback (most recent call last):
  File "E:/work/pycharm/transformers-master/src/transformers/test.py", line 18, in <module>
    model = BertModel.from_pretrained('E:\\work\\pycharm\\transformers-master\\tf_model\\model.ckpt.index', from_tf=True, config=config)
  File "E:\work\pycharm\transformers-master\src\transformers\modeling_utils.py", line 485, in from_pretrained
    model = cls.load_tf_weights(model, config, resolved_archive_file[:-6])  # Remove the '.index'
  File "E:\work\pycharm\transformers-master\src\transformers\modeling_bert.py", line 106, in load_tf_weights_in_bert
    pointer = getattr(pointer, "bias")
  File "E:\Users\Administrator\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertModel' object has no attribute 'bias'
```

It took a long time to fix it, but failed ,can you help me to solve it.thanks a lot!!



Comments:
Could you try with `BertForPreTraining` instead of `BertModel` ?

> Could you try with `BertForPreTraining` instead of `BertModel` ?

@LysandreJik  the error not occurs,thank you very much, but in the previous,when I use 

```py
model = BertForNextSentencePrediction.from_pretrained('bert-base-cased')
```
I can get seq_relationship_logits like 

```
tensor([[ 4.6285, -4.9732]], grad_fn=<AddmmBackward>)
```

after softmax,I can get probs between seq_A and seq_B,whether seq_B is a continuation of seq_A
like 
```
tensor([[9.9993e-01, 6.7607e-05]],  grad_fn=<SoftmaxBackward>)
```
but when I use BertForPreTraining ,

```py
model = BertForPreTraining.from_pretrained(.....)
```
I get seq_relationship_logits like :

```
tensor([[[ -7.3790,  -7.2666,  -7.4841,  ...,  -6.1682,  -5.8256,  -6.2910],
         [ -7.9165,  -8.1490,  -7.9572,  ...,  -6.5870,  -6.3568,  -6.8383],
         [-14.1834, -13.2084, -13.7673,  ...,  -9.0377,  -9.7575,  -9.4470],
         ...,
         [-12.9208, -12.8706, -13.0834,  ..., -10.2187,  -8.6429, -11.6360],
         [-13.2808, -13.3348, -13.2491,  ..., -10.7655,  -8.8089, -11.0420],
         [-15.5444, -15.2074, -15.8938,  ..., -11.9712, -12.5488, -14.7295]]],
       grad_fn=<AddBackward0>)
```

how can I get  probs between seq_A and seq_B ? thanks.




Hi, `BertForNextSentencePrediction` is a model that can only perform the NSP objective. `BertForPreTraining` is a model that can perform both NSP and traditional MLM. 

It, therefore, outputs tensors for both those tasks, the NSP result is the second value in the output tuple of `BertForPreTraining`:

```py
from transformers import BertForNextSentencePrediction, BertForPreTraining, BertTokenizer

nsp = BertForNextSentencePrediction.from_pretrained("bert-base-cased")
bpt = BertForPreTraining.from_pretrained("bert-base-cased")
tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

inputs = tokenizer.encode_plus("I like cats.", "I like dogs too.", return_tensors="pt")
nsp_output = nsp(**inputs)
bpt_outupt = bpt(**inputs)[1]

print(nsp_output)
print(bpt_outupt)
```

returns

```
(tensor([[ 5.1543, -5.8147]], grad_fn=<AddmmBackward>),)
tensor([[ 5.1543, -5.8147]], grad_fn=<AddmmBackward>)
```


> Hi, `BertForNextSentencePrediction` is a model that can only perform the NSP objective. `BertForPreTraining` is a model that can perform both NSP and traditional MLM.
> 
> It, therefore, outputs tensors for both those tasks, the NSP result is the second value in the output tuple of `BertForPreTraining`:
> 
> ```python
> from transformers import BertForNextSentencePrediction, BertForPreTraining, BertTokenizer
> 
> nsp = BertForNextSentencePrediction.from_pretrained("bert-base-cased")
> bpt = BertForPreTraining.from_pretrained("bert-base-cased")
> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> 
> inputs = tokenizer.encode_plus("I like cats.", "I like dogs too.", return_tensors="pt")
> nsp_output = nsp(**inputs)
> bpt_outupt = bpt(**inputs)[1]
> 
> print(nsp_output)
> print(bpt_outupt)
> ```
> 
> returns
> 
> ```
> (tensor([[ 5.1543, -5.8147]], grad_fn=<AddmmBackward>),)
> tensor([[ 5.1543, -5.8147]], grad_fn=<AddmmBackward>)
> ```

OK， I got it ,thanks a lot