migrate BertForQuestionAnswering from pytorch-pretrained-bert not produce the same result
Author: ductm104
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): retrained Bert
Language I am using the model on (English, Chinese....): multilingual - vietnamese
The tasks I am working on is:
*  an official GLUE/SQUaD task: SQUaD 
*  my own task or dataset: same format as SQUaD

## To Reproduce

Steps to reproduce the behavior:

I have trained a bert model on older pytorch-pretrained-bert and it works just fine. recently, I switch the code to the latest version of transformer. I use the following config:

>     bert_model = './my_model'
>     max_seq_length = 160
>     doc_stride = 160
>     predict_batch_size = 20
>     n_best_size=20
>     max_answer_length=30
>     verbose_logging = False
>     no_cuda = True
>     seed= 42
>     do_lower_case= True
>     version_2_with_negative = True
>     null_score_diff_threshold=0.0
>     max_query_length = 64
>     THRESH_HOLD = 0.95

I import 2 class:
`from transformers import BertForQuestionAnswering as bqa1`
`from pytorch_pretrained_bert.modeling import BertForQuestionAnswering as bqa2`
and load 2 model as following :
`model1 = bqa1.from_pretrained(args.bert_model)`
`model2 = bqa2.from_pretrained(args.bert_model)`

and input to models with the same tensors:
`input_ids = torch.ones((1,160),dtype = torch.int64)`
`segment_ids = torch.ones((1,160),dtype = torch.int64)`
`input_mask = torch.ones((1,160),dtype = torch.int64) `
`model(input_ids, segment_ids, input_mask)`

I also check if 2 model has same weights or not by using following guide [https://discuss.pytorch.org/t/check-if-models-have-same-weights/4351/3](guide).

I seed the randomness of torch before inference 2 model by using:
`seed = 0`
`torch.manual_seed(seed)`
`if torch.cuda.is_available():`
`    torch.cuda.manual_seed_all(seed)`
but 2 models still produce difference results.

Comments:
Hello! Have you put these models in `eval()` mode so as to deactivate the dropout modules?

For completeness sake: did you train both models with the same random seed? Or are you just trying to evaluate models that you trained?

My go-to method is:

```python
def set_seed(seed):
    """ Set all seeds to make results reproducible (deterministic mode).
        When seed is a false-y value or not supplied, disables deterministic mode. """

    if seed:
        logging.info(f"Running in deterministic mode with seed {seed}")
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
        np.random.seed(seed)
        random.seed(seed)
        os.environ['PYTHONHASHSEED'] = str(seed)
    else:
        logging.info(f"Running in non-deterministic mode")
```

It does work when I put generated tensors into 2 models but doesn't when I put tensors I save before. Maybe I will rewrite the inference code with new code base and retrain the model.

Are you sure your model is in evaluation mode? 

yes, I already put it in evaluation mode

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.