BertForQuestionAnswering result not match when multiple run in same input
Author: kjyeon1676
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.0.dev0
- Platform: Linux-4.15.0-126-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.7.0 (false)
- Tensorflow version (GPU?): 2.5.0 (false)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik
@Rocketknight1 
 @sgugger 

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): BERT(TFBertForQuestionAnswering, BertForQuestionAnswering)

The problem arises when using:
--> my own modified script
here is my test script
```python
import numpy as np
import os
import tensorflow as tf
from transformers import BertTokenizer, TFBertForQuestionAnswering, AdamWeightDecay

model_name = "bert-base-uncased"
tokenizer = BertTokenizer.from_pretrained(model_name)

tf_model = TFBertForQuestionAnswering.from_pretrained(model_name)
question, text = "who was Jim Henson?", "Jim Henson was a puppet"
input_dict = tokenizer(question, text, return_tensors="tf")

base_output = tf_model({'input_ids':input_dict['input_ids'],
                                         'attention_mask':input_dict['attention_mask'],
                                         'token_type_ids':input_dict['token_type_ids']})
import tensorflow.keras.backend as k
tf.print(base_output.start_logits)
tf.print(base_output.end_logits)
start_logits = base_output.start_logits
end_logits = base_output.end_logits

all_tokens = tokenizer.convert_ids_to_tokens(input_dict["input_ids"].numpy()[0])
answer = ' '.join(all_tokens[tf.math.argmax(start_logits, 1)[0] : tf.math.argmax(end_logits, 1)[0]+1])
print("---------------------answer : ", answer)

# output
1 iteration : ---------------------answer :  henson was
2 iteration : ---------------------answer :
3 iteration : ---------------------answer :  [CLS] who was jim henson
```

same warning message below
```bash
All model checkpoint layers were used when initializing TFBertForQuestionAnswering.

Some layers of TFBertForQuestionAnswering were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['qa_outputs']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

The tasks I am working on is:
--> I used official example in this link : https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering

## To reproduce

Steps to reproduce the behavior:

1. copy this example https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering
2. run python script multiple times with the same input value
3. check the result answer.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

here is start_logits and end_logits value
```python
[[0.146546721 0.337863982 -0.050462883 ... 0.21409747 0.230913743 0.0886169]]
[[-0.432590961 0.0157010294 -0.264513016 ... -0.262505233 -0.097313717 0.101949602]]
---------------------answer :  who was him
[[0.239599198 -0.0761167854 -0.150168374 ... -0.329441965 -0.296196282 -0.43989116]]
[[-0.395110816 -0.316928446 -0.0174004361 ... -0.15449807 -0.0412646905 -0.340780914]]
---------------------answer :  [CLS] who was jim henson
[[0.49121806 -0.028806597 0.371522099 ... 0.544696152 0.163530082 0.184236392]]
[[0.203870535 0.0572335199 -0.129730135 ... 0.0982186 0.130047619 0.0592225939]]
---------------------answer :
[[0.284656644 -0.252363682 -0.441064388 ... 0.0992026776 0.198949382 -0.0191452727]]
[[-0.0616797283 -0.0639260635 0.413451135 ... 0.396001071 0.16053389 0.245075911]]
---------------------answer :  henson was
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
same result for start_logits and end_logits values when multiple runs the python script
and, I thought it was normal to see the following results in the correct answer.
---------------tvm answer :  jim henson ? [SEP] jim henson was a puppet
[[-0.14654148 -0.20532154 -0.293788   -0.22902387 -0.0299019  -0.09931126
  -0.02225712 -0.28276378  0.02211829 -0.19016735 -0.25408638  0.09656907
   0.00328144]]
[[-0.63135976  0.25255007  0.4773104   0.62560356  0.6185883   0.07990392
  -0.2211009   0.2174719   0.2831107   0.18743467 -0.03354458  0.08337761
  -0.20905018]]
```

## Background
- I was doing a test to run TFBertForQuestionAnswering and BertForQuestionAnswering on TVM. But, TF and Pytorch model's output does not match when inputting the same input. What did I miss? Is there any other way to perform or check?

Comments:
You are using the generic BERT checkpoint `bert-base-cased` for a question-answering task, which is why you get the warning telling you that some of the weights are randomly initialized (the weights of the question answering head). Since there is that part that is randomly initialized, you won't get the same results with two consecutive runs, or with PT vs TF.

You should use a checkpoint fine-tuned for question-answering, such as distilbert-base-uncased-distilled-squad. Complete list of available checkpoints is [here](https://huggingface.co/models?pipeline_tag=question-answering)

ok :) i got it. thank you for your explanation!