BigBird inference: same input data gives different outputs 
Author: AmitoVrito
Pull request: no
Labels: name: bug

### System Info

- pytorch==1.10.2
- transformers==4.20.1
-  python=3.9.7
-

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import (BigBirdForQuestionAnswering, BigBirdTokenizer)
import torch 
device = "cuda:0" if torch.cuda.is_available() else "cpu"
tokenizer = BigBirdTokenizer.from_pretrained("model path ")
model = BigBirdForQuestionAnswering.from_pretrained("model path")
def question_answer(model,tokenizer,text, question, handle_impossible_answer=False):  

        encoded_inputs = tokenizer(question, text, return_tensors="pt").to(device)
        start_positions = torch.tensor([1]).to((device), non_blocking=True)
        end_positions = torch.tensor([3]).to((device), non_blocking=True)
        bb_model = model.to((device), non_blocking=True)
        with torch.no_grad(): 
            outputs = bb_model(
                **encoded_inputs,
                start_positions=start_positions,
                end_positions=end_positions,
                output_attentions=True)
        print(outputs)

##note this is a sample input chunks:
chunks = ["Scikit-learn is a free software machine learning library for the Python programming language.",
            "It features various classification, regression and clustering algorithms including support-vector machines", "sklearn is a lib"]
question = "what is sklearn?"

for chunk in chunks:
    print(question_answer(chunk, question,handle_impossible_answer=False))
    print("-----------------------------------------------------")
    print(question_answer(chunk, question,handle_impossible_answer=False))
    print("------xxxxxxx-----------------------------------------------")
```

### Expected behavior

Every time for the exact same input data, the output tensors are the same for the first time. 
However, in the 2nd, 3rd runs and so on the tensors vary from the 1st run.

run1 :
```
BigBirdForQuestionAnsweringModelOutput(loss=tensor(1000012.9375), start_logits=tensor([[-1.3296e+00, -1.0000e+06, -1.0000e+06,  ..., -1.1439e+01,
         -1.0714e+01, -1.0375e+01]]), end_logits=tensor([[-3.0393e+00, -1.0000e+06, -1.0000e+06,  ..., -8.3824e+00,
         -7.1354e+00, -5.7355e+00]]), pooler_output=None, hidden_states=None, attentions=(tensor([[[[1.7520e-01, 3.3716e-03, 1.7215e-03,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [3.4175e-01, 4.7108e-02, 2.2092e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [6.5068e-02, 6.3478e-01, 1.1994e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          ...,
          [3.0409e-03, 6.9234e-05, 1.2098e-04,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [6.5139e-03, 5.8579e-05, 4.8735e-05,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [1.9445e-02, 2.0696e-04, 2.2188e-04,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]],

         [[2.0401e-02, 9.3263e-04, 1.9891e-03,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [1.7520e-02, 1.4839e-02, 1.2475e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [7.2214e-02, 3.1718e-02, 1.0765e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          ...,
          [2.3348e-03, 6.1570e-04, 8.4744e-04,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
...
          [9.8621e-02, 1.7792e-06, 4.8031e-06,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]]]])))

```
run 2:

```
BigBirdForQuestionAnsweringModelOutput(loss=tensor(1000012.9375), start_logits=tensor([[-1.3296e+00, -1.0000e+06, -1.0000e+06,  ..., -1.1439e+01,
         -1.0714e+01, -1.0375e+01]]), end_logits=tensor([[-3.0393e+00, -1.0000e+06, -1.0000e+06,  ..., -8.3824e+00,
         -7.1354e+00, -5.7355e+00]]), pooler_output=None, hidden_states=None, attentions=(tensor([[[[1.7520e-01, 3.3716e-03, 1.7215e-03,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [3.4175e-01, 4.7108e-02, 2.2092e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [6.5068e-02, 6.3478e-01, 1.1994e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          ...,
          [3.0409e-03, 6.9234e-05, 1.2098e-04,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [6.5139e-03, 5.8579e-05, 4.8735e-05,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [1.9445e-02, 2.0696e-04, 2.2188e-04,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]],

         [[2.0401e-02, 9.3263e-04, 1.9891e-03,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [1.7520e-02, 1.4839e-02, 1.2475e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [7.2214e-02, 3.1718e-02, 1.0765e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          ...,
          [2.3348e-03, 6.1570e-04, 8.4744e-04,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
...
          [1.1374e-05, 2.7021e-06, 4.4213e-07,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]]]])))

```
run 3:

```
BigBirdForQuestionAnsweringModelOutput(loss=tensor(1000012.9375), start_logits=tensor([[-1.3296e+00, -1.0000e+06, -1.0000e+06,  ..., -1.1439e+01,
         -1.0714e+01, -1.0375e+01]]), end_logits=tensor([[-3.0393e+00, -1.0000e+06, -1.0000e+06,  ..., -8.3824e+00,
         -7.1354e+00, -5.7355e+00]]), pooler_output=None, hidden_states=None, attentions=(tensor([[[[1.7520e-01, 3.3716e-03, 1.7215e-03,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [3.4175e-01, 4.7108e-02, 2.2092e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [6.5068e-02, 6.3478e-01, 1.1994e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          ...,
          [3.0409e-03, 6.9234e-05, 1.2098e-04,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [6.5139e-03, 5.8579e-05, 4.8735e-05,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [1.9445e-02, 2.0696e-04, 2.2188e-04,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]],

         [[2.0401e-02, 9.3263e-04, 1.9891e-03,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [1.7520e-02, 1.4839e-02, 1.2475e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          [7.2214e-02, 3.1718e-02, 1.0765e-02,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
          ...,
          [2.3348e-03, 6.1570e-04, 8.4744e-04,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00],
...
          [1.1374e-05, 2.7021e-06, 4.4213e-07,  ..., 0.0000e+00,
           0.0000e+00, 0.0000e+00]]]])))

```
Can you please look into this @ydshieh?


Comments:
@NautiyalAmit Sure!

@NautiyalAmit The provided code snippet has several issues that it fails to run. 

The "model path " is not a valid model name on the Hub, and I don't know which one you tried.
The call `print(question_answer(chunk, question,handle_impossible_answer=False))` has missing arguments which gives
```bash
Traceback (most recent call last):
  File "/home/yih_dar_huggingface_co/transformers/run_bigbird.py", line 31, in <module>
    print(question_answer(chunk, question,handle_impossible_answer=False))
TypeError: question_answer() missing 2 required positional arguments: 'text' and 'question'
```

Could you fix the code snippet, please 🙏 . Thank you.

Please find the code with the correct args:
```
from transformers import (BigBirdForQuestionAnswering, BigBirdTokenizer)
import torch 

device = "cuda:0" if torch.cuda.is_available() else "cpu"
tokenizer = BigBirdTokenizer.from_pretrained("model path ")
model = BigBirdForQuestionAnswering.from_pretrained("model path")
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = model.to((device), non_blocking=True)
def question_answer(text, question, handle_impossible_answer=False):
        encoded_inputs = tokenizer(question, text, return_tensors="pt").to(device)
        start_positions = torch.tensor([1]).to((device), non_blocking=True)
        end_positions = torch.tensor([3]).to((device), non_blocking=True)
        # self.model.eval()
        bbmodel = model.to((device), non_blocking=True)
        with torch.no_grad():  # reduce memory consumption
            outputs = bbmodel(
                **encoded_inputs,
                start_positions=start_positions,
                end_positions=end_positions,
                output_attentions=True)
        print(outputs)
##note this is a sample input chunks:
chunks = ["Scikit-learn is a free software machine learning library for the Python programming language.",
            "It features various classification, regression and clustering algorithms including support-vector machines", "sklearn is a lib"]
question = "what is sklearn?"


for chunk in chunks:
    print(question_answer(chunk, question,handle_impossible_answer=False))
    print("-----------------------------------------------------")
    print(question_answer(chunk, question,handle_impossible_answer=False))
    `print("------xxxxxxx-----------------------------------------------")`


Thanks @NautiyalAmit ! The following 2 lines would still fail . Could you specify the exact checkpoint name you used? Thanks.

```python
tokenizer = BigBirdTokenizer.from_pretrained("model path ")
model = BigBirdForQuestionAnswering.from_pretrained("model path")
```

Hi  @ydshieh , you can check on base  checkpoint 0 from: https://console.cloud.google.com/storage/browser/bigbird-transformer/pretrain/bigbr_base?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false

@NautiyalAmit I would like to help if the code snippet is self-contained, which means it should be able to run directly. (In some special case, I agree some manual actions might be necessary).

Here the code snippet is incomplete (missing `model path `). Even the provided GCS link contains TF checkpoint files, which could not be loaded with the `.from_pretrained` method in  `transformers` models.

As you found the issue, you must have something that could run on your side. Please try to help us debug more easily in order to investigate the issue you encountered.

I would guess you have used a model from [HuggingFace Hub](https://huggingface.co/models), with probably an official bigbird model checkpoint. But it would still be very nice if you can specify explicitly in the code snippet. Thank you.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.