T5 batch inference same input data gives different outputs?
Author: ArvinZhuang
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
Hi, I am using a T5 model from a tf check point. I have set model.eval(), but every time I forward the exactly same input data, the outputs were always different. here is my code:
```
import torch
from transformers import *

tokenizer = T5Tokenizer.from_pretrained('t5-base')
config = T5Config.from_pretrained('t5-base')
model = T5ForConditionalGeneration.from_pretrained(
    "../model/t5-base/model.ckpt-1004000", from_tf=True, config=config)
model.eval()

def prediction(documents, query):
    querys = [query] * len(documents)
    encoded_encoder_inputs = tokenizer(documents, padding=True, truncation=True, return_tensors="pt")
    encoded_decoder_inputs = tokenizer(querys, padding=True, truncation=True, return_tensors="pt")
    with torch.no_grad():
        outputs = model(input_ids=encoded_encoder_inputs["input_ids"],
                        labels=encoded_decoder_inputs["input_ids"],
                        attention_mask=encoded_encoder_inputs["attention_mask"])
        batch_logits = outputs[1]
        print(batch_logits)


documents = ['my dog is cute!', "so am I."]
query = "who am I?"
prediction(documents, query)
```

here is my twice outputs (run the above code twice):

```
tensor([[[-14.6796,  -6.2236, -13.3517,  ..., -42.7422, -42.7124, -42.8204],
         [-18.5999,  -4.9112, -10.6610,  ..., -40.7506, -40.7313, -40.8373],
         [-17.3894,  -5.3482, -11.4917,  ..., -41.2223, -41.1643, -41.3228],
         [-18.4449,  -5.9145, -12.0056,  ..., -42.3857, -42.3579, -42.4859]],
        [[-15.7967,  -6.9833, -14.5827,  ..., -41.3168, -41.0326, -40.9567],
         [-18.4241,  -5.7193, -12.0748,  ..., -40.1744, -40.0635, -39.9045],
         [-19.5852,  -5.1691, -12.7764,  ..., -42.2655, -42.0788, -41.9885],
         [-20.3673,  -3.6864, -12.5264,  ..., -40.1189, -40.0787, -39.8976]]])
```

```
tensor([[[-14.6796,  -6.2236, -13.3517,  ..., -42.7422, -42.7124, -42.8204],
         [-18.5848,  -4.9116, -10.6607,  ..., -40.7443, -40.7251, -40.8300],
         [-17.3248,  -5.3050, -11.4988,  ..., -41.1802, -41.1236, -41.2818],
         [-18.3950,  -5.8967, -11.9756,  ..., -42.3553, -42.3273, -42.4553]],
        [[-15.7967,  -6.9833, -14.5827,  ..., -41.3168, -41.0326, -40.9567],
         [-18.5133,  -5.7466, -12.1301,  ..., -40.2199, -40.1078, -39.9497],
         [-19.3636,  -5.1246, -12.7134,  ..., -42.1881, -41.9993, -41.9106],
         [-20.3247,  -3.6335, -12.4559,  ..., -40.0285, -39.9912, -39.8106]]])
```
Am I did anything wrong?

Comments:
When I add torch.manual_seed(0) at the beginning the outputs will be the same.
does the model from tf pretrained has some randomization when loading the model weights?

Hmm, there sholud always be the same. Could you add your script to a google colab that allows to run your checkpoint, so that we can debug?

Hi here is the colab script with my checkpoint uploaded:
https://colab.research.google.com/drive/1Yx7zRkzpaGMMraTkIeIGnJJLnlpUKzQH?usp=sharing

you can download my checkpoint here: https://drive.google.com/drive/folders/1521pvzvkqvEBUvRqZn7CRO-soCXOUZCu?usp=sharing

I think some layers in your T5 model have not been trained and are therefore not saved in your model checkpoint. At initialization this layer is then randomely initialized. 

One thing you can do to verify is to load the model as follows, save it as a PyTorch model and then load the PyTorch model as follows.

```python 
config = T5Config.from_pretrained('t5-base')
model = T5ForConditionalGeneration.from_pretrained(
    "sample_data/model/model.ckpt-1004000", from_tf=True, config=config)
model.save_pretrained("./model")
model = T5ForConditionalGeneration.from_pretrained("/.model")
```

if this command throws a warning that some layers are not initialized, then you know what the problem is.
If not, I will take a look again :-) 


hi, I tried what you said, no warning shows up.
so I guess it is not the case?

Do you think it could be that weights from float64 to float32 or the opposite causing this problem?  because the difference in outputs is tiny. I don't know, this still cannot explain why outputs are different every time.

any updates?

Hey @ArvinZhuang, 

I just downloaded the weights and tried to reproduce the error using your code snippet. In my case the output is deterministic, as expected. Could you make sure that you are using the newest version of transformers and try again?

Hi, the outputs still different on my machine.... very strange.

I'm using transformer v3.3.0
torch v1.6.0
TensorFlow v2.3.1 

![image](https://user-images.githubusercontent.com/46237844/94499979-607f0900-0241-11eb-8b64-3b91d4ef1d31.png)

and btw, I cannot directly load t5 config by using T5Config.from_pretrained('t5-base') now, but the 't5-base' is still in the "https://huggingface.co/models" list. So I copy and past config.json from "https://huggingface.co/t5-base" but the results show this time is very different from the post above, which I think should not be the case because the tf checkpoint and input string are exactly the same as before....


updates: can directly load t5 config by T5Config.from_pretrained('t5-base') now, however, the output logits still very different from my first post above.....

+1 on the issue
I'm using transformer ==3.4.0
torch==1.6.0

I run:

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, AutoConfig

config = AutoConfig.from_pretrained("Vamsi/T5_Paraphrase_Paws", output_hidden_states=True)  
tokenizer = AutoTokenizer.from_pretrained("Vamsi/T5_Paraphrase_Paws")  
model = AutoModelForSeq2SeqLM.from_pretrained("Vamsi/T5_Paraphrase_Paws", config=config).to('cuda')

def prediction(documents, query):
    querys = [query] * len(documents)
    encoded_decoder_inputs = tokenizer(documents, padding=True, truncation=True, return_tensors="pt").to('cuda')
    encoded_encoder_inputs = tokenizer(querys, padding=True, truncation=True, return_tensors="pt").to('cuda')
    with torch.no_grad():
        outputs = model(input_ids=encoded_encoder_inputs["input_ids"],
                        labels=encoded_decoder_inputs["input_ids"],
                        attention_mask=encoded_encoder_inputs["attention_mask"])
        batch_logits = outputs[1]
        print(batch_logits)

documents = ['a', 'b']
query = "who am I?"
prediction(documents, query)    
```

and got:
```
tensor([[[-21.6500,  -9.8658, -13.6561,  ..., -43.1233, -43.0788, -43.0745],
         [-30.3906, -12.7200,  -1.2460,  ..., -41.7208, -41.6774, -41.6465],
         [-15.7073,  -5.9496,  -5.9364,  ..., -36.8553, -36.8221, -36.8052]],

tensor([[[-21.6500,  -9.8658, -13.6561,  ..., -43.1233, -43.0788, -43.0745],
         [-30.3906, -12.7200,  -1.2460,  ..., -41.7208, -41.6774, -41.6465],
         [-20.1459,  -5.3198,  -4.7644,  ..., -37.7978, -37.7850, -37.8202]]],
       device='cuda:0')
```
Note: rerunning ` prediction(documents, query)` produces same deterministic results, suggesting that the inputs to `labels` do affect the `logits` outcome.

Hi ednussi, 
yes, rerunning prediction(documents, query) will give deterministic results. 
However, my issue is rerunning the above chunk of code twice (reload everything, including model.). the outputs are different.
Does this also happen to you?

Update:
I tried the model provided by ednussi, and the outputs of running the code twice are the same. But my tf model still gives two different results, suggesting that loading model from tf gives different outcomes.

Hi @ArvinZhuang,
Yes, similar to you when I run the model call with same input_ids but different labels I got different outcomes (using pytorch). Realized I missed a part of the print so edited my comment above to match the output.
Hoping @patrickvonplaten or someone from the `huggingface` team can take a second look, so we can get to the bottom of this.

Hey @ednussi, if you look into the code, you can see that T5 uses the `labels` to create the `decoder_input_ids` which necessarily do affect the outputs. You can try using deterministic `input_ids` and `decoder_input_ids` and no labels and see if the output stays deterministic (it should).

Thanks @patrickvonplaten.
Was following your suggestion, and after reading through the documentation and code of how the `decoder_input_ids` is used, it became clear why it affects the `logits` and helped clear my confusion.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@ednussi - wanted to ask for your help if you could provide a brief explanation from your reading? Thanks.