Pre-trained model returns different outputs(random outputs)
Author: houdaM97
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello,
I had recently play around Huggingface library, i wrote a simple script for question answering task. and for that,  i used TFXLNetForQuestionAnsweringSimple model (pre-trained model), but i had different outputs for the same inputs and model each time i run the program.
Did i miss something?
here is my script:

context = "Jim Henson was a puppeteer"
question = "Who was Jim Henson ?"

#XLNET
tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
model = TFXLNetForQuestionAnsweringSimple.from_pretrained("xlnet-base-cased")
en_plus = tokenizer.encode_plus(context, question, add_special_tokens=True)
en = en_plus['input_ids']
token_type_ids = en_plus['token_type_ids']
input_ids = tf.constant([en])

segments_tensors = tf.constant([token_type_ids])
outputs = model(input_ids)
start_scores, end_scores = outputs[:2]

ss = tf.argmax(start_scores.numpy()[0]).numpy()
es = tf.argmax(end_scores.numpy()[0]).numpy()

answer = tokenizer.decode(en[ss: es+1], clean_up_tokenization_spaces=True)

print(answer)

Thank you in advance for your help.




Comments:
I found too. "last_hidden_states" was not fixed when I reload pretrain model. I think we miss something. My question is here <https://github.com/huggingface/transformers/issues/2386>, maybe help you.

Hi @houdaM97, this is due to the fact that the pretrained archive `xlnet-base-cased` does not contain keys for the question answering head, only for the base transformer model. This means that the question answering head will be randomly initialized and will output different results at each run.

In order to see which keys are missing, you can set the flag `output_loading_info` to `True` in the `from_pretrained` method:

```py
model, loading_info = TFXLNetForQuestionAnsweringSimple.from_pretrained("xlnet-base-cased", output_loading_info=True)
print("Loading info", loading_info)

#  Loading info {'missing_keys': ['qa_outputs'], 'unexpected_keys': ['lm_loss'], 'error_msgs': []}
```

Here you can see that the `qa_outputs` value is missing and that the `lm_loss` value was present in the checkpoint but not needed for that specific model. In order to use this model for question answering you would first need to fine-tune this `qa_outputs` layers to a question answering task like SQuAD (you can use the [run_squad.py](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py) script for this).

We have a few models which are already fine-tuned on SQuAD, the list is available [here](https://huggingface.co/transformers/pretrained_models.html) (look for squad). You can also use some community fine-tuned models, which are visible [here](https://huggingface.co/models).

Hi @LysandreJik , does the tensorflow version of run_squad.py exist?

Hi @houdaM97, not yet but I'm actively working on it, alongside other projects. I'm aiming at next week for the first working version.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.