Bert (sentence classification) output is non-deterministic for PyTorch (not for TF)
Author: ghost
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): German

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load model:
`config = BertConfig.from_json_file(config_filename)
model = BertForSequenceClassification(config)
state_dict = torch.load(model_filename)
model.load_state_dict(state_dict)`
2. Do inference twice on the same input + compare results.
3. Alternatively, save the first output, load the model from scratch, and run the same inference. Even in this case, the first output will not be the same as the next time.

## Expected behavior

The prediction value should be deterministic. Note that it *is* deterministic when the model parameters are loaded from a TensorFlow file (with `from_tf=True`).

## Environment info
- `transformers` version: 2.10.0
- Platform: Linux-5.3.0-55-generic-x86_64-with-Ubuntu-19.10-eoan
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.0.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
Hi, as a quick note, all the instructions you've shown could be resumed in the simpler `BertForSequenceClassification.from_pretrained`.

Which checkpoint are you trying to load? How did you obtain it?

Hi, thanks for the tip.

The checkpoint is from an own finetuned model. But would that matter? I would expect that the model behaves deterministically, even if I put random tensors with the correct shape into the `state_dict`.

Well, it depends. A few things may be responsible here: 

- Your model is not in eval mode (`model.eval()`), resulting in dropout layers affecting your results
- Your fine-tuned model is lacking some layers, which are therefore initialized randomly.

Can you check the logs by putting the following two lines above your model load?

```py
import logging

logging.basicConfig(level=logging.INFO)
```

Can you also try by using the `from_pretrained` method (given that your model filename is `pytorch_model.bin`)?

```py
config = BertConfig.from_json_file(config_filename) 

model = BertForSequenceClassification.from_pretrained(model_dir, config=config)
```

Or, simpler, if the configuration is in the same folder as your model filename:

```py
model = BertForSequenceClassification.from_pretrained(model_dir)
```

Thanks, @LysandreJik , you were exactly right: After setting `model.eval()`, the PyTorch model also behaves deterministically. Rookie mistake :smile: 

Since you provided the alternative methods, I checked them, too. The logging does not tell me whether or not the model is in eval mode. It just lists some hyperparameters of the model. At least I can see there that there seem to be at least two (`"attention_probs_dropout_prob"` and `"hidden_dropout_prob"`) that make a difference between train and eval mode.

And finally I tried the loading from one line. That also serves to resolve the issue: If you load like that it seems to be set to eval mode automatically. So not the PyTorch variant was to blame, but the mode of loading (and not explicitly setting to eval mode afterwards). Or ultimately me :wink: 

Thanks for the quick and competent response!

The logging is useful when you're loading using `from_pretrained` as it tells you which layers were not initialized with the model. For example if your checkpoint is a base BERT model that you try to load in the sequence classification model, it will load it but the classifier layer would be randomly initialized. The logging would have told you :smile:.

Glad we could resolve your problem!