trainer.predict() returns different values from model.logits
Author: connectlym
Pull request: no

Hi dear authors! 

When I was using my **fine-tuned bert model** to do the sequence classification task, I found the values returned by `trainer.predict(test_dataset)` were very different from what I got from `model(**test_encodings)`. I did not find messages describing what the `predictions` actually are in the documents, so I'm not seeing what `trainer.predict()` returns. Could you please help me explain a bit more?

Here are some of my codes
- predicts with `model(**test_encodings)`
```python
def _predict_with_np(text_a, text_b, tokenizer, model):
    scores = [0, 1]
    encoded_input = tokenizer((text_a, text_b), 
                              truncation=True, padding=True,
                              return_tensors="pt")
    output = model(**encoded_input)
    logit = output.logits[0]
    softmax_score = F.softmax(logit,dim=-1)
    score = scores[torch.argmax(softmax_score)]
    
    return score, logit, softmax_score

def predict_with_np(params, printing=False, NUM=0):

    texts = load_data()

    print("===> Loading fine-tuned model and tokenizer...")
    model = BertForSequenceClassification.from_pretrained(TUNED_MODEL)
    tokenizer = BertTokenizer.from_pretrained(TUNED_TOKENIZER)
    
    print("===> Classifying...")
    for i, text in enumerate(texts):
            if NUM > 0 and i > NUM:
                break
            text_a, text_b = text
            text_a = text_a.strip()
            text_b = text_b.strip()
            score, logit, softmax_score = _predict_with_np(text_a, text_b, tokenizer, model)
```

- predicts with `trainer.predict()`
```python
def predict_with_hf(params):
    test_dataset = load_pt_data()

    print("===> Loading Model and Training Arguments...")
    model = BertForSequenceClassification.from_pretrained(TUNED_MODEL)
    
    training_args = TrainingArguments(
        run_name=params.run_name,
        disable_tqdm=True,
        fp16=params.fp16,
        gradient_accumulation_steps=params.gradient_accumulation_steps,
        do_train=False,
        do_eval=False,
        do_predict=True,
        output_dir=params.output_dir,
    )

    print("===> Predicting...")
    trainer = Trainer(
        model=model,
        args=training_args,
        eval_dataset=test_dataset
    )
           
    results = {}
    logger.info("*** Predict ***")
    result = trainer.predict(test_dataset)
    output_pred_file = os.path.join(training_args.output_dir, "pred_results.txt")
    with open(output_pred_file, "w") as writer:
        logger.info("***** Pred results *****")
        for pred in result.predictions:
            logger.info("  predictions = %s", pred)
            writer.write("predictions = %s\n" % pred)
```

With these two versions, I got outputs like this:
```json
text_a: "Don't worry. I'll take care of it."
text_b: "Why so long?"
score: 0
logits: [0.3749077320098877, -0.15262120962142944]
softmax: [0.6289066076278687, 0.37109342217445374]
predictions: [-0.04395686  0.29134133]
```

I've read several lines of code inside `src/trainer.py` so I guess predictions are supposed to be logits. But actually, they are away different from the logits I have here. 

Am I calculating things in the wrong way, or are the predictions designed to be something else? 

Thanks for reading my long questions!

Comments:
Hi,
I have the same problem different results using `model()` vs `trainer.predict()`.

> Hi,
> I have the same problem different results using `model()` vs `trainer.predict()`.

Thanks for replying. I assumed this feature was made for other usages so I ended up using `model()`. Anyway, I would like to seek an answer for sure.

Reading the new issue's template I guess @sgugger could help us here. I'm sorry to disturb you, could you please give some details on `Trainer.predict()` here?

I solved it by returning to 4.0.1, here both methods return the same results. 

But I still got a problem, before saving the model (so just at the end of the finetuning) with `TrainingArguments(..., load_best_model_at_end=True)` the `trainer.predict()` still differs from `model()`. But after reloading the model with `from_pretrained` with transformers==4.0.1 both methods are equal. So I guess the `trainer.predict()` does really load the best model at the end of the training.

I'm unsure of what the problem is since the code you indicate is not reproducible (what is `model`, `load_pt_data()` etc.). On my side, using an installation form source on current master, here is what I get. First I instantiate a model and tokenizer and preprocess some data (with padding to be able to batch):
```
from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, Trainer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
texts = ["Hello there!", "This is another text"]
tokenized_texts = tokenizer(texts, padding=True)
```
Then I create a Dataset class to be able to feed my `tokenized_texts` to `Trainer`:
```
class SimpleDataset:
    def __init__(self, tokenized_texts):
        self.tokenized_texts = tokenized_texts
    
    def __len__(self):
        return len(self.tokenized_texts["input_ids"])
    
    def __getitem__(self, idx):
        return {k: v[idx] for k, v in self.tokenized_texts.items()}

test_dataset = SimpleDataset(tokenized_texts)
```
Then predicting through `Trainer` like this:
```
trainer = Trainer(model=model)
predictions = trainer.predict(test_dataset)
predictions.predictions
```
returns this:
```
array([[-0.68212456,  0.07081275],
       [-0.59134895,  0.16735002]], dtype=float32)
```
and predicting directly with the model:
```
import torch

model.eval()
pt_inputs = {k: torch.tensor(v).to(trainer.args.device) for k, v in tokenized_texts.items()}
with torch.no_grad():
    output = model(**pt_inputs)
output.logits.cpu().numpy()
```
gives me the exact same result.

Make sure that you preprocess your inputs the same way in both instances, and when using the model directly, that it is in evaluation mode.

> pt_inputs = {k: torch.tensor(v).to(trainer.args.device) for k, v in tokenized_texts.items()}
> with torch.no_grad():
>     output = model(**pt_inputs)
> output.logits.cpu().numpy()

Hi, thanks for your answers! I think the reason I'm having different results is I did not use `model.eval()` but I only had <1000 lines of test data to predict. 

Thank you so much! :)

> I'm unsure of what the problem is since the code you indicate is not reproducible (what is `model`, `load_pt_data()` etc.). On my side, using an installation form source on current master, here is what I get. First I instantiate a model and tokenizer and preprocess some data (with padding to be able to batch):
> 
> ```
> from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, Trainer
> 
> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
> texts = ["Hello there!", "This is another text"]
> tokenized_texts = tokenizer(texts, padding=True)
> ```
> 
> Then I create a Dataset class to be able to feed my `tokenized_texts` to `Trainer`:
> 
> ```
> class SimpleDataset:
>     def __init__(self, tokenized_texts):
>         self.tokenized_texts = tokenized_texts
>     
>     def __len__(self):
>         return len(self.tokenized_texts["input_ids"])
>     
>     def __getitem__(self, idx):
>         return {k: v[idx] for k, v in self.tokenized_texts.items()}
> 
> test_dataset = SimpleDataset(tokenized_texts)
> ```
> 
> Then predicting through `Trainer` like this:
> 
> ```
> trainer = Trainer(model=model)
> predictions = trainer.predict(test_dataset)
> predictions.predictions
> ```
> 
> returns this:
> 
> ```
> array([[-0.68212456,  0.07081275],
>        [-0.59134895,  0.16735002]], dtype=float32)
> ```
> 
> and predicting directly with the model:
> 
> ```
> import torch
> 
> model.eval()
> pt_inputs = {k: torch.tensor(v).to(trainer.args.device) for k, v in tokenized_texts.items()}
> with torch.no_grad():
>     output = model(**pt_inputs)
> output.logits.cpu().numpy()
> ```
> 
> gives me the exact same result.
> 
> Make sure that you preprocess your inputs the same way in both instances, and when using the model directly, that it is in evaluation mode.

I have a more question that how can I load the model without using "from_pretrained" 
![image](https://user-images.githubusercontent.com/36092323/114656624-9d5fa880-9d18-11eb-9eb0-c33f3d175974.png)
Because I have some custom for the the model, nn.Model, it does not inherent from "PreTrainedModel", so I can't load it using "from_pretrained"