[Bug fix] Using loaded checkpoint with --do_predict (instead of random init)
Author: ethanjperez
Pull request: yes

Without this fix, I'm getting near-random validation performance for a trained model, and the validation performance differs per validation run. I think this happens since the `model` variable isn't set with the loaded checkpoint, so I'm using a randomly initialized model. Looking at the model activations, they differ each time I run evaluation (but they don't with this fix).

Comments:
Tagging @srush @nateraw from the original [Lightning GLUE PR](https://github.com/huggingface/transformers/pull/3290) to check I'm not missing something?

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3437?src=pr&el=h1) Report
> Merging [#3437](https://codecov.io/gh/huggingface/transformers/pull/3437?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ccbe839ee0b78a17e74dab218bfae7efe904ac3b&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `88.88%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3437/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3437?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3437      +/-   ##
==========================================
+ Coverage   77.56%   77.60%   +0.04%     
==========================================
  Files         100      100              
  Lines       16970    16967       -3     
==========================================
+ Hits        13162    13167       +5     
+ Misses       3808     3800       -8     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3437?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/processors/utils.py](https://codecov.io/gh/huggingface/transformers/pull/3437/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvdXRpbHMucHk=) | `24.68% <88.88%> (+2.94%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3437/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.85% <0.00%> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3437?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3437?src=pr&el=footer). Last update [83272a3...f12d585](https://codecov.io/gh/huggingface/transformers/pull/3437?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I'll check this out later tonight! I'm on mobile so I've just looked at your commit quickly...looks like you're right. I know in the past I've instantiated the model then called `model.load_from_checkpoint(loaded_ckpt)` so what you've got probably gets the same job done. The benefit of doin it the way I just mentioned is that if you already have a model object available from training, you can just load the best ckpt into that. Either way works though!  

That was fast :smile: Looks good to me!

Thanks for checking :) I'm still not able to reproduce my in-training validation performance though with the --do_predict flag, any ideas? I'm getting identical validation accuracy on different runs now, but the accuracy is still near random

@ethanjperez I just [checked the docs](https://pytorch-lightning.readthedocs.io/en/latest/weights_loading.html), and it looks like the way we were doing it originally was correct.

```python
model = MyLightingModule.load_from_checkpoint(PATH)
model.eval()
y_hat = model(x)
```

The way that I was explaining to do it would require you to use `torch.load` on the checkpoint path, which you would then pass to `model.load_state_dict`. The above method (what we had originally) is probably supposed to do that for you.

I haven't had the chance to recreate the issue, so I'll have to take a look.

Cool thanks! Even with the original way, I was still not able to reproduce my in-training validation performance (just something to look out for when you try) - In particular, I'm loading/running an already trained model  with the `--do_predict` flag without using the `--do_train` flag (I don't think you'd see the issue if you use both `--do_predict` and `--do_train`)

@nateraw @sshleifer Are you guys able to load a trained model successfully with the pytorch-lightning scripts? Even after this patch, I am having issues loading an already trained model, i.e., if I just use `--do_eval` without also using `--do_train`

Sorry for taking so long. I will try to reproduce this today if there is no update on your end!

Filing an issue with what you ran/expected would help :) @ethanjperez 

@sshleifer Just seeing this - were you able to reproduce the issue? I can't remember what exact command I ran, but it was a standard evaluation command (the same as the training command I used, but with a few flags tweaked, e.g. drop the `--do-train` flag and add the `--do-eval` flag)

This is fixed now.