Seq2Seq Example with Bart not Saving Best Model
Author: ncoop57
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@sshleifer


## Information

Model I am using (Bert, XLNet ...): Bart

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

I am using a slightly modified version of the examples/seq2seq/finetune_bart_tiny.sh script, where I just add the `--val_check_interval 0.1 --do_predict` flags to the finetune.py call: 

```
python finetune.py \
--data_dir=cnn_tiny/ \
--model_name_or_path=sshleifer/bart-tiny-random \
--learning_rate=3e-5 \
--train_batch_size=2 \
--eval_batch_size=2 \
--output_dir=$OUTPUT_DIR \
--num_train_epochs=1  \
--gpus=0 \
--val_check_interval 0.1 \
--do_train --do_predict "$@"
```
Which is supposed to save the best performing model based on the val_check_interval and then evaluate the model, as is done in the regular `finetune.sh` script (thought the error is also in this one as well, I am using the tiny version so that it is easier to see the issue).

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: tiny-cnn
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Go through this google colab: https://colab.research.google.com/drive/1xtyvXI6gNAJpSkqYi_0ieWkMFRw3OSm2?usp=sharing

```
._cnn_tiny
cnn_tiny/
cnn_tiny/._train.target
cnn_tiny/train.target
cnn_tiny/._train.source
cnn_tiny/train.source
cnn_tiny/._val.source
cnn_tiny/val.source
cnn_tiny/._val.target
cnn_tiny/val.target
cnn_tiny/._test.source
cnn_tiny/test.source
cnn_tiny/._test.target
cnn_tiny/test.target
Epoch 0:  17%|█▋        | 1/6 [00:00<00:02,  2.20it/s, loss=10.839, v_num=1]
Validating: 0it [00:00, ?it/s]
Epoch 0:  33%|███▎      | 2/6 [00:00<00:01,  2.02it/s, loss=10.839, v_num=1]
Epoch 0:  50%|█████     | 3/6 [00:01<00:01,  2.07it/s, loss=10.839, v_num=1]
Epoch 0:  67%|██████▋   | 4/6 [00:01<00:00,  2.33it/s, loss=10.837, v_num=1]
Validating: 0it [00:00, ?it/s]
Epoch 0:  83%|████████▎ | 5/6 [00:02<00:00,  2.24it/s, loss=10.837, v_num=1]
Epoch 0: 100%|██████████| 6/6 [00:02<00:00,  2.28it/s, loss=10.837, v_num=1]
Epoch 0: 100%|██████████| 6/6 [00:02<00:00,  2.28it/s, loss=10.837, v_num=1]
--2020-10-10 02:28:52--  https://cdn-datasets.huggingface.co/summarization/cnn_tiny.tgz
Resolving cdn-datasets.huggingface.co (cdn-datasets.huggingface.co)... 13.227.209.120, 13.227.209.109, 13.227.209.124, ...
Connecting to cdn-datasets.huggingface.co (cdn-datasets.huggingface.co)|13.227.209.120|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23131 (23K) [application/x-tar]
Saving to: ‘cnn_tiny.tgz’

     0K .......... .......... ..                              100% 44.4M=0s

2020-10-10 02:28:52 (44.4 MB/s) - ‘cnn_tiny.tgz’ saved [23131/23131]

2020-10-10 02:28:54.290821: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
/usr/local/lib/python3.6/dist-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: The validation_epoch_end should not return anything as of 9.1.to log, use self.log(...) or self.write(...) directly in the LightningModule
  warnings.warn(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: The {log:dict keyword} was deprecated in 0.9.1 and will be removed in 1.0.0
Please use self.log(...) inside the lightningModule instead.

# log on a step or aggregate epoch metric to the logger and/or progress bar
# (inside LightningModule)
self.log('train_loss', loss, on_step=True, on_epoch=True, prog_bar=True)
  warnings.warn(*args, **kwargs)
Traceback (most recent call last):
  File "finetune.py", line 440, in <module>
    main(args)
  File "finetune.py", line 429, in main
    trainer.test()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 728, in test
    results = self.__test_using_best_weights(ckpt_path, test_dataloaders)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 740, in __test_using_best_weights
    'ckpt_path is "best", but ModelCheckpoint is not configured to save the best model.'
pytorch_lightning.utilities.exceptions.MisconfigurationException: ckpt_path is "best", but ModelCheckpoint is not configured to save the best model
```

## Expected behavior
The script should save the model with the best performing validation loss and should then use this saved model for evaluation against a test set. This is the same case for the regular `finetune.sh` script. This was working as of Oct 4/5th, but stopped sometime after.


Any help with this issue would be greatly appreciated!

Comments:
Using tiny was very smart.
We upgraded to pytorch_lightning 0.9.0 (`pip install -r examples/requirements.txt`), does that fix your issue?

It worked! But.... I had to update pyarrow from 0.14.1 to 0.17.1 because I was getting the following error:
`AttributeError: module 'pyarrow' has no attribute 'PyExtensionType'`

Which I am guessing is due to y'all epic datasets library requiring pyarrow=>0.17.1:
`ERROR: datasets 1.1.2 has requirement pyarrow>=0.17.1, but you'll have pyarrow 0.14.1 which is incompatible.`

I opened a PR to add this dependency on pyarrow 0.17.1 to the `examples/requirements.txt`: https://github.com/huggingface/transformers/pull/7750#issue-501958657
If the PR can be accepted, I'd say this issue can be fully closed.

Thanks for your help with this!