validation metrics not being logged by Trainer
Author: neel04
Pull request: no

### Who can help
- trainer: @sgugger
## Information

Model I am using (Bert, XLNet ...): BigBird

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Details here (https://discuss.huggingface.co/t/no-loss-being-logged-when-running-mlm-script-colab/8134)
The validation/eval loss is not being logged at all when using wandb or tensorboard - suffice to say its not being logged by Trainer.

Tried different settings for the script, none of which yield any results. 

Comments:
You are not providing a reproducer (which would include the data on which to run the script) so we can reproduce your problem. Re-tested the script on TPU and it does run evaluation every eval_steps, as provided. A few things that could be the problem in your case:
- since you set gradient_accumulation_steps = 500, your evaluation will only happen every 500 x eval_steps, so make sure you have enough training samples to get to that point (you do not provide the size of your trainingset)
- you could have an empty dataset (less than one batch), which would make the evaluation phase empty as well.

```py
!touch dataset.txt
import random
f = open('./dataset.txt', 'w')

for lines in range(50):
    f.write(' '.join(m for m in [str(random.randint(0, 40000)) for i in range(16000)]) + '\n')  #16000 words/(numbers) in one line, with random numbers from 0-40000 only.

f.close()
```
should create 50 sequences; I am using 22,500 for my training and 2,500 for validation. My batch size is `1` due to the long length of sequences, hence I don't believe that I have <1.

Attached is my validation [file](https://drive.google.com/file/d/1-6-db2cM-jpN7rpzXxWg_MpLspr6alWb/view?usp=sharing) of (uncompressed) 116MB. My training file is about 1.3GB.

Perhaps it may be a problem with my validation dataset, but I don't spot it on the surface.

Additionally, I had put a run for 5 hours ~ (`5 epochs`). In the logs from wandb, it had obviously completed in 5 hours - however, the script wouldn't stop running for some reason which in turn wouldn't trigger `wand.finish()`. From the logs, it seems that the script was running for 3 hours more after training, which seems pretty mysterious. 

I don't understand why I am getting weird behaviour. For reference, this is my cell:-
```py
%%bash
python xla_spawn.py --num_cores=8 ./run_mlm.py --output_dir="./results" \
    --model_type="big_bird" \
    --config_name="./config" \
    --tokenizer_name="./tokenizer" \
    --train_file="./dataset.txt" \
    --validation_file="./val.txt" \
    --line_by_line="True" \
    --max_seq_length="16000" \
    --weight_decay="0.01" \
    --per_device_train_batch_size="1" \
    --per_device_eval_batch_size="1" \
    --learning_rate="3e-4" \
    --tpu_num_cores='8' \
    --warmup_steps="1000" \
    --overwrite_output_dir \
    --pad_to_max_length \
    --num_train_epochs=5 \
    --adam_beta1=0.9 \
    --adam_beta2=0.98 \
    --do_train \
    --do_eval \
    #--logging_steps=200 \
    --evaluation_strategy="steps" \
    --eval_steps=250 \
    --eval_accumulation_steps=200 \
    --report_to="all" \
    --logging_dir='./logs' \
    --skip_memory_metrics='False'  \
    --gradient_accumulation_steps=500 \
    --use_fast_tokenizer='True' \
    --logging_first_step='True' \
    #1> >(tee -a ./content/drive/MyDrive/music_dataset/logs/stdout.log) \
    #2> >(tee -a ./content/drive/MyDrive/music_dataset/logs/stderr.log >&2)
```
**EDIT:-** @sgugger Another thing I found, was that despite putting the flag the strategy adjusted by the model is "no" [`evaluation_strategy=IntervalStrategy.NO`] which should have been `steps`

Mmm, if the `evaluation_strategy` is set to no, the problem is that the bash command is badly interpreted. It seems you are running it in a notebook, I don't know how that usually works, but the problem is that all the arguments you typed are not properly consumed.
You should try this in a terminal.

I didn't think of that :100: but it still doesn't work :disappointed: 

I am writing it to a bash file and running it that way; I also put it all the flags together in one single command but that doesn't seem to work either. Trying in a terminal yields same results. 

The problem is that it's not logging the loss at all after the initial one, forget the eval loss. 
```py
python3 xla_spawn.py --num_cores=8 ./run_mlm.py --output_dir="./results" --model_type="big_bird" --config_name="./config" --tokenizer_name="./tokenizer" --train_file="./dataset.txt" --validation_file="./val.txt" --line_by_line="True" --max_seq_length="16000" --weight_decay="0.01" --per_device_train_batch_size="1" --per_device_eval_batch_size="1" --learning_rate="3e-4" --tpu_num_cores='8' --warmup_steps="1000" --overwrite_output_dir --pad_to_max_length --num_train_epochs=5 --adam_beta1=0.9 --adam_beta2=0.98 --do_train --do_eval --logging_steps=200 --evaluation_strategy="steps" --eval_steps=200 --eval_accumulation_steps=200 --report_to="all" --logging_dir='./logs' --skip_memory_metrics='False' --gradient_accumulation_steps=150 --use_fast_tokenizer='True' --logging_first_step='True' 
```
A better view:-
```py

per_device_eval_batch_size=1,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=results,
push_to_hub_organization=None,
push_to_hub_token=None,
remove_unused_columns=True,
report_to=['tensorboard', 'wandb'],
resume_from_checkpoint=None,
run_name=./results,
save_on_each_node=False,
save_steps=500,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=False,
tpu_metrics_debug=False,
tpu_num_cores=8,
use_legacy_prediction_loop=False,
warmup_ratio=0.0,
warmup_steps=1000,
weight_decay=0.01,
)
```

hmm...finally got it to work; not sure what I did but removing `logging_steps`, disabling gradient accumulation and eval accumulation helps a lot - along with using `python3 ....[command]` than `python ...[cmd]` which shouldn't be an issue, but I really don't know why I have to sacrifice accuracy/features for logging to work :thinking: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.