training_loss in TFTrainer
Author: kiyoungkim1
Pull request: yes

# What does this PR do?
The purpose of ```training_loss``` in``` TFTrainer``` is logging.
However, ```training_loss``` shows a very large number while it decreases.
And it is doubled with twice the ```gradient_accumulation_steps```

1. ```training_loss``` is accumulated during epochs. Now, it is only calculated in a step.
2. Like ```Trainer```, ```training_loss``` in ```TFTrainer``` considers ```n_replicas``` and ```gradient_accumulation_steps```.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
 tensorflow: @jplu

Comments:
Here the logs of a training with
```
python run_tf_glue.py --task_name mrpc --model_name_or_path bert-base-cased --output_dir model --num_train_epochs 4 --per_device_train_batch_size 16 --per_device_eval_batch_size 16 --do_train --do_eval --do_predict --logging_steps 10 --overwrite_output_dir --gradient_accumulation_steps 2
```

```
[INFO|trainer_tf.py:522] 2021-01-15 10:27:59,116 >> ***** Running training *****
[INFO|trainer_tf.py:523] 2021-01-15 10:27:59,123 >>   Num examples = 3668
[INFO|trainer_tf.py:525] 2021-01-15 10:27:59,124 >>   Num Epochs = 4
[INFO|trainer_tf.py:526] 2021-01-15 10:27:59,124 >>   Instantaneous batch size per device = 16
[INFO|trainer_tf.py:527] 2021-01-15 10:27:59,124 >>   Total train batch size (w. parallel, distributed & accumulation) = 32
[INFO|trainer_tf.py:530] 2021-01-15 10:27:59,125 >>   Gradient Accumulation steps = 2
[INFO|trainer_tf.py:531] 2021-01-15 10:27:59,136 >>   Steps per epoch = 115
[INFO|trainer_tf.py:532] 2021-01-15 10:27:59,137 >>   Total optimization steps = 460
[INFO|trainer_tf.py:398] 2021-01-15 10:28:50,371 >> {'loss': 0.6347228, 'learning_rate': 4.891304e-05, 'epoch': 0.08695652173913043, 'step': 10}
[INFO|trainer_tf.py:398] 2021-01-15 10:28:56,791 >> {'loss': 0.604829, 'learning_rate': 4.7826084e-05, 'epoch': 0.17391304347826086, 'step': 20}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:03,192 >> {'loss': 0.62615454, 'learning_rate': 4.673913e-05, 'epoch': 0.2608695652173913, 'step': 30}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:09,614 >> {'loss': 0.61436784, 'learning_rate': 4.5652174e-05, 'epoch': 0.34782608695652173, 'step': 40}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:16,163 >> {'loss': 0.60542804, 'learning_rate': 4.456522e-05, 'epoch': 0.43478260869565216, 'step': 50}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:22,633 >> {'loss': 0.60221016, 'learning_rate': 4.347826e-05, 'epoch': 0.5217391304347826, 'step': 60}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:29,129 >> {'loss': 0.59315145, 'learning_rate': 4.2391304e-05, 'epoch': 0.6086956521739131, 'step': 70}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:35,655 >> {'loss': 0.5896678, 'learning_rate': 4.1304345e-05, 'epoch': 0.6956521739130435, 'step': 80}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:42,209 >> {'loss': 0.5796127, 'learning_rate': 4.0217386e-05, 'epoch': 0.782608695652174, 'step': 90}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:48,779 >> {'loss': 0.5678522, 'learning_rate': 3.9130435e-05, 'epoch': 0.8695652173913043, 'step': 100}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:55,365 >> {'loss': 0.55807614, 'learning_rate': 3.8043476e-05, 'epoch': 0.9565217391304348, 'step': 110}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:04,348 >> {'loss': 0.32373077, 'learning_rate': 3.695652e-05, 'epoch': 1.0434782608695652, 'step': 120}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:10,920 >> {'loss': 0.3261666, 'learning_rate': 3.5869565e-05, 'epoch': 1.1304347826086956, 'step': 130}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:17,516 >> {'loss': 0.34052417, 'learning_rate': 3.478261e-05, 'epoch': 1.2173913043478262, 'step': 140}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:24,125 >> {'loss': 0.35018474, 'learning_rate': 3.369565e-05, 'epoch': 1.3043478260869565, 'step': 150}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:30,714 >> {'loss': 0.35887596, 'learning_rate': 3.260869e-05, 'epoch': 1.391304347826087, 'step': 160}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:37,303 >> {'loss': 0.34891757, 'learning_rate': 3.1521737e-05, 'epoch': 1.4782608695652173, 'step': 170}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:43,900 >> {'loss': 0.33256933, 'learning_rate': 3.0434781e-05, 'epoch': 1.5652173913043477, 'step': 180}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:50,481 >> {'loss': 0.32668048, 'learning_rate': 2.934782e-05, 'epoch': 1.6521739130434783, 'step': 190}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:57,079 >> {'loss': 0.31888676, 'learning_rate': 2.8260865e-05, 'epoch': 1.7391304347826086, 'step': 200}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:03,688 >> {'loss': 0.31276095, 'learning_rate': 2.7173912e-05, 'epoch': 1.8260869565217392, 'step': 210}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:10,284 >> {'loss': 0.30366346, 'learning_rate': 2.6086956e-05, 'epoch': 1.9130434782608696, 'step': 220}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:16,885 >> {'loss': 0.2903903, 'learning_rate': 2.5e-05, 'epoch': 2.0, 'step': 230}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:26,095 >> {'loss': 0.15675393, 'learning_rate': 2.3913042e-05, 'epoch': 2.0869565217391304, 'step': 240}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:32,671 >> {'loss': 0.14483282, 'learning_rate': 2.2826087e-05, 'epoch': 2.1739130434782608, 'step': 250}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:39,275 >> {'loss': 0.14147088, 'learning_rate': 2.173913e-05, 'epoch': 2.260869565217391, 'step': 260}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:45,866 >> {'loss': 0.13758971, 'learning_rate': 2.0652174e-05, 'epoch': 2.3478260869565215, 'step': 270}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:52,464 >> {'loss': 0.13357341, 'learning_rate': 1.9565217e-05, 'epoch': 2.4347826086956523, 'step': 280}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:59,049 >> {'loss': 0.12877393, 'learning_rate': 1.8478258e-05, 'epoch': 2.5217391304347827, 'step': 290}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:05,682 >> {'loss': 0.13753517, 'learning_rate': 1.7391301e-05, 'epoch': 2.608695652173913, 'step': 300}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:12,281 >> {'loss': 0.1319594, 'learning_rate': 1.6304344e-05, 'epoch': 2.6956521739130435, 'step': 310}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:18,883 >> {'loss': 0.12644322, 'learning_rate': 1.5217389e-05, 'epoch': 2.782608695652174, 'step': 320}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:25,472 >> {'loss': 0.12481367, 'learning_rate': 1.41304345e-05, 'epoch': 2.869565217391304, 'step': 330}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:32,082 >> {'loss': 0.12073966, 'learning_rate': 1.3043478e-05, 'epoch': 2.9565217391304346, 'step': 340}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:41,403 >> {'loss': 0.10288413, 'learning_rate': 1.1956521e-05, 'epoch': 3.0434782608695654, 'step': 350}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:47,955 >> {'loss': 0.09858045, 'learning_rate': 1.0869565e-05, 'epoch': 3.130434782608696, 'step': 360}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:54,532 >> {'loss': 0.07963112, 'learning_rate': 9.782609e-06, 'epoch': 3.217391304347826, 'step': 370}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:01,104 >> {'loss': 0.08428383, 'learning_rate': 8.6956525e-06, 'epoch': 3.3043478260869565, 'step': 380}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:07,684 >> {'loss': 0.0844244, 'learning_rate': 7.6086967e-06, 'epoch': 3.391304347826087, 'step': 390}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:14,284 >> {'loss': 0.08690852, 'learning_rate': 6.5217405e-06, 'epoch': 3.4782608695652173, 'step': 400}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:20,877 >> {'loss': 0.0832295, 'learning_rate': 5.434781e-06, 'epoch': 3.5652173913043477, 'step': 410}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:27,494 >> {'loss': 0.078029804, 'learning_rate': 4.3478244e-06, 'epoch': 3.6521739130434785, 'step': 420}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:34,095 >> {'loss': 0.079320244, 'learning_rate': 3.2608687e-06, 'epoch': 3.7391304347826084, 'step': 430}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:40,709 >> {'loss': 0.076877564, 'learning_rate': 2.1739122e-06, 'epoch': 3.8260869565217392, 'step': 440}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:47,324 >> {'loss': 0.07551385, 'learning_rate': 1.0869561e-06, 'epoch': 3.9130434782608696, 'step': 450}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:53,941 >> {'loss': 0.07157838, 'learning_rate': 0.0, 'epoch': 4.0, 'step': 460}
```

Nothing seems wrong in the loss computation.

```
eval_acc = 0.8518518518518519
eval_f1 = 0.8954248366013072
eval_acc_and_f1 = 0.8736383442265796
```

@jplu 
Yes, you are right, and I am wrong. 
My dataset format was wrong (```labels``` in dataset for ```TFGPT2LMHead``` should be ```tensor```, but was ```dict```  yesterday). Sorry for the confusion.

However, there is one problem though. ```training_loss``` is not properly calculated with successive training. Run ```run_tf_glue.py``` with ```save_steps```. I train 40 steps, and train again from that ckpt. Results are shown below, where loss increases and decreases.

```
[INFO|trainer_tf.py:398] 2021-01-15 16:19:09,011 >> {'loss': 0.10616198, 'learning_rate': 4.456522e-05, 'epoch': 0.43478260869565216, 'step': 50}
[INFO|trainer_tf.py:398] 2021-01-15 16:19:20,271 >> {'loss': 0.17419913, 'learning_rate': 4.347826e-05, 'epoch': 0.5217391304347826, 'step': 60}
[INFO|trainer_tf.py:398] 2021-01-15 16:19:33,935 >> {'loss': 0.2174806, 'learning_rate': 4.2391304e-05, 'epoch': 0.6086956521739131, 'step': 70}
[INFO|trainer_tf.py:398] 2021-01-15 16:19:47,342 >> {'loss': 0.25698015, 'learning_rate': 4.1304345e-05, 'epoch': 0.6956521739130435, 'step': 80}
[INFO|trainer_tf.py:398] 2021-01-15 16:20:04,633 >> {'loss': 0.28348362, 'learning_rate': 4.0217386e-05, 'epoch': 0.782608695652174, 'step': 90}
[INFO|trainer_tf.py:398] 2021-01-15 16:20:22,000 >> {'loss': 0.29971126, 'learning_rate': 3.9130435e-05, 'epoch': 0.8695652173913043, 'step': 100}
[INFO|trainer_tf.py:398] 2021-01-15 16:20:35,515 >> {'loss': 0.30674613, 'learning_rate': 3.8043476e-05, 'epoch': 0.9565217391304348, 'step': 110}
[INFO|trainer_tf.py:398] 2021-01-15 16:20:57,368 >> {'loss': 0.3384542, 'learning_rate': 3.695652e-05, 'epoch': 1.0434782608695652, 'step': 120}
[INFO|trainer_tf.py:398] 2021-01-15 16:21:10,066 >> {'loss': 0.25661522, 'learning_rate': 3.5869565e-05, 'epoch': 1.1304347826086956, 'step': 130}
[INFO|trainer_tf.py:398] 2021-01-15 16:21:23,626 >> {'loss': 0.2714903, 'learning_rate': 3.478261e-05, 'epoch': 1.2173913043478262, 'step': 140}
[INFO|trainer_tf.py:398] 2021-01-15 16:21:42,122 >> {'loss': 0.262019, 'learning_rate': 3.369565e-05, 'epoch': 1.3043478260869565, 'step': 150}
[INFO|trainer_tf.py:398] 2021-01-15 16:21:55,652 >> {'loss': 0.27134755, 'learning_rate': 3.260869e-05, 'epoch': 1.391304347826087, 'step': 160}
```

This comes from  ```steps_trained_in_current_epoch``` and ```training_loss = self.train_loss.result() / (step + 1)``` in ```TFTrainer```.
For 41 step (1 step after running from cpkt-40 in this case), only one loss is accumulated but step is 40.

I simply revise it by initially saving ```steps_trained_in_current_epoch``` to another constant. 
This may be treated in different PR.

Humm looks to be an issue indeed. You can keep this PR open to fix this, feel free to ask questions if I can help :)