TF loss function output inconsistent with Pytorch one for multiple tasks
Author: janjitse
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-5.10.7-gentoo-x86_64-AMD_Ryzen_9_3950X_16-Core_Processor-with-glibc2.2.5
- Python version: 3.8.7
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.4.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.


 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@jplu, 
## Information


Model I am using (Bert, XLNet ...): TFGPT2LMHeadModel

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

I was converting the example of perplexity calculation of fixed-length models [perplexity calculation of fixed-length models](https://huggingface.co/transformers/perplexity.html) to Tensorflow, and ran into an inconsistency in the implementation of compute_loss, compared to the implementation in the Pytorch version of the model.

For Tensorflow, when calling a model with inputs and labels (model(input_ids = input_ids, labels = labels), there is no reduction being done on the output of SparseCategoricalCrossentropy loss function (i.e. it is called explicitly with  reduction=tf.keras.losses.Reduction.NONE for all tasks), as defined in modeling_tf_utils.py, while for Pytorch, the loss function CrossEntropyLoss() is called with the standard reduction (just the mean), which seems a bit unexpected to me.

After modifying the code to do an explicit tf.math.reduce_mean on the outcome of the model, I was able to reproduce the Pytorch outcome exactly.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Tensorflow version:
    `outputs = model(input_ids, labels = target_ids)`
    `log_likelihood = tf.math.reduce_mean(outputs[0] * trg_len)`
 Pytorch version:
    `outputs = model(input_ids, labels=target_ids)`
    `log_likelihood = outputs[0] * trg_len`


## Expected behavior

Outcome of TFGPT2LMHeadModel.call(input_ids=input_ids,labels=labels) to have same tensor shapes as outcome of GPT2LMHeadModel.call(input_ids=input_ids,labels=labels)


Comments:
Hello!

This is the expected behavior, if you want any reduction on the loss, you have to do it yourself on your side, not inside the respective compute_loss function.

Hi, thanks for the explanation. 
I realized it was probably by design, it's just odd that it differs so much in behavior from the Pytorch version. Is there any plan to bring those more inline in this regards? Probably a breaking change, I don't have a clear overview of how much would  break, even internally within the Transformers library.

Nothing planed to align this with Python and we won't. The reason is because when training with a distribute strategy, TensorFlow doesn't allow a reduction other than `None` or `Sum`. Knowing that we have our own custom trainer and we cannot apply the change you would like as it will make it fails for such cases.

Makes sense, didn't think about the incompatibility of the AUTO reduction with the distribute strategies and the custom trainer.
 
I'll try to make a small patch over the weekend with an update to the documentation in the docstrings, as it's currently not in line with the actual (and intended) output.

That would be awesome! Thanks!