TFWav2Vec2ForCTC & Wav2Vec2ForCTC gives different loss values
Author: thevasudevgupta
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.0.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.10
- PyTorch version (GPU?): 1.8.1 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@will-rice @patrickvonplaten 

## Information

Model I am using: `TFWav2Vec2ForCTC` & `Wav2Vec2ForCTC`

## To reproduce

Steps to reproduce the behavior:
```python
import tensorflow as tf
import torch
from transformers import Wav2Vec2ForCTC, TFWav2Vec2ForCTC

model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")
tf_model = TFWav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")

tf_labels = tf.constant([[3, 54, 65, 76, 21], [32, 42, 434, 76, 231]])
labels = torch.from_numpy(tf_labels.numpy())

tf_speech = tf.random.uniform(shape=(2, 40000))
speech = torch.from_numpy(tf_speech.numpy()).float()

with torch.no_grad():
    out = model(speech, labels=labels)
tf_out = tf_model(tf_speech, labels=tf_labels)

print(out["loss"], tf_out["loss"])
# -> 71.64           -> 16.92
```

## Expected behavior

Loss values from tensorflow & PyTorch model should be similar (Note: logits are perfectly same as expected).

Comments:
try it with labels that are less than the vocab size. I was able to change it to this and they are pretty close.

```
import tensorflow as tf
import torch
from transformers import Wav2Vec2ForCTC, TFWav2Vec2ForCTC

model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")
tf_model = TFWav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")

tf_labels = tf.constant([[12, 10, 1, 2, 3], [4, 5, 6, 7, 8]])
labels = torch.from_numpy(tf_labels.numpy())

tf_speech = tf.random.uniform(shape=(2, 40000))
masks = tf.ones_like(tf_speech)

speech = torch.from_numpy(tf_speech.numpy()).float()

with torch.no_grad():
    out = model(speech, labels=labels)
tf_out = tf_model(tf_speech, labels=tf_labels)

print(out["loss"].numpy(), tf_out["loss"].numpy())
```

88.34665 88.34598

Thanks for the quick reply. It will help. 

It should actually throw an error if labels are > vocab_size! Will open an issue for this

@will-rice @patrickvonplaten 

PyTorch & TensorFlow losses are becoming different if padding indices are set to -100. Checkout this small [Colab notebook](https://colab.research.google.com/drive/190NDNtAKg4y2a-jjMby-XNZ2EYScOm6m?usp=sharing).

This is happening because these [2 lines](https://github.com/huggingface/transformers/blob/2e5dbdf2db4599a6694d0974575a70f9bc3c978e/src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py#L1584) [1584 & 1585] should not be there in TensorFlow implementation. If we just remove them, PyTorch & TensorFlow loss will become same.

So:

```python
# we should remove these lines
flattened_labels = tf.boolean_mask(labels, labels_mask)            
flattened_labels = tf.reshape(flattened_labels, [labels.shape[0], -1])

# rather replace it with
flattened_labels = labels
```


There is one other bug also in TensorFlow implementation. `training` argument should be passed in this [line](https://github.com/huggingface/transformers/blob/2e5dbdf2db4599a6694d0974575a70f9bc3c978e/src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py#L1225) because right now spec augmentation is not getting applied even when `config.apply_spec_augment = True`.

Now, if we pass training arg in above linked line, spec augmentation does't work & rather throws an error. This needs to be fixed as well, I think.

Good catches! I'm working on a fix for these. Were you still wanting to open a PR for the label error or would you like me to just roll that one into these?

I am fine if you are going to fix the label error messages.

Closing this issue as it is fixed in #12289