Param grad None despite model training with requires_grad=True
Author: Remorax
Pull request: no

### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-3.10.0-1160.80.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): 2.13.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to call the HubertForCTC class and fine-tune the Hubert-base model with Librispeech 100h using CTC Loss. I notice that during training, the .grad values of the: a) model parameters (ie. self.hubert.parameters) as well as b) the output layer parameters (self.lm_head.parameters) is always None (even after several backprop updates), though requires_grad is True for all of these parameters. More confusingly, the loss is also decreasing normally and the WER is improving. Could someone explain why? Unless I am missing something, the .grad value should be set after backpropagation, is it not?

FYIM I have followed the Huggingface blog on fine-tuning Wav2Vec2 and adapted it for Hubert. I provide my [train.py](https://gist.github.com/Remorax/9a68143c56f2457969a3ab6a4b360d90) and my [config file](https://gist.github.com/Remorax/2cde160f4fd87166ada46796746b9c6f) here. 

Steps to reproduce:

1. Replace lines 1234-1245 of [modelling_hubert.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/hubert/modeling_hubert.py) with this snippet (adds some print statements):

```
outputs = self.hubert(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        print (f"No. of params: {len([p for p in list(self.hubert.parameters())])}")
        print (f"No. of params with grad updated: {len([p for p in list(self.hubert.parameters()) if p.grad])}")
        print (f"No. of params with requires grad updated: {len([p for p in list(self.hubert.parameters()) if p.requires_grad])}")
        
        hidden_states = outputs[0]
        hidden_states = self.dropout(hidden_states)

        logits = self.lm_head(hidden_states)
        print (f"No. of params with grad updated in LM Head: {len([p for p in list(self.lm_head.parameters()) if p.grad])}")
        print (f"No. of params with requires grad updated in LM Head: {len([p for p in list(self.lm_head.parameters()) if p.requires_grad])}")

```
2. Download [train.py](https://gist.github.com/Remorax/9a68143c56f2457969a3ab6a4b360d90) and [config.json](https://gist.github.com/Remorax/2cde160f4fd87166ada46796746b9c6f), and call train.py as follows:

```
model_name="facebook/hubert-base-ls960"
prefix="results/hubert_debug"
config_path="<path_to_config>"
rm -rf ${DIR}/${prefix}

python3 train.py \
    --model_name $model_name --save_prefix ${prefix} \
    --num_workers 24 --language "en" \
    --trainer_config $config_path

```

The output I always get is:

No. of params: 211
No. of params with grad updated: 0
No. of params with requires grad updated: 211

No. of params with grad updated in LM Head: 0
No. of params with requires grad updated in LM Head: 2

### Expected behavior

Since I am fine-tuning all parameters, ideally, I should get:

No. of params: 211
No. of params with grad updated: 211
No. of params with requires grad updated: 211

No. of params with grad updated in LM Head: 2
No. of params with requires grad updated in LM Head: 2

Comments:
cc @sanchit-gandhi since this is an audio model.

Hey @Remorax - the print statements you've added are in the forward pass of the model, where the gradients will always be zero. The gradients are only computed in the back prop, which is triggered once the forward pass is completed. Once the back propagation is complete, the gradients are used to compute the parameter updates (using the optimiser), and the parameter updates applied to the parameters. The gradients and parameters updates are then set to zero, and we go again:
https://github.com/huggingface/transformers/blob/2fac3422389d5b4284482f036409222b3beba822/src/transformers/trainer.py#L1892C38-L1892C38

This is why the gradients are always zero for the print statements you've added (they're reset after each parameter update step). If the gradients were truly zero for every training step, then we'd never make any parameter updates, and the train loss would stay constant. The fact that your loss is decreasing normally means that the gradients are being computed and parameter updates applied to the params. Hope that explains it.

Thanks so much, that helps! Yes I suspected it was training anyway but didn't know that .grad was reset at every backprop step - thought it would be visible in the forward pass as well.

Thanks for clarifying!