`hidden_dropout` is not used in SEW-D
Author: gau-nernst
Pull request: no

### System Info

NA

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Doing a quick Ctrl+F in https://github.com/huggingface/transformers/blob/main/src/transformers/models/sew_d/modeling_sew_d.py reveals that `config.hidden_dropout` is never used. Instead, `config.activation_dropout` is used in places where `config.hidden_dropout` is used in Wav2Vec2. Furthermore, `config.activation_dropout` is not even documented (https://github.com/huggingface/transformers/blob/main/src/transformers/models/sew_d/configuration_sew_d.py, Wav2Vec2 config also does not document `config.activation_dropout`).

To be more specific, for Wav2Vec2, `config.activation_dropout` is only used in the middle of the MLP block

https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L655

`config.hidden_dropout` is used at the end of MLP block, end of attention block, after adding positional embeddings

https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L664
https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L685
https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L761

Meanwhile, for SEW-D, there is no dropout in the middle of MLP block, and `config.activation_dropout` is used for end of MLP block, end of attention block, and positional embeddings in DeBERTa attention

https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/sew_d/modeling_sew_d.py#L676
https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/sew_d/modeling_sew_d.py#L725
https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/sew_d/modeling_sew_d.py#L966

I guess the reason for this is due to translating from the original implementation, where DeBERTa's `hidden_dropout_prob` is fixed to `activation_dropout`

https://github.com/asappresearch/sew/blob/5c66f391bf137b12d4860c4cde24da114950b0fe/sew_asapp/models/squeeze_wav2vec2_deberta.py#L98

Nevertheless, this creates discrepancy between similar models i.e. `hidden_dropout`/`attention_dropout` means different things for Wav2Vec2 and SEW-D (and documentation on the configuration is not accurate). More importantly, when using SEW-D, if we increase `hidden_dropout`, nothing changes. Just by reading the documentation alone, which does not mention `activation_dropout`, there is no way to increase dropout for SEW-D layers.

### Expected behavior

I propose replacing `config.activation_dropout` with `config.hidden_dropout` in the SEW-D code. We would need to update various SEW-D configs also, since from what I see, the configs use `activation_dropout=0` and `hidden_dropout=0.1`.

Comments:
Some related question: the check `utils/check_config_attributes.py` doesn't detect this attribute `hidden_dropout` is not used. I need to check why this happens. Assign myself for this part

On a slightly related issue, `ConvLayer` copied from DeBERTa is never used, because SEW-D does not use it.

https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/sew_d/modeling_sew_d.py#L1011-L1023

And `config.hidden_dropout_prob` (L1022) does not exist for SEW-D config.

Thanks for the extra piece of information, @gau-nernst 🤗 

I see there is a line
```
self.conv = ConvLayer(config) if getattr(config, "conv_kernel_size", 0) > 0 else None
```
Despite in the configuration definition file, there is no `conv_kernel_size`, we are not 100% sure if there is any config on the Hub using this config/model with a manually added `conv_kernel_size`.

It's not clear why it is done this way rather than having explicitly `conv_kernel_size` in `SEWDConfig`.

Also, `hidden_dropout` is used in `src/transformers/models/sew_d/convert_sew_d_original_pytorch_checkpoint_to_pytorch.py` although I doubt there is no real usage.

On my side, I am open to remove `hidden_dropout` from the config definition as long as @sanchit-gandhi also agrees, but just to avoid the confusion in the future.

Regarding `conv_kernel_size`, it's probably because it's being copied from DeBERTa. For reference, the original implementation hard-coded it to 0.

https://github.com/asappresearch/sew/blob/5c66f391bf137b12d4860c4cde24da114950b0fe/sew_asapp/models/squeeze_wav2vec2_deberta.py#L101-L102

At the current state, `ConvLayer` does not work anyway since it uses a DeBERTa's config key `hidden_dropout_prob`, which does not exist for SEW-D (need to replace with something else like `activation_dropout`).

https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/sew_d/modeling_sew_d.py#L1011-L1022

I think this is not a big issue since it's unlikely that someone will use this part of the code. But I think removing `ConvLayer` and

```python
self.conv = ConvLayer(config) if getattr(config, "conv_kernel_size", 0) > 0 else None
```

can make the code slightly cleaner.

About `hidden_dropout` and `activation_dropout`, personally I feel removing `activation_dropout` is better, and replace it with `hidden_dropout`, so that the behavior is closer to Wav2Vec2 and SEW. It's just my personal opinion.

Regardless of what config definition you will settle on, documentation for the config definitely needs an update.

> About `hidden_dropout` and `activation_dropout`, personally I feel removing `activation_dropout` is better, and replace it with `hidden_dropout`, so that the behavior is closer to Wav2Vec2 and SEW. It's just my personal opinion.

This sounds like a breaking change (IIRC)?

Hey @gau-nernst! Thanks for opening this issue and highlighting the inconsistency between these models!

Unfortunately, I don't think we can blindly swap `activation_dropout` for `hidden_dropout` in SEW. This would cause a breaking change, which is something we try to avoid as much as possible in the `transformers` library.

Instead, I would be inclined to better document what each of these dropout terms refers to. IMO, if there's more transparency of what these terms are, there'll be better understanding around them. Would you like to open a PR to update the configuration files of SEW and Wav2Vec2 to better document the dropout terms? This would be a most welcome update!

Happy to remove `hidden_dropout`, `ConvLayer` and `conv_kernel_size` if they're entirely un-used.