named_parameters not showing embedding matrix of RobertaLMHead (more a question than a bug)
Author: ClaartjeBarkhof
Pull request: no

## Environment info
- `transformers` version: 4.2.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no 

### Who can help
@LysandreJik @patrickvonplaten 

## Information

Model I am using (Bert, XLNet ...): RobertaForCausalLM

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load standard RobertaForCausalLM
2. Record the difference between .named_parameters() and .state_dict() of the model
```
from transformers import RobertaForCausalLM, RobertaConfig
import torch

config = RobertaConfig.from_pretrained("roberta-base")
config.is_decoder = True
model = RobertaForCausalLM.from_pretrained('roberta-base', config=config)

named_params = [n for n, _ in model.named_parameters()]
print("Difference: ", [n for n in list(model.state_dict().keys()) if n not in named_params])
```

This outputs:
```
Difference:  ['roberta.embeddings.position_ids', 'lm_head.decoder.weight', 'lm_head.decoder.bias']
```

## Expected behavior

I would expect the parameters: `'lm_head.decoder.weight', 'lm_head.decoder.bias'` to show up in `.named_parameters()`, why do they not? That `'roberta.embeddings.position_ids'` does not show up in `.named_parameters()` is expected as they are not learned parameters, but just of help with getting the position embedding, but this is always the same I believe.

I would like to tie my input embedding matrix weights to my output embedding matrix. But now I am not really sure how to go about this. I thought of just doing:

`model.lm_head.decoder.weight = model.roberta.embeddings.word_embeddings.weight` but because of this thing with the named_parameters, I am not sure if this will work as expected. Also, the output embedding has a bias `lm_head.decoder.bias`, while the the input embeddings don't. My goal is to have the initial hidden state space be the same space as the last hidden state space.

Thanks in advance, 

Claartje


Comments:
I found out that the input embedding and output embedding weights are tied by default, which is why the output embedding weights end up in state_dict, but not in the named_parameters (they are overcomplete).

I've found this issue too when trying to log parameters/gradients to Weights & Biases. It doesn't log `roberta.lm_head.decoder`.

I can't quite work out the logic of [this code](https://github.com/huggingface/transformers/blob/main/src/transformers/models/roberta/modeling_roberta.py#L1143-L1145) in `RobertaLMHead`:
```py
        self.decoder = nn.Linear(config.hidden_size, config.vocab_size)
        self.bias = nn.Parameter(torch.zeros(config.vocab_size))
        self.decoder.bias = self.bias
```

If I understand what @ClaartjeBarkhof is saying, this is the reason for `decoder` not showing up in `named_parameters()`. Is that right? I can't quite make the connection between tying weights and then the thing no longer being considered a named parameter. And follow up question, what does that code actually do? How does it differ from just having the first line?