Why is the lm_head layer in GPT2LMHeadModel not a parameter?
Author: jimmyjimmy94
Pull request: no

I loaded the model by 
```
from transformers import GPT2LMHeadModel
gpt2 = GPT2LMHeadModel.from_pretrained('distilgpt2')
```
doing `[n for n,p in gpt2.named_parameters()]` gives me:
```
['gpt2.transformer.wte.weight', 'gpt2.transformer.wpe.weight', 'gpt2.transformer.h.0.ln_1.weight', 'gpt2.transformer.h.0.ln_1.bias', 'gpt2.transformer.h.0.attn.c_attn.weight', 'gpt2.transformer.h.0.attn.c_attn.bias', 'gpt2.transformer.h.0.attn.c_proj.weight', 'gpt2.transformer.h.0.attn.c_proj.bias', 'gpt2.transformer.h.0.ln_2.weight', 'gpt2.transformer.h.0.ln_2.bias', 'gpt2.transformer.h.0.mlp.c_fc.weight', 'gpt2.transformer.h.0.mlp.c_fc.bias', 'gpt2.transformer.h.0.mlp.c_proj.weight', 'gpt2.transformer.h.0.mlp.c_proj.bias', 'gpt2.transformer.h.1.ln_1.weight', 'gpt2.transformer.h.1.ln_1.bias', 'gpt2.transformer.h.1.attn.c_attn.weight', 'gpt2.transformer.h.1.attn.c_attn.bias', 'gpt2.transformer.h.1.attn.c_proj.weight', 'gpt2.transformer.h.1.attn.c_proj.bias', 'gpt2.transformer.h.1.ln_2.weight', 'gpt2.transformer.h.1.ln_2.bias', 'gpt2.transformer.h.1.mlp.c_fc.weight', 'gpt2.transformer.h.1.mlp.c_fc.bias', 'gpt2.transformer.h.1.mlp.c_proj.weight', 'gpt2.transformer.h.1.mlp.c_proj.bias', 'gpt2.transformer.h.2.ln_1.weight', 'gpt2.transformer.h.2.ln_1.bias', 'gpt2.transformer.h.2.attn.c_attn.weight', 'gpt2.transformer.h.2.attn.c_attn.bias', 'gpt2.transformer.h.2.attn.c_proj.weight', 'gpt2.transformer.h.2.attn.c_proj.bias', 'gpt2.transformer.h.2.ln_2.weight', 'gpt2.transformer.h.2.ln_2.bias', 'gpt2.transformer.h.2.mlp.c_fc.weight', 'gpt2.transformer.h.2.mlp.c_fc.bias', 'gpt2.transformer.h.2.mlp.c_proj.weight', 'gpt2.transformer.h.2.mlp.c_proj.bias', 'gpt2.transformer.h.3.ln_1.weight', 'gpt2.transformer.h.3.ln_1.bias', 'gpt2.transformer.h.3.attn.c_attn.weight', 'gpt2.transformer.h.3.attn.c_attn.bias', 'gpt2.transformer.h.3.attn.c_proj.weight', 'gpt2.transformer.h.3.attn.c_proj.bias', 'gpt2.transformer.h.3.ln_2.weight', 'gpt2.transformer.h.3.ln_2.bias', 'gpt2.transformer.h.3.mlp.c_fc.weight', 'gpt2.transformer.h.3.mlp.c_fc.bias', 'gpt2.transformer.h.3.mlp.c_proj.weight', 'gpt2.transformer.h.3.mlp.c_proj.bias', 'gpt2.transformer.h.4.ln_1.weight', 'gpt2.transformer.h.4.ln_1.bias', 'gpt2.transformer.h.4.attn.c_attn.weight', 'gpt2.transformer.h.4.attn.c_attn.bias', 'gpt2.transformer.h.4.attn.c_proj.weight', 'gpt2.transformer.h.4.attn.c_proj.bias', 'gpt2.transformer.h.4.ln_2.weight', 'gpt2.transformer.h.4.ln_2.bias', 'gpt2.transformer.h.4.mlp.c_fc.weight', 'gpt2.transformer.h.4.mlp.c_fc.bias', 'gpt2.transformer.h.4.mlp.c_proj.weight', 'gpt2.transformer.h.4.mlp.c_proj.bias', 'gpt2.transformer.h.5.ln_1.weight', 'gpt2.transformer.h.5.ln_1.bias', 'gpt2.transformer.h.5.attn.c_attn.weight', 'gpt2.transformer.h.5.attn.c_attn.bias', 'gpt2.transformer.h.5.attn.c_proj.weight', 'gpt2.transformer.h.5.attn.c_proj.bias', 'gpt2.transformer.h.5.ln_2.weight', 'gpt2.transformer.h.5.ln_2.bias', 'gpt2.transformer.h.5.mlp.c_fc.weight', 'gpt2.transformer.h.5.mlp.c_fc.bias', 'gpt2.transformer.h.5.mlp.c_proj.weight', 'gpt2.transformer.h.5.mlp.c_proj.bias', 'gpt2.transformer.ln_f.weight', 'gpt2.transformer.ln_f.bias']
```
while running `print (gpt2)` gives me:
```
GPT2LMHeadModel(
  (transformer): GPT2Model(
    (wte): Embedding(50257, 768)
    (wpe): Embedding(1024, 768)
    (drop): Dropout(p=0.1, inplace=False)
    (h): ModuleList(
      (0): Block(
        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): Attention(
          (c_attn): Conv1D()
          (c_proj): Conv1D()
          (attn_dropout): Dropout(p=0.1, inplace=False)
          (resid_dropout): Dropout(p=0.1, inplace=False)
        )
        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          (c_fc): Conv1D()
          (c_proj): Conv1D()
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (1): Block(
        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): Attention(
          (c_attn): Conv1D()
          (c_proj): Conv1D()
          (attn_dropout): Dropout(p=0.1, inplace=False)
          (resid_dropout): Dropout(p=0.1, inplace=False)
        )
        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          (c_fc): Conv1D()
          (c_proj): Conv1D()
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (2): Block(
        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): Attention(
          (c_attn): Conv1D()
          (c_proj): Conv1D()
          (attn_dropout): Dropout(p=0.1, inplace=False)
          (resid_dropout): Dropout(p=0.1, inplace=False)
        )
        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          (c_fc): Conv1D()
          (c_proj): Conv1D()
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (3): Block(
        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): Attention(
          (c_attn): Conv1D()
          (c_proj): Conv1D()
          (attn_dropout): Dropout(p=0.1, inplace=False)
          (resid_dropout): Dropout(p=0.1, inplace=False)
        )
        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          (c_fc): Conv1D()
          (c_proj): Conv1D()
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (4): Block(
        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): Attention(
          (c_attn): Conv1D()
          (c_proj): Conv1D()
          (attn_dropout): Dropout(p=0.1, inplace=False)
          (resid_dropout): Dropout(p=0.1, inplace=False)
        )
        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          (c_fc): Conv1D()
          (c_proj): Conv1D()
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (5): Block(
        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): Attention(
          (c_attn): Conv1D()
          (c_proj): Conv1D()
          (attn_dropout): Dropout(p=0.1, inplace=False)
          (resid_dropout): Dropout(p=0.1, inplace=False)
        )
        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          (c_fc): Conv1D()
          (c_proj): Conv1D()
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
    )
    (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  )
  (lm_head): Linear(in_features=768, out_features=50257, bias=False)
)
```

My question is why is the lm_head layer not included as the model's parameters? It bothers me at the moment because I am trying to only finetune the LM layer and realise I cant because doing something like `torch.optim.Adam([p for p in self.parameters() if p.requires_grad], lr=lr, eps=1e-08)` will results in an error as the parameter list is empty


Comments:
It is tied to the input layer, see https://github.com/huggingface/transformers/issues/3799 or the docs here: https://huggingface.co/transformers/model_doc/gpt2.html#transformers.GPT2LMHeadModel. 

If you want to know more about how to fine-tune the model, maybe this helps: https://github.com/huggingface/transformers/issues/1816 or you could ask a question in the forum: https://discuss.huggingface.co/