Modify position_embeddings from pre_trained model
Author: brucenguyen1110
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

When I load a model like below: 
`model1 = BertForSequenceClassification.from_pretrained('bert-base-uncased')`
```
BertForSequenceClassification(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (word_embeddings): Embedding(30522, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): BertEncoder(
      (layer): ModuleList(...
```

I want to change Embedding size from 512 to 1024, but when I try to add like this and get an error:
`model = BertForSequenceClassification.from_pretrained('bert-base-uncased', max_position_embeddings=1024)`
> RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
	size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([1024, 768]).

May I know how to change configs of pre-trained model layers?

Comments:
I think you **cannot** change this parameter because doing so you're trying to load weights with (512, 768) shape into an architecture with (1024, 768), and it's not possible.
If my statement is true (maybe some authors of Transformers can confirm or deny my statement), maybe a way to avoid that end users like you try to change this parameter would be to make this variable private, such as `_max_position_embeddings`.

> ##  Questions & Help
> When I load a model like below:
> `model1 = BertForSequenceClassification.from_pretrained('bert-base-uncased')`
> 
> ```
> BertForSequenceClassification(
>   (bert): BertModel(
>     (embeddings): BertEmbeddings(
>       (word_embeddings): Embedding(30522, 768, padding_idx=0)
>       (position_embeddings): Embedding(512, 768)
>       (token_type_embeddings): Embedding(2, 768)
>       (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
>       (dropout): Dropout(p=0.1, inplace=False)
>     )
>     (encoder): BertEncoder(
>       (layer): ModuleList(...
> ```
> 
> I want to change Embedding size from 512 to 1024, but when I try to add like this and get an error:
> `model = BertForSequenceClassification.from_pretrained('bert-base-uncased', max_position_embeddings=1024)`
> 
> > RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
> > size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([1024, 768]).
> 
> May I know how to change configs of pre-trained model layers?

> I think you **cannot** change this parameter because doing so you're trying to load weights with (512, 768) shape into an architecture with (1024, 768), and it's not possible.
> If my statement is true (maybe some authors of Transformers can confirm or deny my statement), maybe a way to avoid that end users like you try to change this parameter would be to make this variable private, such as `_max_position_embeddings`.
> 
As I check with `vars(BertForSequenceClassification.from_pretrained('bert-base-uncased'))`:
```
{'_backend': <torch.nn.backends.thnn.THNNFunctionBackend at 0x1e557269400>,
 '_parameters': OrderedDict(),
 '_buffers': OrderedDict(),
 '_backward_hooks': OrderedDict(),
 '_forward_hooks': OrderedDict(),
 '_forward_pre_hooks': OrderedDict(),
 '_state_dict_hooks': OrderedDict(),
 '_load_state_dict_pre_hooks': OrderedDict(),
 '_modules': ...
 'training': False,
 'config': {
   "attention_probs_dropout_prob": 0.1,
   "finetuning_task": null,
   "hidden_act": "gelu",
   "hidden_dropout_prob": 0.1,
   "hidden_size": 768,
   "initializer_range": 0.02,
   "intermediate_size": 3072,
   "is_decoder": false,
   "layer_norm_eps": 1e-12,
   "max_position_embeddings": 512, <============= This is the one
   "num_attention_heads": 12,
   "num_hidden_layers": 12,
   "num_labels": 2,
   "output_attentions": false,
   "output_hidden_states": false,
   "output_past": true,
   "pruned_heads": {},
   "torchscript": false,
   "type_vocab_size": 2,
   "use_bfloat16": false,
   "vocab_size": 30522
 },
 'num_labels': 2}
```
So I decided to replace `config` with `config=BertConfig(max_position_embeddings=1024)`:
```
{
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 1024, <============== It changed
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 30522
}
```
But the same error is occurred when `BertForSequenceClassification.from_pretrained('bert-base-uncased', config=config)`:
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-9-cfc2c553c1d9> in <module>
      1 config=BertConfig(max_position_embeddings=1024)
----> 2 BertForSequenceClassification.from_pretrained('bert-base-uncased', config=config)

C:\ProgramData\Anaconda3\lib\site-packages\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    457             if len(error_msgs) > 0:
    458                 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 459                                 model.__class__.__name__, "\n\t".join(error_msgs)))
    460 
    461         if hasattr(model, 'tie_weights'):

RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
	size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([1024, 768]).
```
... 🚶 


Sorry, but it is obvious that it doesn't work. As a said before, BERT was trained with a **particular** architecture (i.e. with 512 as max positional embeddings), and it was saved with this shape. You cannot load weights that doesn't match your architecture!

> > I think you **cannot** change this parameter because doing so you're trying to load weights with (512, 768) shape into an architecture with (1024, 768), and it's not possible.
> > If my statement is true (maybe some authors of Transformers can confirm or deny my statement), maybe a way to avoid that end users like you try to change this parameter would be to make this variable private, such as `_max_position_embeddings`.
> 
> As I check with `vars(BertForSequenceClassification.from_pretrained('bert-base-uncased'))`:
> 
> ```
> {'_backend': <torch.nn.backends.thnn.THNNFunctionBackend at 0x1e557269400>,
>  '_parameters': OrderedDict(),
>  '_buffers': OrderedDict(),
>  '_backward_hooks': OrderedDict(),
>  '_forward_hooks': OrderedDict(),
>  '_forward_pre_hooks': OrderedDict(),
>  '_state_dict_hooks': OrderedDict(),
>  '_load_state_dict_pre_hooks': OrderedDict(),
>  '_modules': ...
>  'training': False,
>  'config': {
>    "attention_probs_dropout_prob": 0.1,
>    "finetuning_task": null,
>    "hidden_act": "gelu",
>    "hidden_dropout_prob": 0.1,
>    "hidden_size": 768,
>    "initializer_range": 0.02,
>    "intermediate_size": 3072,
>    "is_decoder": false,
>    "layer_norm_eps": 1e-12,
>    "max_position_embeddings": 512, <============= This is the one
>    "num_attention_heads": 12,
>    "num_hidden_layers": 12,
>    "num_labels": 2,
>    "output_attentions": false,
>    "output_hidden_states": false,
>    "output_past": true,
>    "pruned_heads": {},
>    "torchscript": false,
>    "type_vocab_size": 2,
>    "use_bfloat16": false,
>    "vocab_size": 30522
>  },
>  'num_labels': 2}
> ```
> 
> So I decided to replace `config` with `config=BertConfig(max_position_embeddings=1024)`:
> 
> ```
> {
>   "attention_probs_dropout_prob": 0.1,
>   "finetuning_task": null,
>   "hidden_act": "gelu",
>   "hidden_dropout_prob": 0.1,
>   "hidden_size": 768,
>   "initializer_range": 0.02,
>   "intermediate_size": 3072,
>   "is_decoder": false,
>   "layer_norm_eps": 1e-12,
>   "max_position_embeddings": 1024, <============== It changed
>   "num_attention_heads": 12,
>   "num_hidden_layers": 12,
>   "num_labels": 2,
>   "output_attentions": false,
>   "output_hidden_states": false,
>   "output_past": true,
>   "pruned_heads": {},
>   "torchscript": false,
>   "type_vocab_size": 2,
>   "use_bfloat16": false,
>   "vocab_size": 30522
> }
> ```
> 
> But the same error is occurred when `BertForSequenceClassification.from_pretrained('bert-base-uncased', config=config)`:
> 
> ```
> RuntimeError                              Traceback (most recent call last)
> <ipython-input-9-cfc2c553c1d9> in <module>
>       1 config=BertConfig(max_position_embeddings=1024)
> ----> 2 BertForSequenceClassification.from_pretrained('bert-base-uncased', config=config)
> 
> C:\ProgramData\Anaconda3\lib\site-packages\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
>     457             if len(error_msgs) > 0:
>     458                 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
> --> 459                                 model.__class__.__name__, "\n\t".join(error_msgs)))
>     460 
>     461         if hasattr(model, 'tie_weights'):
> 
> RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
> 	size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([1024, 768]).
> ```
> 
> ...