Urgent: RoBERTa-Large-MNLI does not work for 2-way classification anymore
Author: wyin-Salesforce
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (RoBERTa):

Language I am using the model on (English):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (RTE)

## To Reproduce

Steps to reproduce the behavior:

    pretrain_model_dir = 'roberta-large-mnli' #'roberta-large' , 'roberta-large-mnli'
    model = RobertaForSequenceClassification.from_pretrained(pretrain_model_dir, num_labels=2)

It will have error message as follows:

>     model = RobertaForSequenceClassification.from_pretrained(pretrain_model_dir, num_labels=num_labels)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_utils.py", line 411, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).

This only happened yesterday when I used the pretrained 3-way roberta-large-mnli model for a 2-way classification task; seems like the a bug in initializating or neglecting the classifier's parameters


<!-- Add any other context about the problem here. -->

Comments:
Please choose a better title for your post and specify (or remove) the first part of your post. 

As far as I can tell this is an issue specific to the mnli model. As you say it's pre-trained with three final out features. When loading the state dict into the model, all weights from the pretrained model are "transferred" to the initialized model, this is a one-to-one mapping. Since RobertaForSequenceClassification has a classification head, which you can configure a.o. with `num_labels` it _can_ clash with the classification head of the pretrained model.

The intuitive solution would be to just load all weight excluding the classifier - so that torch doesn't try to load those mis-matching states, if and only if the num_labels specified in `from_pretrained` are not the same as the ones inside the models `self.config`. **However**, I'm not sure if that is the right approach, since one method (from_pretrained) then does different things with the same given pretrained model. In one case you use it completely, in the other you only use part.

Thanks for the hint. But I do not think this is the mnli model problem, because it worked before even the label size is not 3. This is my old log:

09/11/2019 22:37:46 - INFO - pytorch_transformers.modeling_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-mnli-config.json from cache at /root/.cache/torch/pytorch_transformers/54eef9bf74f919edd81b765fee413c8229620f3e271a51bdcdc67797422ef3f3.233bd69ec613d2ebcb1d55823dfc5b1e109157918e13bdbde6db7f694e1a0039
09/11/2019 22:37:46 - INFO - pytorch_transformers.modeling_utils -   Model config {
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "num_attention_heads": 16,
  "num_hidden_layers": 24,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 1,
  "vocab_size": 50265
}

09/11/2019 22:37:46 - INFO - pytorch_transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-mnli-pytorch_model.bin from cache at /root/.cache/torch/pytorch_transformers/1c2e185bc053ae7261ce2289653438a4c05b871ff7f30eaee1cdb787154410e0.c1823b934e18e923174ff260ba955eef25b2205f48fe2655c432a5fb805f8c8a
09/11/2019 22:38:02 - INFO - pytorch_transformers.modeling_utils -   Weights of RobertaForSequenceClassification not initialized from pretrained model: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']
09/11/2019 22:38:02 - INFO - pytorch_transformers.modeling_utils -   Weights from pretrained model not used in RobertaForSequenceClassification: ['lm_head.weight', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias']

You can see that the system can automatically detect which part of the mnli parameters not used to initialize, then it will neglect it; but now the transformers only output error message, I found it from yesterday.

My code is the same, but behave differently now

It's odd since `modeling_utils` hasn't seen any updates apart from the naming update. I'm not sure where else to look for this issue.

My "pytorch_transformers" was installed maybe 3 weeks ago, but the latest "transformers" was installed yesterday. But both will make the same error, in different lines:

"transformers" in line 411:
File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_utils.py", line 411, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).

"pytorch_transformers" in line 594:
File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_utils.py", line 594, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).


But both come from the same reason.  Since the "pytorch_transformers" worked for me a couple days ago (even though i found "transformer" did not work for me yesterday, but very likely something changed before yesterday, since I haven't run the piece of code for some days).

Now, I kind of agree with you that the "roberta-large-mnli" model itself had something changed recently, which makes it unable to neglect the mismatch of hyperparameters.


Also having this issue using the `roberta-large-mnli` model on a single-document (not paired) multiclass classification task.

I guess the simplest solution would be to load the model with previous num_labels and than directly change its num_labels and initialize a new classifier layer in the `run_glue.py` script. This way you won't need to modify any of the `transformers` code.

This is what I do:
```
    # for num_labels(mnli)
    num_labels_old = config_class.from_pretrained(args.model_name_or_path).num_labels
    config = config_class.from_pretrained(args.config_name if args.config_name else args.model_name_or_path,
                                          num_labels=num_labels_old,
                                          finetuning_task=args.task_name,
                                          cache_dir=args.cache_dir if args.cache_dir else None)
    tokenizer = tokenizer_class.from_pretrained(args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
                                                do_lower_case=args.do_lower_case,
                                                cache_dir=args.cache_dir if args.cache_dir else None)
    if num_labels != num_labels_old:
        config.num_labels = num_labels_old
        model = model_class.from_pretrained(args.model_name_or_path,
                                            from_tf=bool('.ckpt' in args.model_name_or_path),
                                            config=config,
                                            cache_dir=args.cache_dir if args.cache_dir else None)
        config.num_labels = num_labels
        logger.info('Reintializing model classifier layer...')
        model.num_labels = num_labels
        model.classifier = RobertaClassificationHead(config)

    else:
        model = model_class.from_pretrained(args.model_name_or_path,
                                            from_tf=bool('.ckpt' in args.model_name_or_path),
                                            config=config,
                                            cache_dir=args.cache_dir if args.cache_dir else None)

```

Of course, it would be better to modify the `transformers` code directly. 


Hi felicity,

Sorry for the late reply. I actually have forgotten how i solve that, or
gave up. I will check my code when I finish a deadline in the next  couple
of days. Thanks for sharing your experience.

Best.

On Thu, Nov 28, 2019 at 8:01 AM felicitywang <notifications@github.com>
wrote:

> I'm getting the same error. Did you solve this problem? @wyin-Salesforce
> <https://github.com/wyin-Salesforce> @pmbaumgartner
> <https://github.com/pmbaumgartner> Would really appreciate it if you
> could share your solutions. Thank you.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/1348?email_source=notifications&email_token=AM2XN4JEMO7KYWJ4OFXFSYDQV7TPFA5CNFSM4I3D6BP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFM7XIA#issuecomment-559545248>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM2XN4IUZKSK4D22E2WI3ZDQV7TPFANCNFSM4I3D6BPQ>
> .
>


-- 


Wenpeng Yin
Research Scientist @ Salesforce Research, Palo Alto
https://sites.google.com/site/yinwenpeng1987/


Thanks @wyin-Salesforce  . If people are still having trouble with this, the solution above worked. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I want to evaluate the pre-trained **roberta-large-mnli** model on a **2-way classification task**. I tried to imitate what @felicitywang posted by adding these four lines after calling config/tokenizer/model in run_glue.py (after line 134):
```
    num_labels = 2 # ADDED
    config.num_labels = num_labels # ADDED
    model.num_labels = num_labels # ADDED
    model.classifier = RobertaClassificationHead(config) # ADDED
```

However, I'm still getting the following error (from line 131) when I run my modified run_glue.py:
```
RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).
```

where the line 131 is the last line ('cache_dir=...') of this code block:
```
    model = AutoModelForSequenceClassification.from_pretrained(
        model_args.model_name_or_path,
        from_tf=bool(".ckpt" in model_args.model_name_or_path),
        config=config,
        cache_dir=model_args.cache_dir,
    )
```

Does anyone know how to make it work?

> I want to evaluate the pre-trained **roberta-large-mnli** model on a **2-way classification task**. I tried to imitate what @felicitywang posted by adding these four lines after calling config/tokenizer/model in run_glue.py (after line 134):
> 
> ```
>     num_labels = 2 # ADDED
>     config.num_labels = num_labels # ADDED
>     model.num_labels = num_labels # ADDED
>     model.classifier = RobertaClassificationHead(config) # ADDED
> ```
> 
> However, I'm still getting the following error (from line 131) when I run my modified run_glue.py:
> 
> ```
> RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
> 	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
> 	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).
> ```
> 
> where the line 131 is the last line ('cache_dir=...') of this code block:
> 
> ```
>     model = AutoModelForSequenceClassification.from_pretrained(
>         model_args.model_name_or_path,
>         from_tf=bool(".ckpt" in model_args.model_name_or_path),
>         config=config,
>         cache_dir=model_args.cache_dir,
>     )
> ```
> 
> Does anyone know how to make it work?

@scarletcho 
If you just want to evaluate the pretrained roberta-large-mnli on a new dataset without any fine-tuning; let's say your new dataset has two classes "entail" and "non_entail", then you just manually combine the outputs "neutral" and "contradict" as a single output "non_entail".

If you want to load this pretrained model and fine-tune on your 2-way dataset, today I just found the following approach works for using N-way fine-tuning:

`    model_config = BartConfig.from_pretrained(pretrain_model_dir)
    model_config.num_labels=new_num_labels
    model = BartForSequenceClassification.from_pretrained(pretrain_model_dir, config=model_config)`
    
I tried Bart, but it should work for roberta too (here "pretrain_model_dir" is string "facebook/bart-large", you can use "roberta-large-mnli" instead)

> I guess the simplest solution would be to load the model with previous num_labels and than directly change its num_labels and initialize a new classifier layer in the `run_glue.py` script. This way you won't need to modify any of the `transformers` code.
> 
> This is what I do:
> 
> ```
>     # for num_labels(mnli)
>     num_labels_old = config_class.from_pretrained(args.model_name_or_path).num_labels
>     config = config_class.from_pretrained(args.config_name if args.config_name else args.model_name_or_path,
>                                           num_labels=num_labels_old,
>                                           finetuning_task=args.task_name,
>                                           cache_dir=args.cache_dir if args.cache_dir else None)
>     tokenizer = tokenizer_class.from_pretrained(args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
>                                                 do_lower_case=args.do_lower_case,
>                                                 cache_dir=args.cache_dir if args.cache_dir else None)
>     if num_labels != num_labels_old:
>         config.num_labels = num_labels_old
>         model = model_class.from_pretrained(args.model_name_or_path,
>                                             from_tf=bool('.ckpt' in args.model_name_or_path),
>                                             config=config,
>                                             cache_dir=args.cache_dir if args.cache_dir else None)
>         config.num_labels = num_labels
>         logger.info('Reintializing model classifier layer...')
>         model.num_labels = num_labels
>         model.classifier = RobertaClassificationHead(config)
> 
>     else:
>         model = model_class.from_pretrained(args.model_name_or_path,
>                                             from_tf=bool('.ckpt' in args.model_name_or_path),
>                                             config=config,
>                                             cache_dir=args.cache_dir if args.cache_dir else None)
> ```
> 
> Of course, it would be better to modify the `transformers` code directly.

Hi,

I am using this code to solve this issue.  What is `RobertaClassificationHead(config)` ? I cannot find this from huggingface.

> > I want to evaluate the pre-trained **roberta-large-mnli** model on a **2-way classification task**. I tried to imitate what @felicitywang posted by adding these four lines after calling config/tokenizer/model in run_glue.py (after line 134):
> > ```
> >     num_labels = 2 # ADDED
> >     config.num_labels = num_labels # ADDED
> >     model.num_labels = num_labels # ADDED
> >     model.classifier = RobertaClassificationHead(config) # ADDED
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > However, I'm still getting the following error (from line 131) when I run my modified run_glue.py:
> > ```
> > RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
> > 	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
> > 	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > where the line 131 is the last line ('cache_dir=...') of this code block:
> > ```
> >     model = AutoModelForSequenceClassification.from_pretrained(
> >         model_args.model_name_or_path,
> >         from_tf=bool(".ckpt" in model_args.model_name_or_path),
> >         config=config,
> >         cache_dir=model_args.cache_dir,
> >     )
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Does anyone know how to make it work?
> 
> @scarletcho
> If you just want to evaluate the pretrained roberta-large-mnli on a new dataset without any fine-tuning; let's say your new dataset has two classes "entail" and "non_entail", then you just manually combine the outputs "neutral" and "contradict" as a single output "non_entail".
> 
> If you want to load this pretrained model and fine-tune on your 2-way dataset, today I just found the following approach works for using N-way fine-tuning:
> 
> ` model_config = BartConfig.from_pretrained(pretrain_model_dir) model_config.num_labels=new_num_labels model = BartForSequenceClassification.from_pretrained(pretrain_model_dir, config=model_config)`
> 
> I tried Bart, but it should work for roberta too (here "pretrain_model_dir" is string "facebook/bart-large", you can use "roberta-large-mnli" instead)

It is ok to use roberta-large, but it stills has the error in roberta-large-mnli.

I fix the issue when I use `transformers=2.3.0` and I put num_labels in the config and then put the config into the model.

I'm using `transformers=4.20.1` and [this example code](https://github.com/huggingface/transformers/blob/24a85cca61fda92b9376fe45da1dcb10c8853066/examples/pytorch/text-classification/run_glue.py) (the most recent commit that passed all the automated testing) and I'm still running into this error.

In the code, it looks like they do [add the num_labels](https://github.com/huggingface/transformers/issues/1348#issuecomment-888779209) to the config, and then put the config into the model, but I'm still getting the error.

The exact command I'm running is `python run_glue.py --train_file sg_train_dataset.csv --validation_file sg_test_dataset.csv --do_train --do_eval --model_name roberta-large-mnli --output_dir output --overwrite_output_dir` where `sg_test_dataset.csv` is a CSV file with three columns, "sentence1", "sentence2" and "label" and "label" is either 0 or 1.

Any suggestions on how to fix it?