How do I change the classification head of a model from multi-label to multi-class?
Author: pratikchhapolika
Pull request: no

 am downloading the model https://huggingface.co/unitary/unbiased-toxic-roberta/tree/main and then using it.

**Transformer Version: '4.11.3'**

**unbiased-toxic-roberta**

The model is trained on Jigsaw data which has **16 classes ( Multi-label )** and use **BCE** loss function

`Labels is like : sentence1---> label [0,0,..1,0..,0]`

**I am using this model to fine-tune on binary classification problem. ( 0 and 1 as my labels)**

`Labels is like : sentence1---> label 1`
`Labels is like : sentence2---> label 0`

My loss function is simply an accuracy loss defined below.





I have written the below code:



```
def compute_metrics(eval_pred):
    logits, labels = eval_pred
   

    predictions = np.argmax(logits, axis=-1)
    
    acc = np.sum(predictions == labels) / predictions.shape[0]
    
    return {"accuracy" : acc}
```


```
model = tr.RobertaForSequenceClassification.from_pretrained("/home/pc/unbiased_toxic_roberta",num_labels=2)


#ignore_mismatched_sizes=True,
model.to(device)



training_args = tr.TrainingArguments(
    output_dir='/home/pc/1_Proj_hate_speech/results_roberta',          # output directory
    overwrite_output_dir = True,
    num_train_epochs=20,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=32,   # batch size for evaluation
    learning_rate=2e-5,
    warmup_steps=1000,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs3',            # directory for storing logs
    logging_steps=1000,
    evaluation_strategy="epoch"
    ,save_strategy="epoch"
    ,load_best_model_at_end=True
)


trainer = tr.Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_data,         # training dataset
    eval_dataset=val_data,             # evaluation dataset
    compute_metrics=compute_metrics
)

```


Error:

```
- classifier.out_proj.weight: found shape torch.Size([16, 768]) in the checkpoint and torch.Size([2, 768]) in the model instantiated
- classifier.out_proj.bias: found shape torch.Size([16]) in the checkpoint and torch.Size([2]) in the model instantiated
- 
```

**How can I change the code to accommodate the number of classes and  accuracy loss instead of BCE**

Comments:
One can replace the head by setting the `ignore_mismatched_sizes` argument to `True` in the `from_pretrained` method, like so:

```
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("unitary/unbiased-toxic-roberta", num_labels=2, ignore_mismatched_sizes=True)
```

This will print a warning, indicating which parameters are randomly initialized.
                 

> One can replace the head by setting the `ignore_mismatched_sizes` argument to `True` in the `from_pretrained` method, like so:
> 
> ```
> from transformers import AutoModelForSequenceClassification
> 
> model = AutoModelForSequenceClassification.from_pretrained("unitary/unbiased-toxic-roberta", num_labels=2, ignore_mismatched_sizes=True)
> ```
> 
> This will print a warning, indicating which parameters are randomly initialized.

**This introduces another error:**

```
Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at /home/pc/unbiased_toxic_roberta and are newly initialized because the shapes did not match:
- classifier.out_proj.weight: found shape torch.Size([16, 768]) in the checkpoint and torch.Size([2, 768]) in the model instantiated
- classifier.out_proj.bias: found shape torch.Size([16]) in the checkpoint and torch.Size([2]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.


```


```
loading configuration file /home/pc/unbiased_toxic_roberta/config.json
Model config RobertaConfig {
  "architectures": [
    "RobertaForSequenceClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "classifier_dropout": null,
  "eos_token_id": 2,
  "function_to_apply": "sigmoid",
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "problem_type": "multi_label_classification",
  "transformers_version": "4.10.3",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 50265
}
```

```

File "/home/pc/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2958, in binary_cross_entropy_with_logits
    raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
ValueError: Target size (torch.Size([2])) must be the same as input size (torch.Size([2, 2]))
```


**Here is the model structure:**

```
(11): RobertaLayer(
          (attention): RobertaAttention(
            (self): RobertaSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): RobertaSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): RobertaIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): RobertaOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
  )
  (classifier): RobertaClassificationHead(
    (dense): Linear(in_features=768, out_features=768, bias=True)
    (dropout): Dropout(p=0.1, inplace=False)
    (out_proj): Linear(in_features=768, out_features=2, bias=True)
  )
)


```

Apologies, you also have to set the problem_type to "single_label_classification":

```
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("unitary/unbiased-toxic-roberta", problem_type="single_label_classification", num_labels=2, ignore_mismatched_sizes=True)
```
This ensures the cross-entropy loss is used instead of the binary cross-entropy with logits (BCE) loss.