Specifying num_labels in from_pretrained without id2label leads to unexpected error
Author: BramVanroy
Pull request: no

This is a "soft" bug report. I _think_ it is the intended usage, but it is not very intuitive in terms of "which arguments go where" in a from_pretrained config call.

To use hyperparameter search, I wrote the following `model_init`. Our pipeline is dynamic, and allows for both classification and regression (num_labels=1) problems. In the latter case, `label2id` and `id2label` are None, which seems natural but which also is the root of the current issue. 

```python
from os import PathLike
from typing import Dict, Optional, Union

from transformers import BertConfig, AutoModelForSequenceClassification


def model_init(model_name_or_path: Union[str, PathLike],
               num_labels: int,
               label2id: Optional[Dict[str, int]] = None,
               id2label: Optional[Dict[int, str]] = None):
    config = BertConfig.from_pretrained(model_name_or_path,
                                        num_labels=num_labels,
                                        label2id=label2id,
                                        id2label=id2label)

    # If you do not include this, you'll get an error
    # config.num_labels = num_labels

    return AutoModelForSequenceClassification.from_pretrained(model_name_or_path, config=config)


if __name__ == "__main__":
    model_init("bert-base-cased", 1)  # also errors with any other integers - due to lack of label mappings

```

I found that when using regression with its respective arguments (as in the example), the code would error out on me. Although I later found that the issue is not num_labels=1 but the lack of id2label maps. Trace:

```
Traceback (most recent call last):
  File "C:\Users\bramv\AppData\Roaming\JetBrains\PyCharm2021.3\scratches\scratch_18.py", line 21, in <module>
    model_init("bert-base-cased", 1)
  File "C:\Users\bramv\AppData\Roaming\JetBrains\PyCharm2021.3\scratches\scratch_18.py", line 17, in model_init
    return AutoModelForSequenceClassification.from_pretrained(model_name_or_path, config=config)
  File "C:\Users\bramv\.virtualenvs\cefr-de-L5vhpNeN\lib\site-packages\transformers\models\auto\auto_factory.py", line 447, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "C:\Users\bramv\.virtualenvs\cefr-de-L5vhpNeN\lib\site-packages\transformers\modeling_utils.py", line 1493, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "C:\Users\bramv\.virtualenvs\cefr-de-L5vhpNeN\lib\site-packages\transformers\models\bert\modeling_bert.py", line 1504, in __init__
    self.num_labels = config.num_labels
  File "C:\Users\bramv\.virtualenvs\cefr-de-L5vhpNeN\lib\site-packages\transformers\configuration_utils.py", line 252, in __getattribute__
    return super().__getattribute__(key)
  File "C:\Users\bramv\.virtualenvs\cefr-de-L5vhpNeN\lib\site-packages\transformers\configuration_utils.py", line 391, in num_labels
    return len(self.id2label)
TypeError: object of type 'NoneType' has no len()
```

To investigate what is happening, I cloned the library and printed out 1. whether num_labels is available in the kwargs passed to init; 2. when the num_labels.setter is called with which value. I print a newline at the end of the __init__ call. 

```
num_labels in kwargs False 
num_labels setter called 2

num_labels setter called 1
num_labels in kwargs False 
num_labels setter called 2

num_labels in kwargs False 
num_labels setter called 2
```

What is going on here? Why is init called three times? In the second one, it seems that for a brief moment the setter is called with the correct value (1), but then it gets overwritten with the default of 2 again (?)

What confuses me the most is that despite that the setter is clearly being called, which in turn should also set the id2label map, I still get the `'NoneType' has no len()` error on the final config.

Setting `config.num_labels = num_labels` solves the issue, which explicitly calls the setter and creates the `id2label` map. But that does not feel very intuitive because I set num_labels already in the `from_pretrained` call, where I _can_ also set the label mappings. So I don't run into problems when I specify the label2id/id2label maps (as expected). 

**tl;dr** you can specify id2label maps in from_pretrained() but only specifying `num_labels` won't work. You have to explicitly call the num_labels setter to ensure that the label maps are created. It would be more user-friendly if this could be done on-the-fly without the extra manual `config.num_labels = num_labels` step.

## Environment info
- `transformers` version: 4.17.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2
- PyTorch version (GPU?): 1.11.0+cu113 (True)


@sgugger @LysandreJik 

Sorry for the long post, and for tagging. Wasn't sure who to tag for this so feel free to pass it on to someone else.


Comments:
This is a bit tricky and I'm not entirely sure it's a bug. The crux of the issue is that you are technically doing this:
```
from transformers import BertConfig

config = BertConfig.from_pretrained(
    "bert-base-cased",
     num_labels=1,
     label2id=None,
     id2label=None,
)
```

So a `BertConfig` is instantiated from the checkpoint. Then the `num_labels` are changed to be set at 1, then the `label2id` is changed to be set as `None` and likewise for the `id2label`.

When then requesting `num_labels` (which is not a direct attribute if you look in the code), the lib tries to fetch` len(id2label)` which is `None` since you forced that by passing the kwarg.

We could ignore `None` values in kwargs passed along to `BertConfig.from_pretrained` but the idea (documented) is that what you passes erases the value of the checkpoint, so the behavior is not entirely wrong.

Thanks for the quick reply!

Isn't the order of setting the arguments reversed? First te label maps are set (None as a default), and then num_labels is set. When the maps are None, num_labels is taken from kwargs.

https://github.com/huggingface/transformers/blob/d55fcbcc50a27f4f5ad8a5c83786905e55212fa0/src/transformers/configuration_utils.py#L304-L311

This in turn calls the setter, which generates the label maps if they do not exist:

https://github.com/huggingface/transformers/blob/d55fcbcc50a27f4f5ad8a5c83786905e55212fa0/src/transformers/configuration_utils.py#L388-L399

So this is what I had expected to happen with my example:

1. label maps are None
2. -> num_labels is taken from kwargs and `set`
3. -> num_labels.setter is called
4. -> because the given label maps are None, they are re-generated for num_labels==1

But this doesn't seem to be happening and I don't quite follow why not. Is it because the label maps are populated with the values from the pretrained model, overwriting my own kwargs instead of the other way around? So these won't actually be None, but the values from the pretrained config?

https://github.com/huggingface/transformers/blob/d55fcbcc50a27f4f5ad8a5c83786905e55212fa0/src/transformers/configuration_utils.py#L304-L305

You are reading the wrong code. When using `from_pretrained` with kwargs as you do:
- the config is fully instantiated from the checkpoint
- then each kwarg in order is set in this instantiated config

You're absolutely right. 

Is it an idea to allow _either_ the id2label map _or_ num_labels? (in init or at least in the pretrained call) It seems to me that id2label is automatically generated anyway if not given based on num_labels, and conversly num_labels is actually the number of labels in the id2label mapping. So the init should only every require one of those two if I'm not mistaken.

That may take away a lot of confusion.

We could add defensive checks, that's a good idea (at least that the two args are consistent when passed together).

Did that in the PR mentioned above if you want to have a look.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.