Converting all model Config classes to dataclasses
Author: norabelrose
Pull request: no
Labels: name: Feature request

It seems that we could save a lot of boilerplate code and potentially prevent some bugs if we migrated all of the model config classes over to being dataclasses. Already many of our classes (BaseModelOutput, TrainingArguments, etc.) are dataclasses, so we are already committed to having dataclasses as a dependency.

It's relatively low priority, but I would be willing to help implement the change since I'm kind of a neat freak about code.

Comments:
I think it would be nice indeed.

It's been on the TO-DO list for a long time (cc @julien-c) but I think nobody's reached it yet so feel free to tackle it :)

Full (and tested) backward compatibility will be paramount though

@thomwolf Thanks for responding— I can start working on this once I get done with my current project, which is adding support for Performer-style attention ([see issue here](https://github.com/huggingface/transformers/issues/7675)).

Seems like this issue is a bit stale but I could use it as a first issue to start contributing. Mind if I take this?

What do you think @sgugger @patrickvonplaten?

Not sure it's worth the time: what exactly would we gain from it apart avoiding storing all args from the init? Since we would need to convert all configurations together (for inheritance you need to go from dataclasses to dataclasses) this is work that can't be split across several PRs and needs to happen all at once.

I also had trouble several times in `TrainingArguments` where the fact it's a dataclass made things harder than they should be, so we may very well lost some of the features of configs (settings params with harmonized names for instance).

> Not sure it's worth the time: what exactly would we gain from it apart avoiding storing all args from the init? Since we would need to convert all configurations together (for inheritance you need to go from dataclasses to dataclasses) this is work that can't be split across several PRs and needs to happen all at once.
> 
> I also had trouble several times in `TrainingArguments` where the fact it's a dataclass made things harder than they should be, so we may very well lost some of the features of configs (settings params with harmonized names for instance).

I think this is a bit easier than you thought because `class` can inherit from `dataclass` and vice versa. We don't have to have a gigantic PR that changes 150+ files. We can change `PretrainedConfig` into `dataclass` first and do the rest in separate PRs in a backward compatible fashion. See the example blow:  
```
@dataclass
class PretrainedConfig:
    x: int = 10
    y: int = 15
    
    def __post_init__(self):
        # for the logic that wouldn't fit in the data class constructor, we can add it here
        self.z = self.x+ self.y

@dataclass
class AlbertConfig(PretrainedConfig):
    xy: int = 100

# this should be backward compatible
class BertConfig(PretrainedConfig):
    def __init__(self, a, b, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.a = a
        self.b = b
```
But I do agree that this is not the best ROI and quite a bit of trivial work just to shave off some boilerplate code.

In that case, okay for me if you want to try to convert `PretrainedConfig` first. If the tests don't pass and you struggle to fix them though, don't spend too much time on it and look for another way to contribute :-)

Hmm this is more complicated than I thought. `PretrainedConfig`, with all that logic in the constructor, properties and class methods, is too heavy of a class to be a textbook `dataclass`. You are right that I should look for something else :)

I've successfully wrapped a dataclass with a decorator. Maybe it's better in this way.

```python
def pretrained_config(cls):
    class Wrapper(transformers.PretrainedConfig):
        pass
    Wrapper.__name__ = cls.__name__
    if hasattr(cls, "model_type"):
        setattr(Wrapper, "model_type", cls.model_type)
    def initializer(self, **kwargs):
        for key, value in vars(cls).items():
            if key.startswith("__") or key == "model_type":
                continue
            value = kwargs.pop(key, value)
            setattr(self, key, value)
        transformers.PretrainedConfig.__init__(self, **kwargs)
    Wrapper.__init__ = initializer
    return Wrapper


@pretrained_config
class MyConfig:
    model_type = "my_config"
    hidden_size: int = 4096
```


@btnkij have you tested your implementation for config hierarchy?