Unified freezing interface
Author: ZhaofengWu
Pull request: no
Labels: name: Good Second Issue

name: WIP

# 🚀 Feature request

We can freeze the backbone of individual models by finding the field name of the backbone (e.g., `bert` in `BertForSequenceClassification`) and setting `.requires_grad`. However, because the name of this field differs for different models, I don't think there's currently an easy way to do this that works for all models. This contradicts the philosophy that `AutoModel(ForXXX)` should hide all the implementation details.

Ideally, we can pass a `trainable_backbone` flag in the config or to the `__init__` function directly that controls this behavior.

Comments:
Hello, thanks for opening an issue! You're right that this is not as accessible as it could/should be. As a workaround, in PyTorch the base model name should always be set according to the `base_model_prefix`. In order to retrieve the base model, it should be possible to do:
```py
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
base_model = getattr(model, model.base_model_prefix)
```
If you want this to work with `AutoModel` too where the model is the backbone, you can do this instead:
```py
base_model = getattr(model, model.base_model_prefix, model)
```

Thanks! I will keep this issue open since I think this can be more elegantly supported, or at least this logic can be encapsulated with a flag. But feel free to close it.

I find this super relevant honestly.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

It would be nice to have a property to access the base model, similar to the `get_input_embeddings` and `get_output_embeddings` methods. Putting this as a "good second issue" if anyone wants to work on it!

I believe I can take a shot at it if its OK :)

Bringing awareness to a related issue #13413, which argues for a unified interface for something else -- if all models will be edited for this issue anyway, maybe both can be fixed at the same time.

I think the two are separate issues -  let's start with this issue! Feel free to go ahead and open a PR @shabie.

Hey @LysandreJik 

So I started to see how it could be implemented and saw this:

https://github.com/huggingface/transformers/blob/91758e399f8c4bf81820a8af6a257682ccea0223/src/transformers/modeling_utils.py#L547-L552

Doesn't this do what you mentioned?

Indeed, good catch @shabie, my bad! 

So nothing for me to do actually :) and it does solve your problem @ZhaofengWu doesn't it?

On another note, I find the `base_model` still a bit problematic. Take for example a custom token classifier class (it is borrowed from the upcoming book by @lewtun I hope he doesn't mind me quoting this) I am defining:

```python3
class XLMRobertaForTokenClassification(RobertaPreTrainedModel):
    config_class = XLMRobertaConfig

    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels
        self.roberta = RobertaModel(config, add_pooling_layer=False)
        ...
        self.init_weights()
```

Now if the variable name of the base model is not defined like this `self.roberta` then pre-trained weights are not loaded correctly precisely due to how `base_model` works.

I did make this mistake while I was trying it out and was wondering if it would be possible to give people heads up that if you are initializing this class as a backbone, then make sure the attribute is named like this.

This may be accomplished using perhaps the [`__init_subclass__`](https://docs.python.org/3/reference/datamodel.html#object.__init_subclass__) or [`__new__`](https://stackoverflow.com/a/674369/7996306). I am not sure at this point. I can dig deeper if this is genuinely seen as a problem.



Yes, this method does solve my problem -- thanks!

Since the problem is solved with the property found earlier, it seems this issue can be closed.