How to  freeze  GPT-2 model  layers with Tensorflow/Keras?
Author: kmkarakaya
Pull request: no

### Feature request

I read the post https://github.com/huggingface/transformers/issues/12881.
It seems that no progress on feezing GPT layers with keras has been reported.
 
Are there any updates or work around for freezing  GPT-2 model  layers at Tensorflow?
Thank you

### Motivation

For finetuning, I want to update only some layers

### Your contribution

I can test the solution

Comments:
cc @Rocketknight1 @gante 

Hi @kmkarakaya 👋 

⚠️ Disclaimer: this is untested with `.fit()`, please let me know if it works! I'll edit the comment afterwards depending on your answer.

The weights can be frozen by setting a `Layer` attribute to `False`, as per [Keras' docs](https://keras.io/guides/transfer_learning/#freezing-layers-understanding-the-trainable-attribute). The original issue you linked attempts to freeze the `.weights` attribute of a model, which looks similar... but not the same :D

To reach the layer you want to freeze, the best way is to navigate the code of the original model and find its attribute name. For instance, let's say you have to freeze GPT-2's word embeddings. If you check the [code](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_tf_gpt2.py#L322), you see that it is a `Layer` inside `TFGPT2MainLayer` named `wte`. `TFGPT2MainLayer`, in its turn, is a `transformer` attribute in the instantiable classes.

In summary, do NOT do this 
```python
from transformers import TFAutoModelForCausalLM
model = TFAutoModelForCausalLM.from_pretrained('gpt2')
model.weights[6]._trainable=False
model.compile()
model.summary()
```
(notice that ALL weights are trainable)

and do this instead
```python
from transformers import TFAutoModelForCausalLM
model = TFAutoModelForCausalLM.from_pretrained('gpt2')
model.transformer.wte.trainable=False
model.compile()
model.summary()
```
(a large amount of weights are now tagged as untrainable)

@gante thank you for the reply.

Actually, I attempted reading the code but I could not locate the transformer block names to use them in freezing.
As you suggested the "wte" name can be used to freeze the embedding layer.
However, when I want to freeze let's say the first 3 transformer blocks how should I code it?

I tried to freeze the first transformer block (model.transformer.h_._0.trainable=False) but generated the error msg: 

![image](https://user-images.githubusercontent.com/41159849/185100015-a65423ee-5f23-409e-b383-2ca8c742aebd.png)


However,[ in the code](https://github.com/huggingface/transformers/blob/c99e984657b64dd8f19de74405bbf13763ab4f2b/src/transformers/models/gpt2/modeling_tf_gpt2.py#L322), as far as I understand, the block names are like h__._x.
![image](https://user-images.githubusercontent.com/41159849/185100408-66854c2a-156b-4572-91c0-fac1a662fcc7.png)



Also, I check these names by "model.transformer.trainable_variables".

Could you help me to freeze transformer block weights so that I can partially fine-tune a transformer model?

Thanks a lot.


@gante I now tried the below code:
![image](https://user-images.githubusercontent.com/41159849/185101607-8303dee4-e57e-49d7-a4a4-4738d23ed90b.png)
It seems to be working.
Do you think that it is the correct way of freezing a transformer block's weights in GPT2?
Because, I'm not sure :)
Thanks.

Hey @kmkarakaya 👋 

That's true, some names are non-trivial to match, due to internal name mapping. In this case, the layer is named `model.transformer.h[0]`, so setting `model.transformer.h[0].trainable=False` will work. In general, if you see `layer_._{index}`, you can access them as `layer[index]`.

@gante Thank you for the reply, I noted the info.

I checked if I can freeze the model by freezing all the embeddings and the blocks as below.
But it seems there is something missing from my attention :)
What else should I freeze in a GPT2 model other than the embedding and the blocks?
![image](https://user-images.githubusercontent.com/41159849/185105549-5bbfe0a7-3c14-49cb-aab0-a4d49153bce4.png)
 

If you want to freeze everything, then you should freeze `model.transformer` (which is a layer). I don't know all parameters on top of my head, so if there are parameters remaining you should attempt to find the trainable parameters :)

I recognized that "wpe" is missing:
![image](https://user-images.githubusercontent.com/41159849/185109289-502c8721-77f3-4288-844b-33501bbc9cce.png)

However, I cannot set it to untrainable.
![image](https://user-images.githubusercontent.com/41159849/185108667-b286fb87-fd5f-4eb5-a7b0-77a87151928f.png)

Frankly, I don't know what this layer does.
Do you have any clue?
Thanks


`wpe` are the position embeddings. Their code is [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_tf_gpt2.py#L329) -- I don't know how to freeze these ones :(

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> `wpe` are the position embeddings. Their code is [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_tf_gpt2.py#L329) -- I don't know how to freeze these ones :(

what about a new variable?

`model.transformer.wpe= tf.Variable(model.transformer.wpe, trainable=False)
`