Tensorflow GPT-2 model incapable of freezing layers
Author: Elysium1436
Pull request: no

I am trying to finetune gpt-2 by freezing some layers according to [this article](https://arxiv.org/pdf/2103.05247.pdf). Freezing the specified layers doesn't change the number of trainable parameters, even though accessing the .trainable attribute of the weights of the model shows that they are False.
```python
from transformers import TFAutoModelForCausalLM
model = TFAutoModelForCausalLM.from_pretrained('gpt2')

#Picking random weight
w = model.weights[6]
w #<tf.Variable 'tfgp_t2lm_head_model_2/transformer/h_._0/attn/c_proj/weight:0' shape=(768, 768) dtype=float32, numpy=...
w._trainable = False
w.trainable #False
#Confirming that trainable is false in the model
model.weights[6].trainable #False
model.compile(...)
model.summary()
```
prints
```
Model: "tfgp_t2lm_head_model_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
transformer (TFGPT2MainLayer multiple                  124439808 
=================================================================
Total params: 124,439,808
Trainable params: 124,439,808
Non-trainable params: 0
_________________________________________________________________
```
Using the ```.get_weights()``` method returns only numpy arrays, so I use .weights.

Freezing all weights the same way in the run_clm.py tensorflow script results in the same summary, and the loss value at each step does decrease, indicating that the weights are being updated. Am I missing something or is this a bug?


Comments:
Hi, I think this is going to be quite difficult in Keras given the way our models are implemented, as I believe Keras only supports freezing weights on Layer objects, and we haven't implemented the individual pieces of GPT2 as Keras Layers.

If you'd like to only train specific pieces of your model, I'd recommend writing a manual eager training loop with GradientTape, see [here](https://www.tensorflow.org/guide/autodiff). For example, something like this (note: untested code!) would work, assuming you have a batch of data as a dict with at least `'input_ids'` and `'labels'` keys:
```
trainable_weights = model.weights[6:8]  # Just picking a list of some random weights to update, you can pick specific ones!
optimizer = tf.keras.optimizers.Adam(5e-5)
with tf.GradientTape() as tape:
    outputs = model(batch)
    loss = outputs['loss']
    grads = tape.gradient(loss, trainable_weights)
optimizer.apply_gradients(zip(grads, model.trainable_weights))
```


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Are there any updates or work around for freezing  GPT-2 model  layers at Tensorflow?
Thank you

(in case you stumble upon this issue and you have the same question, check #18282)