GPT-2 pretrained model fails to load when TF v2 behaviour is disabled
Author: rifatarefin
Pull request: no

I am trying to use GPT-2 in a codebase that is written for Tensorflow 1.x. However, I am running the code against TF 2.x installation binaries with `tf.disable_v2_behavior()` flag. Without this `tf.disable_v2_behavior()` flag, GPT-2 pretrained model loads fine, but the model fails to load if the flag is used. Here is my code :
```
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior() #works fine without this line

from transformers import TFGPT2Model
model = TFGPT2Model.from_pretrained('gpt2') #fails
```
Here is the error:

```
>>> TFGPT2Model.from_pretrained('gpt2')
2022-02-15 10:17:08.792655: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home1/07782/marefin/.local/lib/python3.8/site-packages/transformers/modeling_tf_utils.py", line 1467, in from_pretrained
    model(model.dummy_inputs)  # build the network with dummy inputs
  File "/home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 783, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 695, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

    /home1/07782/marefin/.local/lib/python3.8/site-packages/transformers/models/gpt2/modeling_tf_gpt2.py:628 call  *
        outputs = self.transformer(
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:763 __call__  **
        self._maybe_build(inputs)
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:2084 _maybe_build
        self.build(input_shapes)
    /home1/07782/marefin/.local/lib/python3.8/site-packages/transformers/models/gpt2/modeling_tf_gpt2.py:241 build
        self.wpe = self.add_weight(
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:441 add_weight
        variable = self._add_variable_with_custom_getter(
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py:810 _add_variable_with_custom_getter
        new_variable = getter(
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_utils.py:127 make_variable
        return tf_variables.VariableV1(
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/ops/variables.py:260 __call__
        return cls._variable_v1_call(*args, **kwargs)
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/ops/variables.py:206 _variable_v1_call
        return previous_getter(
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/ops/variables.py:199 <lambda>
        previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/ops/variable_scope.py:2612 default_variable_creator
        return resource_variable_ops.ResourceVariable(
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/ops/variables.py:264 __call__
        return super(VariableMetaclass, cls).__call__(*args, **kwargs)
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py:1584 __init__
        self._init_from_args(
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py:1722 _init_from_args
        initial_value = initial_value()
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/keras/initializers/initializers_v2.py:413 __call__
        dtype = _assert_float_dtype(_get_dtype(dtype))
    /home1/07782/marefin/.local/lib/python3.8/site-packages/tensorflow/python/keras/initializers/initializers_v2.py:948 _assert_float_dtype
        raise ValueError('Expected floating point type, got %s.' % dtype)

    ValueError: Expected floating point type, got <dtype: 'int32'>.
```

I am using TF 2.5 with transformers v4.12.5 in CentOS 7. Is there any way around to make this work with TF v2 behavior disabled?

Comments:
Hi @rifatarefin 👋  Sadly, we don't support TF 1.x, everything we've built was with TF 2.x in mind. This implies that the snippet you want to run shouldn't be able to and that you have four main alternatives to your problem:
1. Create two local environments, one with TF 1.x other with TF 2.x, load GPT-2 in your TF 2.x environment and make it communicate with your TF 1.x environment (which runs your model);
2. Use our [HF API](https://huggingface.co/inference-api) with GPT-2. It's essentially the same as the alternative above, except that you don't have the hassle of setting up the second environment;
3. Fork our code and modify GPT-2 so as to make it TF 1.x compatible. I have no guarantee this would work. If you do make it work, we'd be interested in knowing the solution;
4. Upgrade your project to TF 2.x. But since you are raising this issue, I'd assume it's a very hard task.

I found `TFGPT2Model` pretrained weights load with `tf.compat.v1.disable_eager_execution()`. The TF 1.x code didn't raise any error with this flag. Although not sure if I will get the desired output this way. Any suggestions @gante ?

Awesome -- if you didn't get any errors then it possibly works well 👌 

To double-check, I'd suggest that you run two things after loading with `tf.compat.v1.disable_eager_execution()`:
1. See if you can reproduce the output of [our integration tests](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_gpt2.py#L430) locally (this would confirm the correctness of the model);
2. Try running inputs of variable length on the same loaded `model` object (TF 1.x is not as friendly with dynamic input lengths).

If these 2 tests run well, I'd say you should be able to pull it off, at least for forward passes 😉  If it fails on the 2nd test but passes the 1st, let me know, I may have a work-around.

Thank you so much! I will keep you posted @gante 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.