ValueError: Expected floating point type, got <dtype: 'int32'> for TFGPT2LMHeadModel
Author: farazk86
Pull request: no

Hi,

I am trying to serve a gpt2 model online using Google cloud. But when creating model environment I get the error: 
```
Create Version failed. Bad model detected with error: "Failed to load model: Unexpected error when loading the model: in user code:\n\n /tmp/custom_lib/transformers/modeling_tf_gpt2.py:551 call *\n transformer_outputs = self.transformer(inputs, **kwargs)\n /tmp/custom_lib/transformers/modeling_tf_gpt2.py:321 call *\n inputs_embeds = self.wte(input_ids, mode=\"embedding\")\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py:758 __call__ **\n self._maybe_build(inputs)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py:2131 _maybe_build\n self.build(input_shapes)\n /tmp/custom_lib/transformers/modeling_tf_utils.py:1522 build\n \"weight\", shape=[self.vocab_size, self.hidden_size], initializer=get_initializer(self.initializer_range)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py:447 add_weight\n caching_device=caching_device)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/training/tracking/base.py:743 _add_variable_with_custom_getter\n **kwargs_for_getter)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py:141 make_variable\n shape=variable_shape if variable_shape else None)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variables.py:259 __call__\n return cls._variable_v1_call(*args, **kwargs)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variables.py:220 _variable_v1_call\n shape=shape)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variables.py:198 <lambda>\n previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variable_scope.py:2598 default_variable_creator\n shape=shape)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variables.py:263 __call__\n return super(VariableMetaclass, cls).__call__(*args, **kwargs)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1434 __init__\n distribute_strategy=distribute_strategy)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1567 _init_from_args\n initial_value() if init_from_fn else initial_value,\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py:121 <lambda>\n init_val = lambda: initializer(shape, dtype=dtype)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/init_ops_v2.py:445 __call__\n dtype = _assert_float_dtype(dtype)\n /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/init_ops_v2.py:1037 _assert_float_dtype\n raise ValueError(\"Expected floating point type, got %s.\" % dtype)\n\n ValueError: Expected floating point type, got <dtype: 'int32'>.\n (Error code: 0)"
```

I saved the model from a fine-tuned GPT2 model:

```python
tf_model = TFGPT2LMHeadModel.from_pretrained("checkpoint-8000", from_pt=True)
tf_model.save_pretrained("tensorflow-model")

model_class, tokenizer_class = TFGPT2LMHeadModel, GPT2Tokenizer
tokenizer = tokenizer_class.from_pretrained('tensorflow-model')
model = model_class.from_pretrained('tensorflow-model')
```

This model works on my local machine using ``model.generate()``. But I get the error above when creating model environment on GCP.

I dont know if this is a Google cloud issue or transformers issue. However, when looking at the model created by the line 
```python
model = model_class.from_pretrained('tensorflow-model')
``` 
I can see that the ``model.dtype`` and ``model.variable_dtype`` is float32. 

Can anyone help why google cloud thinks this model expects a ``float32`` input and not ``int32``. Can I change anything in model to ensure that correct input dtype?

Thanks

Comments:
@jplu do you have any experience with creating model environments on GCP with TensorFlow?

Hello!

Can you first share your local env and the env you are using on your GCP machine?  

Hi @jplu 

Thanks for the response. 

For my local environment I'm on Python 3.6, tenensorflow 2.4, transformers 2.8.0. Is this what you meant by local environment? 

On GCP, here is my model version settings:

```
Model    sentence_generator
Model location    gs://gpt2-checkpoint/tensorflow-model/
Creation time    Jan 16, 2021, 3:51:39 PM
Last use time
Python version    3.7
Runtime version    2.2
Custom code and dependencies    gs://gpt2-checkpoint/staging-dist/generator_package-0.6.tar.gz
Prediction class    generator_class_tf.GeneratorClass
Machine type    Single core CPU
Auto scaling minimum nodes    1
```
Below is my ``setup.py`` file:

```python
from setuptools import setup


setup(
    name="generator_package",
    version="0.6",
    include_package_data=True,
    scripts=["generator_class_tf.py"],
    install_requires=['transformers==2.8.0']
)
```


Yes, this is what I meant. I see that you are using an old version of transformers, can you update to the last release please.

> Yes, this is what I meant. I see that you are using an old version of transformers, can you update to the last release please.

With the latest version of transformers I get this error:

> Create Version failed. Bad model detected with error: "Failed to load model: Unexpected error when loading the model: problem in generator_class_tf - DistributionNotFound: The 'tqdm>=4.27' distribution was not found and is required by this application, \nTry: pip install transformers -U or pip install -e '.[dev]' if you're working with git master (Error code: 0)"

And to the best of my knowledge, I dont think we can ``pip install`` anything with Google cloud prediction environment.



You can just replace your `setup.py` file with
```
from setuptools import setup


setup(
    name="generator_package",
    version="0.6",
    include_package_data=True,
    scripts=["generator_class_tf.py"],
    install_requires=['transformers==4.2.1']
)
```

> You can just replace your `setup.py` file with
> 
> ```
> from setuptools import setup
> 
> 
> setup(
>     name="generator_package",
>     version="0.6",
>     include_package_data=True,
>     scripts=["generator_class_tf.py"],
>     install_requires=['transformers==4.2.1']
> )
> ```

Thanks but this is how I had my ``setup.py`` when I got the error above relating to ``tqdm``. 

Ok, then did you try:
```
from setuptools import setup


setup(
    name="generator_package",
    version="0.6",
    include_package_data=True,
    scripts=["generator_class_tf.py"],
    install_requires=['transformers==4.2.1', 'tqdm>=4.27']
)
```

Yes, I have. still get the same error :(

> Create Version failed. Bad model detected with error: "Failed to load model: Unexpected error when loading the model: problem in generator_class_tf - DistributionNotFound: The 'tqdm>=4.27' distribution was not found and is required by this application, \nTry: pip install transformers -U or pip install -e '.[dev]' if you're working with git master (Error code: 0)"

According to this page, it should work, https://cloud.google.com/ai-platform/training/docs/packaging-trainer

So the problem might come from somewhere else. I suppose you can run your model as expected locally?

Yes, that's correct. It works without problems on my own machine.

This looks to be a problem with GCP. I'll lodge this as a bug on their issue tracker and update here on any progress made.



I'm seeing the same issue for my deployment, using transformers 4.5.0. But it seems indeed to be a GCP issue.
Have you seen any comments from Google on this @farazk86  ?

Thanks in advance!

> I'm seeing the same issue for my deployment, using transformers 4.5.0. But it seems indeed to be a GCP issue.
> Have you seen any comments from Google on this @farazk86 ?
> 
> Thanks in advance!

Yes, this is a GCP issue.
Unfortunately, I gave up in the end. As the issue I created on Google issue tracker also did not help, they were asking for me to provide information from methods within transformers library that I was not familiar with or knew about. It was too much of a hassle - I just gave up.

Alright, thanks for the quick reply! That is too bad, I will keep trying myself, and let you know if I find a solution.

Just for my curiosity, did you instead take any alternative approach (than using Custom Prediction Routines) in order to serve a Transformer model on Google?
I read some people had success with using Docker containers on the "Cloud Run" API.

No, I just gave up on cloud entirely. And you are right, I had also determined that Docker works as other people on stackoverflow had achieved to deploy using Docker. But I had no experience with docker and just moved on to other projects.

If you do manage to figure it out, then yes please do let me know even though now the $300 introductory credits are also expired :)

I will keep you informed on my progress for sure.

Could you provide the link to the issue tracker / bug report you submitted with GCP? 
If that is a public page that is.

Thanks in advance!

sure, I'll try to find it. 

> I will keep you informed on my progress for sure.
> 
> Could you provide the link to the issue tracker / bug report you submitted with GCP?
> If that is a public page that is.
> 
> Thanks in advance!

Here are both my submitted issues, based on my multiple tries at making this work: https://issuetracker.google.com/issues/177648341 and https://issuetracker.google.com/issues/178236762

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.