Problems when restoring the pretrain weights for TFbert
Author: BraceLau
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using Bert:

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.when I try to reload the weights for TFbert by using: 
    model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased')
2. I got the error 
    ValueError: Expected floating point type, got <dtype: 'int32'>.
3. Does anybody know how to fix this?

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Hi, could you provide information about your setup? Which TensorFlow version are you using, which python version, which Transformers version? Thank you.

> Hi, could you provide information about your setup? Which TensorFlow version are you using, which python version, which Transformers version? Thank you.

Hi, the information about my setup:
Python version: 3.6.9
Tensorflow Version: 2.0.0b1
PyTorch Transformers version (or branch):2.1.1

Thank you

I can't reproduce this on my side while on the same versions. Would you happen to have a folder called "bert-base-uncased" in the same directory from which you're calling the python script?

Hi,

I have exactly the same error while doing the same task...

Python version: Python 3.5.3
Tensorflow Version: 2.0.0-beta1
PyTorch Transformers version : transformers (2.1.1)


In my environment, **it works as expected**!
@YumingLiu1996 @LeonardoGracioS 

N.B: It works without `force_download=True` parameter too.

```
>>> from transformers import TFBertForSequenceClassification
2019-11-12 14:38:24.462663: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-12 14:38:24.466451: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3408000000 Hz
2019-11-12 14:38:24.466942: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56248b3c66f0 executing computations on platform Host. Devices:
2019-11-12 14:38:24.466956: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
>>> model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased', force_download=True)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 313/313 [00:00<00:00, 154303.85B/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 536063208/536063208 [00:48<00:00, 11140340.62B/s]
>>> import platform
>>> platform.platform()
'Linux-4.15.0-66-generic-x86_64-with-debian-buster-sid'
>>> platform.python_version()
'3.6.9'
>>> import torch
>>> torch.__version__
'1.0.1'
>>> import tensorflow as tf
>>> tf.__version__
'2.0.0'
>>> import transformers
>>> transformers.__version__
'2.1.1'
```

> I can't reproduce this on my side while on the same versions. Would you happen to have a folder called "bert-base-uncased" in the same directory from which you're calling the python script?



> I can't reproduce this on my side while on the same versions. Would you happen to have a folder called "bert-base-uncased" in the same directory from which you're calling the python script?

No, I don't have such as folder.. 

> I can't reproduce this on my side while on the same versions. Would you happen to have a folder called "bert-base-uncased" in the same directory from which you're calling the python script?

I redownload tensorflow 2.0.0 and transformers. And now I encounter another problem:

```
h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5f.pyx in h5py.h5f.open()

OSError: Unable to open file (truncated file: eof = 485070229, sblock->base_addr = 0, stored_eof = 497933648)
```
Do you know the solution for this problem?


@YumingLiu1996 the error `OSError: Unable to open file`  tells you that the file was not downloaded in its entirety. Please add the `force_download=True` parameter when downloading the pre-trained model:

```py
model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased', force_download=True)
```

I solved the issue by using a Google Cloud Platform virtual machine with a TF 2.0 environment.

> I solved the issue by using a Google Cloud Platform virtual machine with a TF 2.0 environment.

I am using a Google Cloud Platform with an K80 and I have the TF2 installed but it doesn't works...
the same problem here guys. 

I don't know why, but it only works on Google Colab until this comment. 

> > I solved the issue by using a Google Cloud Platform virtual machine with a TF 2.0 environment.
> 
> I am using a Google Cloud Platform with an K80 and I have the TF2 installed but it doesn't works...
> the same problem here guys.
> 
> I don't know why, but it only works on Google Colab until this comment.

Do you have exactly the same error ? Did you start your virtual machine with a TF 2.0 environment and not just a "pip install tensorflow=2.0.0" ?

I did through the pip install, because I have this machine running some
other things. So, the problem is the same but I think could be the
Tensorflow version.

I will try to build TF 2 from source.



Em sex, 15 de nov de 2019 06:07, Samuel Leonardo Gracio <
notifications@github.com> escreveu:

> I solved the issue by using a Google Cloud Platform virtual machine with a
> TF 2.0 environment.
>
> I am using a Google Cloud Platform with an K80 and I have the TF2
> installed but it doesn't works...
> the same problem here guys.
>
> I don't know why, but it only works on Google Colab until this comment.
>
> Do you have exactly the same error ? Did you start your virtual machine
> with a TF 2.0 environment and not just a "pip install tensorflow=2.0.0" ?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/1780?email_source=notifications&email_token=AB4RK3ZWJWXUQDPECFSVTPLQTZRGXA5CNFSM4JLK64N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEEZC4Q#issuecomment-554275186>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB4RK32QNPOWOL3OVKAP2FTQTZRGXANCNFSM4JLK64NQ>
> .
>


same problem here:

    OS:
    Python version: 3.6.8
    PyTorch version: 1.2.0
    TF version: 2.0.0b1
    PyTorch Transformers version (or branch): master (Nov 25, 2019)

model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')

ValueError: Expected floating point type, got <dtype: 'int32'>.

looks like a tensorflow problem


I've found your problem! It's a TensorFlow bug in version 2.0.0beta1. If you uninstall this version with `pip uninstall tensorflow` and after that you install the TensorFlow with version 2.0.0a0 with `pip install tensorflow==2.0.0,` **the code works as expected**!

I've tried different settings and it works as expected!
- PyTorch 1.2.0, TensorFlow(CPU-version) 2.0.0
- PyTorch 1.2.0, TensorFlow(GPU-version) 2.0.0
- PyTorch 1.3.1, TensorFlow(CPU-version) 2.0.0
- PyTorch 1.3.1, TensorFlow(GPU-version) 2.0.0

> same problem here:
> 
> ```
> OS:
> Python version: 3.6.8
> PyTorch version: 1.2.0
> TF version: 2.0.0b1
> PyTorch Transformers version (or branch): master (Nov 25, 2019)
> ```
> 
> model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
> 
> ValueError: Expected floating point type, got <dtype: 'int32'>.
> 
> looks like a tensorflow problem

thanks. I tryied with a0 but it has even more problem, like 

'tensorflow.python.keras.api._v2.keras.layers' has no attribute 'LayerNormalization'

which are said to be bugs in a0 and patched in b1. 

However b1 does not work because
ValueError: Expected floating point type, got <dtype: 'int32'>.

Looks like this is a deadlock.

> I've found your problem! It's a TensorFlow bug in version 2.0.0beta1. If you uninstall this version with `pip uninstall tensorflow` and after that you install the TensorFlow with version 2.0.0a0 with `pip install tensorflow==2.0.0,` **the code works as expected**!
> 
> I've tried different settings and it works as expected!
> 
>     * PyTorch 1.2.0, TensorFlow(CPU-version) 2.0.0
> 
>     * PyTorch 1.2.0, TensorFlow(GPU-version) 2.0.0
> 
>     * PyTorch 1.3.1, TensorFlow(CPU-version) 2.0.0
> 
>     * PyTorch 1.3.1, TensorFlow(GPU-version) 2.0.0
> 
> 
> > same problem here:
> > ```
> > OS:
> > Python version: 3.6.8
> > PyTorch version: 1.2.0
> > TF version: 2.0.0b1
> > PyTorch Transformers version (or branch): master (Nov 25, 2019)
> > ```
> > 
> > 
> > model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
> > ValueError: Expected floating point type, got <dtype: 'int32'>.
> > looks like a tensorflow problem



I don't know what to say. I've just tested with Transformers v-2.2.0 and it works as expected. Sorry, but in my environment it works as I've said some days ago. How can i help you? How can i replicate your problem? Which OS are you using?

> thanks. I tryied with a0 but it has even more problem, like
> 
> 'tensorflow.python.keras.api._v2.keras.layers' has no attribute 'LayerNormalization'
> 
> which are said to be bugs in a0 and patched in b1.
> 
> However b1 does not work because
> ValueError: Expected floating point type, got <dtype: 'int32'>.
> 
> Looks like this is a deadlock.
> 
> > I've found your problem! It's a TensorFlow bug in version 2.0.0beta1. If you uninstall this version with `pip uninstall tensorflow` and after that you install the TensorFlow with version 2.0.0a0 with `pip install tensorflow==2.0.0,` **the code works as expected**!
> > I've tried different settings and it works as expected!
> > ```
> > * PyTorch 1.2.0, TensorFlow(CPU-version) 2.0.0
> > 
> > * PyTorch 1.2.0, TensorFlow(GPU-version) 2.0.0
> > 
> > * PyTorch 1.3.1, TensorFlow(CPU-version) 2.0.0
> > 
> > * PyTorch 1.3.1, TensorFlow(GPU-version) 2.0.0
> > ```
> > 
> > 
> > > same problem here:
> > > ```
> > > OS:
> > > Python version: 3.6.8
> > > PyTorch version: 1.2.0
> > > TF version: 2.0.0b1
> > > PyTorch Transformers version (or branch): master (Nov 25, 2019)
> > > ```
> > > 
> > > 
> > > model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
> > > ValueError: Expected floating point type, got <dtype: 'int32'>.
> > > looks like a tensorflow problem

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.