Load of pre-trained t5 model from Tf to HuggingFace
Author: ghost
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hi everybody,

I have the following problem:
I want to convert my custom pre-trained t5 model(trained by following the example provided by Raffel et al on t5 GitHub page), however whenever I try to translate the pre-trained model from tf to pytorch with convert_t5_original_tf_checkpoint_to_pytorch.py script, after the instantiation of it I got the following error: 

INFO:transformers.modeling_t5:Loading TF weight encoder/block_011/layer_001/layer_norm/scale_slot_v with shape [768]
INFO:transformers.modeling_t5:Loading TF weight encoder/final_layer_norm/scale with shape [768]
INFO:transformers.modeling_t5:Loading TF weight encoder/final_layer_norm/scale_slot_v with shape [768]
INFO:transformers.modeling_t5:Loading TF weight global_step with shape []
  <b>File "/Users/antonio/opt/anaconda3/lib/python3.7/site-packages/tensorflow-2.2.0rc2-py3.7-macosx-10.9-x86_64.egg/tensorflow/python/training/py_checkpoint_reader.py", line 70, in get_tensor
    self, compat.as_bytes(tensor_str))
RuntimeError: /Users/antonio/Downloads/tf_path/model.ckpt-306400.data-00000-of-00002; No such file or directory</b>

I can assume that the error is on global_step
One more thing, since I have trained a t5 base model, Is fine to use your json config t5-base with it? or I should create one from scratch for my specific case?

Thanks in advance and have a good day!


Comments:
Hi @antoniomastro1996, 

Can you post your environment information here as written in the "Issue" template. Also are you sure that the path: `/Users/antonio/Downloads/tf_path/model.ckpt-306400.data-00000-of-00002` exist? 

Also can you add the link the Raffel example script? 

Hi @patrickvonplaten this is my environment:

<b>
- `transformers` version: 2.8.0<br>
- Platform: Darwin-19.4.0-x86_64-i386-64bit<br>
- Python version: 3.7.6<br>
- PyTorch version (GPU?): 1.4.0 (False)<br>
- Tensorflow version (GPU?): 2.2.0-rc2 (False)<br>
- Using GPU in script?: no<br>
- Using distributed or parallel set-up in script?: no<br>
</b>
<br>

About the training process, I have adapted the following Colab link: https://tiny.cc/t5-colab
This is the one provided by Raffel et all on their GitHub page.

About the path, yes is fine, it exists. To me seem strange the error manifest itself on this global_step param. Furthermore I have seen the t5_modeling code and this global_step alongside with other variables is in a sort of black list if we perform fine-tuning.

Hi @patrickvonplaten
I've just solved, I changed the pre-trained model and worked fine.

However, if I try to load the model with the following command:
model = T5ForConditionalGeneration.from_pretrained("/content/drive/My Drive/t5_model.bin")

I get the following error: 
<b>UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte</b>

Like I said in the previous comment, I've used this config file
https://s3.amazonaws.com/models.huggingface.co/bert/t5-small-config.json
where I have just removed the part related to the task in such a way  the final output appears like this:

{
  "architectures": [
    "T5WithLMHeadModel"
  ],
  "d_ff": 2048,
  "d_kv": 64,
  "d_model": 512,
  "decoder_start_token_id": 0,
  "dropout_rate": 0.1,
  "eos_token_id": 1,
  "initializer_factor": 1.0,
  "is_encoder_decoder": true,
  "layer_norm_epsilon": 1e-06,
  "model_type": "t5",
  "n_positions": 512,
  "num_heads": 8,
  "num_layers": 6,
  "output_past": true,
  "pad_token_id": 0,
  "relative_attention_num_buckets": 32,
  "vocab_size": 32128
}

Where am I going wrong?

Sorry to answer this late!  Loading the model via:
```python
model = T5ForConditionalGeneration.from_pretrained("/content/drive/My Drive/")
```

should work :-)