Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
Author: etetteh
Pull request: no
Labels: name: wontfix

<!-- A clear and concise description of the question. -->

I recently downloaded the [ALBERT_base_v2](https://storage.googleapis.com/albert_models/albert_base_v2.tar.gz) TF pretrained model and converted it to a pytorch with the following code:
`(base) enoch@enoch-pc:~/dl_repos/transformers/src/transformers$ python convert_albert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path '/home/enoch/Documents/experiment/ALBERT_pretrained_models/albert_base_v2/albert_base/model.ckpt-best.index' --albert_config_file '/home/enoch/Documents/experiment/ALBERT_pretrained_models/albert_base_v2/albert_base/albert_config.json' --pytorch_dump_path '/home/enoch/Documents/experiment/albert_base_v2.ckpt'`

However, when I run 

`tokenizer = AlbertTokenizer.from_pretrained('albert_base_v2.ckpt')`

I get 
```
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<timed exec> in <module>

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    307 
    308         """
--> 309         return cls._from_pretrained(*inputs, **kwargs)
    310 
    311     @classmethod

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
    456         # Instantiate tokenizer.
    457         try:
--> 458             tokenizer = cls(*init_inputs, **init_kwargs)
    459         except OSError:
    460             raise OSError(

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_albert.py in __init__(self, vocab_file, do_lower_case, remove_space, keep_accents, bos_token, eos_token, unk_token, sep_token, pad_token, cls_token, mask_token, **kwargs)
    109 
    110         self.sp_model = spm.SentencePieceProcessor()
--> 111         self.sp_model.Load(vocab_file)
    112 
    113     @property

~/anaconda3/lib/python3.7/site-packages/sentencepiece.py in Load(self, filename)
    116 
    117     def Load(self, filename):
--> 118         return _sentencepiece.SentencePieceProcessor_Load(self, filename)
    119 
    120     def LoadOrDie(self, filename):

RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(73) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 
```

Please, how do I use or load the downloaded pretrained models? Thank you.

Comments:
You're trying to load a checkpoint in a tokenizer. Use `AlbertModel` to load the model, not `AlbertTokenizer`.

Okay. Thanks.
These two work okay
model = TFAlbertModel.from_pretrained('albert-base-v2')
model = TFAlbertForSequenceClassification.from_pretrained('albert-base-v2')

because they are being downloaded from the internet. The issue is how to load the one I have downloaded, which is loacted here, experiment/ALBERT_pretrained_models/albert_base_v2.tar.gz, on my system. Please kindly help or provide sample code 

what is in your tar.gz file ?

you should probably untar into a folder and it _should_ just work.

These are the content of the tar.gz:
30k-clean.model  albert_config.json, model.ckpt-best.index, 30k-clean.vocab,  model.ckpt-best.data-00000-of-00001, model.ckpt-best.meta


You would need to convert it using the `convert_albert_original_tf_checkpoint_to_pytorch`, as you did in your first question. You can then load the exported dump using `AlbertModel`.

I converted it using model.ckpt-best.index, albert_config.json and saved it as albert_base_v2.ckpt.
When I run
`model = TFAlbertModel.from_pretrained('albert_base_v2.ckpt')`

I get 
```
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-2-3fd924302d83> in <module>
----> 1 model = TFAlbertModel.from_pretrained('albert_base_v2.ckpt')

~/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    277                 force_download=force_download,
    278                 resume_download=resume_download,
--> 279                 **kwargs,
    280             )
    281         else:

~/anaconda3/lib/python3.7/site-packages/transformers/configuration_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    173 
    174         """
--> 175         config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
    176         return cls.from_dict(config_dict, **kwargs)
    177 

~/anaconda3/lib/python3.7/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    223             if resolved_config_file is None:
    224                 raise EnvironmentError
--> 225             config_dict = cls._dict_from_json_file(resolved_config_file)
    226 
    227         except EnvironmentError:

~/anaconda3/lib/python3.7/site-packages/transformers/configuration_utils.py in _dict_from_json_file(cls, json_file)
    312     def _dict_from_json_file(cls, json_file: str):
    313         with open(json_file, "r", encoding="utf-8") as reader:
--> 314             text = reader.read()
    315         return json.loads(text)
    316 

~/anaconda3/lib/python3.7/codecs.py in decode(self, input, final)
    320         # decode input (taking the buffer into account)
    321         data = self.buffer + input
--> 322         (result, consumed) = self._buffer_decode(data, self.errors, final)
    323         # keep undecoded input until the next call
    324         self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
```

Please, what am I doing wrong?

Does the error still happen if you use `AlbertModel` instead of `TFAlbertModel` ?

> Does the error still happen if you use `AlbertModel` instead of `TFAlbertModel` ?

Yes the error still happens. Same error report.

Please is there a fix for this

On Tue, Jan 28, 2020, 16:49 Lysandre Debut <notifications@github.com> wrote:

> Does the error still happen if you use AlbertModel instead of
> TFAlbertModel ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2667?email_source=notifications&email_token=AGZQ72EECOAB4HT3X7CLBTTRABO2RA5CNFSM4KMUNI72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKEB5WA#issuecomment-579346136>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGZQ72DOAMFVHKA6WBG5XNTRABO2RANCNFSM4KMUNI7Q>
> .
>


This might be far-fetched but did you do any of the conversion process on Windows and now you're trying to load the model on Linux (or the other way around)? That _might_ explain encoding issues. 

> This might be far-fetched but did you do any of the conversion process on Windows and now you're trying to load the model on Linux (or the other way around)? That _might_ explain encoding issues.

No please. I did everything on a Linux machine. In fact, I only use Linux. Will try to find a fix. 

Other far-fetched idea: did you train the model on Python 2 and now try to load it in Python 3 or vice-versa?

> Other far-fetched idea: did you train the model on Python 2 and now try to load it in Python 3 or vice-versa?

I downloaded the ALBERT official pre-trained model. I didn't train anything.

I had no issues loading your checkpoint in both `AlbertModel` and `TFAlbertModel`. Here is what I did:

- Download your file
- Untar `albert_base_v2.tar.gz` into a folder, I called mine `albert_base`
- run the convert command:
```
python convert_albert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path=albert_base/model.ckpt-best --albert_config_file=albert_base=albert_config.json --pytorch_dump_path=albert_base/pytorch_model.bin
```
- When loading a model from a directory, it requires `config.json` and `pytorch_model.bin`, so rename the config:
```
cp albert_base/albert_config.json albert_base/config.json
```
- Now if you `ls` the folder, here are the contents:
```
30k-clean.model  30k-clean.vocab  albert_config.json  config.json  model.ckpt-best.data-00000-of-00001  model.ckpt-best.index  model.ckpt-best.meta  pytorch_model.bin
```

Then, in Python (pytorch):

```py
from transformers import AlbertModel

model = AlbertModel.from_pretrained("albert_base")
```

or in tf:

```py
from transformers import TFAlbertModel

model = TFAlbertModel.from_pretrained("albert_base", from_pt=True)
```

If you didn't train anything, you could have loaded the albert model using the simple command:

```py
from transformers import AlbertModel

model = AlbertModel.from_pretrained("albert-base-v2")
```

Thanks a lot for your help. I tried your fix and it worked.
What I missed from my previous approaches were the .bin file, renaming the
config file and the main thing was I wasn't passing model.ckpt-best but
instead one of the model.ckpt-best (.index).

Another question is, can I pass embeddings from a different pretrained
model?
I'm using a clinical dataset, and I'm wondering if it's possible to learn
embeddings and pass it to the existing AlbertModel.

On Mon, Feb 3, 2020, 21:27 Lysandre Debut <notifications@github.com> wrote:

> I had no issues loading your checkpoint in both AlbertModel and
> TFAlbertModel. Here is what I did:
>
>    - Download your file
>    - Untar albert_base_v2.tar.gz into a folder, I called mine albert_base
>    - run the convert command:
>
> python convert_albert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path=albert_Base/model.ckpt-best --albert_config_file=albert_base=albert_config.json --pytorch_dump_path=albert_base/pytorch_model.bin
>
>
>    - When loading a model from a directory, it requires config.json and
>    pytorch_model.bin, so rename the config:
>
> cp albert_base/albert_config.json albert_base/config.json
>
>
>    - Now if you ls the folder, here are the contents:
>
> 30k-clean.model  30k-clean.vocab  albert_config.json  config.json  model.ckpt-best.data-00000-of-00001  model.ckpt-best.index  model.ckpt-best.meta  pytorch_model.bin
>
> Then, in Python (pytorch):
>
> from transformers import AlbertModel
>
> model = AlbertModel.from_pretrained("albert_base")
>
> or in tf:
>
> from transformers import TFAlbertModel
>
> model = TFAlbertModel.from_pretrained("albert_base", from_pt=True)
>
> If you didn't train anything, you could have loaded the albert model using
> the simple command:
>
> from transformers import AlbertModel
>
> model = AlbertModel.from_pretrained("albert-base-v2")
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2667?email_source=notifications&email_token=AGZQ72AOUYFUP4SJ3IBF6DDRBCD45A5CNFSM4KMUNI72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKVOV4Y#issuecomment-581626611>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGZQ72CSMUWJGYURTSK3IE3RBCD45ANCNFSM4KMUNI7Q>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.