[Albert] SentencePiece Error with AlbertTokenizer
Author: jmwoloso
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):Albert v2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: Sequence Classification

## To Reproduce

Steps to reproduce the behavior:

```
from transformers import AlbertTokenizer
from pyspark.sql import functions as F, types as T

tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")

# load data into spark
df = spark.read...

# df.columns =>  ["id", "text"]

# create a function to create the tokens from a supplied text column
def create_tokens(text=None, tokenizer=None):
    tokens = ["[CLS]"]
    tokens.extend(tokenizer.tokenize(text))
    tokens.append("[SEP]")
    return tokens

def create_tokens_udf = F.udf(lambda z: create_tokens(z, tokenizer=tokenizer), T.ArrayType(T.StringType()))

# apply the udf to the text
tokenized_df = df.withColumn("tokens", create_tokens_udf(F.column("text")))

# trigger the transformation
tokenized_df.cache().count()
```

The following traceback is observed (i've excluded the Py4J tracebacks for clarity):

```
File "<command-3594705570096092>", line 10, in create_tokens
  File "/databricks/python/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 302, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 438, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/tokenization_albert.py", line 90, in __init__
    self.sp_model.Load(vocab_file)
  File "/databricks/python/lib/python3.7/site-packages/sentencepiece.py", line 118, in Load
    return _sentencepiece.SentencePieceProcessor_Load(self, filename)
RuntimeError: Internal: unk is not defined.

```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

The text gets tokenized as expected.

## Environment

* OS: Linux(?)
* Python version: 3.7.3
* PyTorch version: ?
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU N/A
* Distributed or parallel setup: Distributed (Databricks)
* Any other relevant information: 

## Additional context

Thanks a million for this amazing library!

Comments:
Actually, looking more closely, this seems to be a `sentencepiece` issue, right?

Hi! Is there a way you could isolate where the error happens? Is it when you're initializing the tokenizer with the line `tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")` ?

Hi! No initialization seems to work fine, it is when I actually attempt to apply the tokenizer in the `create_tokens` function. so:

`tokens.extend(tokenizer.tokenize(text))`

I also have the same issues for fine-tuning model on my own task:
```python
INFO:tensorflow:loading sentence piece model
I0110 02:56:05.196459 139883558270784 tokenization.py:240] loading sentence piece model
Traceback (most recent call last):
  File "run_classifier.py", line 494, in <module>
    tf.app.run()
  File "/root/anaconda3/envs/jupyterlab/lib/python3.7/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/root/anaconda3/envs/jupyterlab/lib/python3.7/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/root/anaconda3/envs/jupyterlab/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "run_classifier.py", line 204, in main
    spm_model_file=FLAGS.spm_model_file)
  File "/home/vigosser/ALBERT/tokenization.py", line 254, in from_scratch
    return FullTokenizer(vocab_file, do_lower_case, spm_model_file)
  File "/home/vigosser/ALBERT/tokenization.py", line 241, in __init__
    self.sp_model.Load(spm_model_file)
  File "/root/anaconda3/envs/jupyterlab/lib/python3.7/site-packages/sentencepiece.py", line 118, in Load
    return _sentencepiece.SentencePieceProcessor_Load(self, filename)
RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(73) [model_proto->ParseFromArray(serialized.data(), serialized.size())]
```

@vigosser I believe this is an issue with SentencePiece itself rather than Transformers. I was looking at the repo for SentencePiece and it is a little confusing; according to this [issue](https://github.com/google/sentencepiece/issues/344) it seems that we shouldn't be using SentencePiece as of this past summer, but instead should use tf.text, but according to that same issue, the integration is not complete.

I also have a feeling this is broken due to something in the TF 2.0 API, but that's not based on anything in particular.

Thoughts @LysandreJik ?

@jmwoloso  this problem happened because of the wrong "spm_model_file"
the command as follow slove the problem
```bash
python run_classifier.py \
                                --task_name=mail \
                                --do_predict=true  \
                                --do_train=true    \
                                --do_eval=true \
                                --spm_model_file=$modelpath/30k-clean.model \
                                --data_dir=/data \
                                --vocab_file=$modelpath/30k-clean.vocab  \
                                --albert_config_file=$modelpath/albert_config.json  \
                                --init_checkpoint=$modelpath/model.ckpt-best.index \
                                --max_seq_length=128  \
                                --train_batch_size=8 \
                                --output_dir=/data/output \
                                --learning_rate=15e-6  \
                                --num_train_epochs=3.0  \

```

Glad you found a solution to your issue @vigosser! My issue is different than your though (at least I think it is). I'm using Albert from within my own custom script and just trying to tokenize some text so that I can train on it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@jmwoloso Did you get around to solve your problem? 
I have a similar issue when using my trained sentencepiece tokenizer with Albert to train my corpus.