HuggingFace/BigBird RuntimeError: Internal: src/sentencepiece_processor.cc
Author: jtfields
Pull request: no
Labels: name: bug

### System Info

```shell
I'm able run the HuggingFace/BigBird code for a binary classification on a proprietary essay dataset in Google Colab with no errors.  I wanted to access more powerful GPU's and converted the code from .ipynb to .py to run on Marquette's supercomputer (called Raj).  Raj does not allow me to access the roberta model remotely so I changed the first line of code below to the second for local access (and also copied the bigbird-roberta-base files to Raj):

tokenizer = BigBirdTokenizer.from_pretrained('google/bigbird-roberta-base')
tokenizer = BigBirdTokenizer.from_pretrained('<my user path on Raj>/bigbird-roberta-base')

However, this gives me the following error:
RuntimeError: Internal: src/sentencepiece_processor.cc(890) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 

I did confirm that sentencepiece 0.1.96 is installed and I'm using Python version 3.6.8.

Any help or suggestions is appreciated!
```


### Who can help?

@ydshieh, @SaulLu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

from transformers import BigBirdTokenizer, BigBirdForSequenceClassification
print('Loading tokenizer...')
tokenizer = BigBirdTokenizer.from_pretrained(<my user path>/bigbird-roberta-base')
# Tokenize all of the sentences and map the tokens to thier word IDs.
input_ids = []
# Record the length of each sequence (in terms of BERT tokens).
lengths = []
print('Tokenizing comments...')
# For every sentence...
for sen in train.data:
    # Report progress.
    if ((len(input_ids) % 1000) == 0):
        print('  Read {:,} comments.'.format(len(input_ids)))
    # `encode` will:
    #   (1) Tokenize the sentence.
    #   (2) Prepend the `[CLS]` token to the start.
    #   (3) Append the `[SEP]` token to the end.
    #   (4) Map tokens to their IDs.
    encoded_sent = tokenizer.encode(
                        str(sen), # Sentence to encode.  Added str due to error.
                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'
                   )
    # Add the encoded sentence to the list.
    input_ids.append(encoded_sent)
    # Record the non-truncated length.
    lengths.append(len(encoded_sent))
print('DONE.')

print('   Min length: {:,} tokens'.format(min(lengths)))
print('   Max length: {:,} tokens'.format(max(lengths)))
print('Median length: {:,} tokens'.format(np.median(lengths)))

### Expected behavior

```shell
The first print statement should generate:
Tokenizing comments...
  Read 0 comments.
DONE.
       454 comments

The second group of three print statements should generate:
Min length: 90 tokens
   Max length: 995 tokens
Median length: 826.5 tokens
```


Comments:
I'm not too sure what's the root cause of this, but I've created a [google colab](https://colab.research.google.com/drive/1x12Bc6aDU9sLOCI99bGKXh9zUQzKDcaR?usp=sharing) reproducing the bug - it seems like the vocab file path is not being passed in properly when it's not in the VOCAB_FILE_NAME mapping as well as a potential workaround. 

@jtfields do you think the workaround could work? Instantize your BigBirdTokenizer outside of Raj in your local, then save the pretrained tokenizer into a directory and copy the files into a directory on Raj, then instantize from that directory instead and but with an AutoTokenizer (cells 7-9).

Thank you for the fast response on this bug.  I'm trying the workaround you provided but finding that Spacy is not very cooperative due to the length of my essays.  I first had to develop a workaround for the max_length of 100,000 and now have a handle_filename_too_long error.  Is there another option besides spacy which isn't so restrictive?

I was able to tokenize the essays in Google Colab and copy these files to Marquette's Raj supercomputer.  However, this bug should stay open until a fix is available to run the tokenization files locally.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.