Importing horovod.tensorflow crashes AlbertTokenizer but not BertTokenizer
Author: jarednielsen
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Albert tokenizers began crashing after I reorded my import statements with `isort`. Tracked down the bug to very strange behavior: importing `horovod.tensorflow` before `AlbertTokenizer` causes a crash, while importing `AlbertTokenizer` first does not. This behavior does not occur with `BertTokenizer`, only with `AlbertTokenizer`.

## To reproduce

Steps to reproduce the behavior:

```bash
docker run -it nvcr.io/nvidia/tensorflow:20.03-tf2-py3 /bin/bash # TF 2.1, horovod 0.19.0
pip install transformers==2.8.0
```

```python
import horovod.tensorflow as hvd
from transformers import AlbertTokenizer, BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
print("BERT success!") # this succeeds
tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
print("ALBERT success!") # this causes a CoreDump
```
outputs
```error
BERT success!
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/stubs/common.cc:86] This program was compiled against version 3.6.1 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.8.0).  Contact the program author
for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "/sentencepiece/src/builtin_pb/sentencepiece_model.pb.cc".)
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  This program was compiled against version 3.6.1 of the Protocol Buffer runtime library, which is not compatible with the
installed version (3.8.0).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "/sentencepiece/src/builtin_pb/sentencepiece_model.pb.cc".)
Aborted (core dumped)
```

However, the code below succeeds. The only difference is that the transformers import comes first:
```python
from transformers import AlbertTokenizer, BertTokenizer
import horovod.tensorflow as hvd

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
print("BERT success!") # this succeeds
tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
print("ALBERT success!") # this succeeds
```

This bug is a bit bewildering, to be honest. I can stop sorting my imports, I guess... Hoping that someone can identify the root cause.


Comments:
I think I remember someone mentioning this before. @LysandreJik does it ring any bell?

The error at the time was due to https://github.com/scipy/scipy/issues/11237. I'll look into it and try to reproduce @jarednielsen.

The `AlbertTokenizer` is using `SentencePiece` which is based on protobuffs. This seems to be the error, which would point to an error with `SentencePiece` rather than with `AlbertTokenizer`. Would you mind trying to import `XLNetTokenizer`, which is also based on `SentencePiece` and show us the results?

Same issue occurs with `XLNetTokenizer`. Would resolving https://github.com/huggingface/tokenizers/issues/53 enable us to move away from protobuf and fix this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Don't think this is stale; still waiting on a fix in the tokenizers repo.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Not stale

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.