Protobuf
Author: chschoenenberger
Pull request: no

## Environment info
- `transformers` version: 4.2.2
- Platform: aws/codebuild/amazonlinux2-x86_64-standard:3.0 AND Windows-10-10.0.17763-SP0
- Python version: 3.8.3 AND 3.8.7
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False


### Who can help
@thomwolf @LysandreJik 

Models:

- T-Systems-onsite/cross-en-de-roberta-sentence-transformer

Packages:
- pipenv
- sentence-transformers

## Information

Model I am using (Bert, XLNet ...): T-Systems-onsite/cross-en-de-roberta-sentence-transformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a new empty project with pipenv
2. Install sentence-transformers
3. Call SentenceTransformer('T-Systems-onsite/cross-en-de-roberta-sentence-transformer')

```
Traceback (most recent call last):
  File "C:/Source/pythonProject/main.py", line 4, in <module>
    SentenceTransformer('T-Systems-onsite/cross-en-de-roberta-sentence-transformer')
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\sentence_transformers\SentenceTransformer.py", line 87, in __init__
    transformer_model = Transformer(model_name_or_path)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\sentence_transformers\models\Transformer.py", line 31, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, cache_dir=cache_dir, **tokenizer_args)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\models\auto\tokenization_auto.py", line 385, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\tokenization_utils_base.py", line 1768, in from_pretrained
    return cls._from_pretrained(
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\tokenization_utils_base.py", line 1841, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\models\xlm_roberta\tokenization_xlm_roberta_fast.py", line 133, in __init__
    super().__init__(
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\tokenization_utils_fast.py", line 89, in __init__
    fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\convert_slow_tokenizer.py", line 659, in convert_slow_tokenizer
    return converter_class(transformer_tokenizer).converted()
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\convert_slow_tokenizer.py", line 301, in __init__
    requires_protobuf(self)
  File "C:\Users\chrs\.virtualenvs\pythonProject-WdXdK-Rq\lib\site-packages\transformers\file_utils.py", line 467, in requires_protobuf
    raise ImportError(PROTOBUF_IMPORT_ERROR.format(name))
ImportError: 
XLMRobertaConverter requires the protobuf library but it was not found in your environment. Checkout the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment.
```

## Expected behavior

Somehow the protobuf dependency doesn't get installed properly with Pipenv and when I try initializing a SentenceTransformer Object with the T-Systems-onsite/cross-en-de-roberta-sentence-transformer it crashes. It can be resolved by manually installing Protobuf. I saw, that it is in your dependencies. This might be a Pipenv or SentenceTransformer issue as well but I thought I would start with you folks.

The error occured on our Cloud instance as well as on my local windows machine. If you think the issue is related to another package please let me know, then I will contact them 😊

Thanks a lot


Comments:
Just to make sure, can you try installing sentencepiece? `pip install sentencepiece`

Pip says
`Requirement already satisfied: sentencepiece in c:\users\chrs\.virtualenvs\pythonproject-wdxdk-rq\lib\site-packages (0.1.95)`
Pipenv "installs it" (I guess it just links it) and writes it to the lock-file. Running the example again I get the same error about Protobuf.

Okay, thank you for trying. Could you show me the steps you did to get this error, seeing as you get the errors on both your cloud instance and your windows machine? I'll try it on my Windows machine and try to reproduce the issue to find out what's happening.

Yeah the steps are as follows:
1. Create a new pipenv environment
2. Install sentence-transformers
3. Create a python file with the following content
`from sentence-transformers import SentenceTransformer`
`SentenceTransformer('T-Systems-onsite/cross-en-de-roberta-sentence-transformer')`
4. Run the python file => Error

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Facing the same issue with T5. Following demo code:

```
from transformers import AutoTokenizer, T5ForConditionalGeneration

model_name = "allenai/unifiedqa-t5-small" 
tokenizer = AutoTokenizer.from_pretrained(model_name)
```


I had the same problem. I had tried many things like here [Link](https://stackoverflow.com/questions/31308812/no-module-named-google-protobuf) but nothing fixed the problem.

With the same environment I worked with the fastai library, which installs quite a few packages. So I created a new environment without fastai and now it works.

name: [NAME]
channels:
  - conda-forge
  - pytorch
dependencies:
  - python=3.8
  - pandas
  - numpy
  - scikit-learn
  - seaborn
  - pytest
  - twine
  - pip
  - ftfy
  - xlrd
  - ipykernel
  - notebook
  - pip
  - pip:
    - azureml-core==1.0.*
    - azureml-sdk==1.0.*
    - pandas==1.0.5
    - numpy==1.17.*
    - fastavro==0.22.*
    - pandavro==1.5.*
    - sentencepiece==0.1.95
	- datasets==1.8.0
	- transformers==4.7.0
	- seqeval==1.2.2
	- tensorflow==2.5.0
	- ipywidgets==7.6.3

As mentioned over [here](https://github.com/huggingface/transformers/issues/9515#issuecomment-869188308), `pip install protobuf` could help. 

This is still a problem.

On an ubuntu cloud instance, I installed in a venv:
```
torch
transformers
pandas
seaborn
jupyter
sentencepiece
protobuf==3.20.1
```

I had to downgrade protobuf to 3.20.x for it to work.

Expected behaviour would be that it works without the need to search the internet to land at this fix.

Thanks @raoulg. I had the same issue working with the pegasus model, actually from an example in huggingface's new book. Downgrading to 3.20.x was the solution.

I didn't have to downgrade, just install a missing `protobuf` (latest version). This can be reproduced in e.g. a `Hugging Face` example for e.g. DONUT document classifier using our latest CUDA 11.8 containers: `mirekphd/cuda-11.8-cudnn8-devel-ubuntu22.04:20230928`. Note that the official `nvidia/cuda/11.8.0-cudnn8-devel-ubuntu22.04` containers seem to come with `protobuf` already preinstalled, so you won't reproduce the bug there).

Perhaps `protobuf` should be added explicitly as a dependency of `transformers`?

I'm still facing the same error. I have fine tuned mistral model, but I'm trying to inference it, it's still giving me:

Could not complete request to HuggingFace API, Status Code: 500, Error: \nLlamaConverter requires the protobuf library but it was not found in your environment. Checkout the instructions on the\ninstallation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones\nthat match your environment. Please note that you may need to restart your runtime after installation.\n

I've done: pip install protobuf, in both env (fine tuning and inferencing)