TypeError: Descriptors cannot not be created directly. - protobuf version bug
Author: offchan42
Pull request: no

### System Info

- `transformers` version: 4.25.1
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.8.15
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: I have GPU but I didn't run any code other than `import transformers`
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

To reproduce the behavior:

1. install latest version of `protobuf` using `pip install -U protobuf`. I have `4.21.12`
2. run `python -c "import transformers"`
3. you will see the following error

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\off99\anaconda3\lib\site-packages\transformers\__init__.py", line 30, in <module>
    from . import dependency_versions_check
  File "C:\Users\off99\anaconda3\lib\site-packages\transformers\dependency_versions_check.py", line 17, in <module>
    from .utils.versions import require_version, require_version_core
  File "C:\Users\off99\anaconda3\lib\site-packages\transformers\utils\__init__.py", line 34, in <module>
    from .generic import (
  File "C:\Users\off99\anaconda3\lib\site-packages\transformers\utils\generic.py", line 33, in <module>
    import tensorflow as tf
  File "C:\Users\off99\anaconda3\lib\site-packages\tensorflow\__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "C:\Users\off99\anaconda3\lib\site-packages\tensorflow\python\__init__.py", line 37, in <module>
    from tensorflow.python.eager import context
  File "C:\Users\off99\anaconda3\lib\site-packages\tensorflow\python\eager\context.py", line 29, in <module>
    from tensorflow.core.framework import function_pb2
  File "C:\Users\off99\anaconda3\lib\site-packages\tensorflow\core\framework\function_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "C:\Users\off99\anaconda3\lib\site-packages\tensorflow\core\framework\attr_value_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "C:\Users\off99\anaconda3\lib\site-packages\tensorflow\core\framework\tensor_pb2.py", line 16, in <module>
    from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
  File "C:\Users\off99\anaconda3\lib\site-packages\tensorflow\core\framework\resource_handle_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
  File "C:\Users\off99\anaconda3\lib\site-packages\tensorflow\core\framework\tensor_shape_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "C:\Users\off99\anaconda3\lib\site-packages\google\protobuf\descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
    -> <module 'google._upb._message' from 'C:\\Users\\off99\\anaconda3\\lib\\site-packages\\google\\_upb\\_message.cp38-win_amd64.pyd'...
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
```

I don't want to downgrade version as suggested by the error message because other packages depend on latest version of `protobuf` specifically `google-cloud-documentai`. 

### Expected behavior

No error

Please suggest the best course of action.

Comments:
Hi, @off99555 You might need to create a separate environment and then install transformers where you can have a different protobuf version than rest of your system.

@susnato but that is not considered trivial or acceptable, no? https://stackoverflow.com/a/6572017/2593810
Do you mind sharing how you would do it in a way that is not so hacky?

@off99555 I would install Anaconda from [here](https://docs.anaconda.com/anaconda/install/windows/) and then I would create a new environment using `conda create --name venv python=3.9` in terminal  and then activate it using `conda activate venv` , and finally I would install transformers there with the right protobuf version. (Installing anything in the venv will not interfere with you local python installation)
Btw never install anything in your base environment(it is the environment that you get by default after installing anaconda).

I hope it helps.

@susnato but I'm using the `google-cloud-documentai` package in the same project that uses `transformers` package. They both require different versions of `protobuf`. I cannot just install different versions of `protobuf` on different environments because I need both environments to be activated which would cause the version collision issue again.

Hi, @off99555 
I checked and it's actually grpcio-status which is causing the problem, (grpcio-status==1.51.1 requires protobufprotobuf>=4.21.6), you can run `pip install grpcio-status==1.33.2 protobuf==3.19.6` to fix the issue. 
btw, `google-cloud-documentai` will be version 2.7.0 then.

let me know if it worked or not.

I am not sure why you are opening the issue here. It is TensorFlow that has problem with this version of Transformers, not Transformers itself. The issue should be raised in the TensorFlow repository. We have just pinned protobuf to avoid this problem.

> I am not sure why you are opening the issue here. It is TensorFlow that has problem with this version of Transformers, not Transformers itself. The issue should be raised in the TensorFlow repository. We have just pinned protobuf to avoid this problem.

It's because I don't know exactly what is the cause. I was confused. All I know is that I run `import transformers` and it gave me the error so I open the issue here. 

Thanks for pointing out that it's related to tensorflow. So the conflict is simply that `tensorflow` uses old version of `protobuf` whereas `grpcio-status` is using the latest `protobuf` version.

So I just downgraded `grpcio-status` according to what @susnato suggests and it seems to resolve the conflict. Thank you!

Just wanted to flag that maybe the protobuf version in this repo _should_ be updated – I'm trying to write a gRPC service and am currently getting around this issue by passing `use_fast=False` to my HF pipeline. I can't downgrade my `grpcio-tools` package seeing the last version to not support Protobuf v3 was released in 2020, and would prefer to not have my dependencies _that_ out-of-date. So I believe using such an old PB version will impact anyone trying to write microservices using HF, not just the transitive dependencies of other packages like Tensorflow.

@Nickersoft We sadly cannot remove the pin on `protobuf` until TensorFlow starts supporting version 4.0. Looks like TensorFlow 2.12 will solve this, so we just have to wait for it to be out.

In the meantime, you can always remove the pin in a source install.

@sgugger also wanted to flag that unpinning protobuf might not be sufficient to fully resolve this issue. [This](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/sentencepiece_model_pb2.py) file will need to be regenerated with newer version of protobuf as well (not sure if there are other such files in transformers code base).

cc @Narsil for this. Could you open a new issue specific to this @yigor and re-ping me and Narsil there? This way we'll be sure this does not slip through the crack.

The new sentence piece proto generated file is this : https://github.com/google/sentencepiece/blob/master/python/src/sentencepiece/sentencepiece_model_pb2.py

My specific version of this error only relates to the  "google/pegasus-cnn_dailymail" model.     The second fix suggested in the error message worked for me - I just added
```
import os
os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"]="python"
```
at the head of my Jupyter Notebook

You can risk in your base installment and downgrade that lib with:
pip install protobuf==3.20.*
...... in Anaconda prompt, with admin privileges (if on Windows first disable win defender and any other anti-viruses)

AttributeError: module 'google.protobuf.message_factory' has no attribute 'GetMessageClass'