[BUG] Protobuf not being correctly installed
Author: yinweisu
Pull request: no

### System Info

`transformers==4.30.2`
Protobuf not being installed along with transformers even when it's specified in `setup.py`.
This can be an issue if the environment already installs latest version of protobuf, which is not compatible with tranformers

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

pip install transformers -U
pip freeze | grep protobuf

### Expected behavior

protobuf with correct pinned version should be installed

Comments:
Transformers is only compatible with protobuf<4.0 because packages we depend on (sentencepiece if I recall correctly) do not support protobuf 4 yet. There is little we can do on our side until they support it.

cc @ydshieh in case I said something wrong.

Hi @yinweisu 

You can uninstall `protobuf` and reinstall it to get `3.20.3`.

Regarding protobuf 4, unfortunately, we get errors (see below) for sentencepiece based tokenizers. I am also told that `google/sentencepiece` doesn't support protobuf 4 yet, but from its GitHub page, I can't see this information. I plan to open an issue to ask questions there.


```bash
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
```

@yinweisu 

Maybe you want/could/love to give [this issue](https://github.com/google/sentencepiece/issues/889) a 👍 🤗 😆 

Thanks! I know you can reinstall it. The issue is that we are an open source project, which depends on transformers and other libraries. Other libraries will install newer version of protobuf, which doesn't work with transformers. This means users consume our package will run into protobuf errors. Ideally, transformers should correctly pin protobuf and install the correct version while doing `pip install`. The fact is that, transformers does pin it in the setup.py, though it's not taking effect

> Transformers is only compatible with protobuf<4.0 because packages we depend on (sentencepiece if I recall correctly) do not support protobuf 4 yet. There is little we can do on our side until they support it.
> 
> cc @ydshieh in case I said something wrong.

@sgugger I understand it doesn't work with protobuf>4.0. But transformers should correctly install protobuf < 4.0 as its dependency, which is not what's happening right now

@yinweisu 

I probably need to check what happens if a higher version is already installed when a user installs `transformers`.

Could you show us your way of installing `transformers`? 🙏  Thanks.

It will just use the higher version. We've got this problem in our own package installation.

You should be easily reproduce it via
```bash
pip install -U protobuf
pip install transformers
pip freeze | grep protobuf
```

@yinweisu After looking `setup.py`, at the end in that file, the `install_requires` only contains a few libraries:

https://github.com/huggingface/transformers/blob/66954ea25e342fd451c26ec1c295da0b8692086b/setup.py#L415-L426

and `protobuf` is not in this list. It's why `protobuf` is not checked and updated in the case you mentioned.

From README:

> Then, you will need to install at least one of Flax, PyTorch or TensorFlow.

I believe `install_requires` is set to be minimal as it is in current version. But @sgugger knows better than me regarding this and may have some comments.



protobuf is not a core dependency of Transformers, it is only installed through other packages (most likely sentencepiece). I don't think any of the packages here have a dependency on protobuf but I may be msitaken.

https://github.com/huggingface/transformers/blob/main/setup.py#L145
Protobuf is listed here. So if the library requires a specific version of dependency to work, it should be installed along, right? 

This list only contains soft dependencies. This one is used when you do `pip install transformers[sentencepiece]` for instance.

So I only install transformers, but I can still run into this problem if protobuf is either already presented in my env or being installed along with other packages. If transformers requires a specific version of protobuf to function, it should pin that version and install it.

As mentioned in 

> You should install 🤗 Transformers in a (new) virtual environment.

And you can simply install as `pip install transformers[dev]` or something smaller `pip install transformers["dev-torch"]`.

@ydshieh I really don't want to argue with you and I appreciate your fast response, but imagine such an use case:

Our package AutoGluon, depends on `transformers` and package `foo`, both of which are in our setup.py.
package `foo` has `protobuf` in its dependency as `protobuf>=3.15.3` let's say while transformer didn't pin `protobuf`

When user creates a new virtual environment and do a pip install autogluon, it will install `foo` and `transformers,` which will then install the latest `protobuf` because the latest `protobuf` satisfy `protobuf>=3.15.3`

You see the problem here? It's not about if it's a fresh env or not.

The only way to solve this as AutoGluon for now is that we had to manually pin `protobuf` by looking at both requirements from `transformers `and `foo` and figure out what's the version that satisfy both even if its not a direct dependency for us.  This is not maintainable as we need to do this everytime `foo` or `transformers` updated their dependency on `protobuf`. Then imagine what if there are multiple packages in our dependency that requires `protobuf`. If `transformer` pin `protobuf`, pip will auto-resolve the version

>  if its not a direct dependency for us.

It's not a direct dependency for `transformers` neither. It's only for the usage of the tokenizers based on `sentencepiece`.

It's not practical for us to put all packages in the `setup.py` as hard dependency.

> imagine what if there are multiple packages in our dependency that requires protobuf. 

Let's not overthink and just make your package `AutoGluon` work for now. Maintenance is not an easy task, and if there has something to be updated in the future, that's it.

@yinweisu 

FYI: after #24599, we no longer pin `protobuf` in `transformers`.