Use protobuf 4
Author: ydshieh
Pull request: yes

# What does this PR do?

I move forward to generate the new `src/transformers/utils/sentencepiece_model_pb2.py` using the protocol buffer compiler. No test failure but have a quality check fails. I can probably removed the undefined part `_TRAINERSPEC` which doesn't seem used.

Comments:
I have checked to make sure `protobuf==4.23.3` is installed and used in the CI.

@Narsil mentioned we should keep the old/new version files, and determine which one to use (by the protobuf version numbers?).

Let me know more details about this please.

_The documentation is not available anymore as the PR was closed or merged._

Will merge once the CI for protobuf 4 a nd protobuf 3 both green.

Don't hesitate to leave comments later if any @Narsil 🙏 

Great PR. I hope this doesn't have any unforeseen consequence (I don't know what are the breaking changes between protobuf 3 and 4)

> I don't know what are the breaking changes between protobuf 3 and 4

Yeah, me neither. I just rely on our CI 🤞 

@ydshieh I'm now getting

```
  File "/home/fxmarty/hf_internship/transformers/src/transformers/utils/sentencepiece_model_pb2_new.py", line 16, in <module>
    DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
TypeError: Couldn't build proto file into descriptor pool: Invalid default '0.9995' for field sentencepiece.TrainerSpec.character_coverage of type 2
```

with protobuf=4.23.4 & transformers main when doing `from transformers.utils import sentencepiece_model_pb2_new`. Any idea what's wrong? protobuf 3.20.3 works well for me.

Hi @fxmarty !

Hmm, I remember I got `4.23.3` when I made this PR. Not sure if it's the reason. Let me check.

Hi again

~~@fxmarty `4.23.4` works for me.~~




```
(py39) λ pip show protobuf
Name: protobuf
Version: 4.23.4

Python 3.9.13 (main, Oct 13 2022, 21:23:06) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers.utils import sentencepiece_model_pb2
>>> from transformers.utils import sentencepiece_model_pb2_new
>>>
```

Hm, there is something wrong and I am looking deeper.

OK, PR #24622 makes this failing.

cc @ArthurZucker 



Comes from #24690. More details: 
This does not fix the version error, but fixes the issue with 3.20.3, when we cannot use seqio or anything importing protobuf: 
```python 
! pip install protobuf=3.20.3
from transformers import AutoTokenizer
from seqio import SentencePieceVocabulary
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "sentencepiece_model.proto":
  sentencepiece_model.proto: A file with this name is already in the pool.
```

@fxmarty We are not able to reproduce the situation you have. Could you specify your transformers commit that is installed? Thanks.

(The above discussion is another story)

@ydshieh It appears I can't reproduce anymore the issue I had. Must have messed something up in my install that I fixed since then. Maybe the `pip uninstall protobuf && pip install --no-binary protobuf protobuf==3.20.3` that I used in the meanwhile helped to fix things.

@ydshieh Oh, actually I have this issue only running in my scripting IDE pyzo, but not in a terminal. The same python env is used so quite weird.

@ArthurZucker @ydshieh How does `sentencepiece_model_pb2_new.py` define `TrainerSpec.character_coverage`?

Specifically, how is https://github.com/huggingface/transformers/blob/33aafc26ee68df65c7d9457259fc3d59f79eef4f/src/transformers/utils/sentencepiece_model_pb2_new.py#L17 generated? If I use `decode()` on it python complains that `'utf-8' codec can't decode byte 0x80 in position 43`.

Hi @fxmarty 

That file is being generated by protobuf compile. I don't have the courage to read it ...

When I enable the support to use protobuf v4, I ran the whole CI (non-slow), and no failed test.

Could you show us your usage (related to protobuf) that produce some failures - maybe I can help ?