Mitigate a conflict when using sentencepiece
Author: tengomucho
Pull request: yes

# What does this PR do?

This is a fix to a conflict between the fast tokenizers usage and sentencepiece module.
This is due to the fact that protobuf C implementation uses a global pool for all added descriptors, so if two different files add descriptors, they will end up conflicting.

I added a test showing the problem and a guard when loading the proto file to mitigate the problem. Note that the problem is not completly removed: if for any obscure reason an invalid sentencepiece proto file was loaded before, it will keep on using that. Also, if sentencepiece was loaded _after_ the tokenizers load the proto file, the error will occur again (but at least there will be a way to avoid it).


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Original discussion in internal slack: https://huggingface.slack.com/archives/C014N4749J9/p1725371896664959
- [x] Did you write any new necessary tests?


## Who can review?

@ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33327). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey! Could you provide the reproducer as well? 
I really don't mind merging as is, but just want to make sure, you 1st have a project that imported protobuf, and 2. used `from transformers.convert_slow_tokenizer import import_protobuf` and that failed right? 
What is a bit weird is we have this guard:

```python
def import_protobuf(error_message=""):
    if is_protobuf_available():
        import google.protobuf

        if version.parse(google.protobuf.__version__) < version.parse("4.0.0"):
            from transformers.utils import sentencepiece_model_pb2
        else:
            from transformers.utils import sentencepiece_model_pb2_new as sentencepiece_model_pb2
        return sentencepiece_model_pb2
    else:
        raise ImportError(PROTOBUF_IMPORT_ERROR.format(error_message))
```
is there no way to check if it was not already imported? (meaning try important, if the error is import error, we expect ? 

Hello! I saw first this problem because I tried to use [Jetstream/Pytorch](https://github.com/google/jetstream-pytorch), and they use sentencepiece and its [`sentencepiece_pb.py`](https://github.com/google/sentencepiece/blob/master/python/src/sentencepiece/sentencepiece_model_pb2.py). In transformers there is [a copy](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/sentencepiece_model_pb2_new.py) of the same file, with few changes to make some variables globals. So, while in python world they are technically two different modules, the C protobuf extension shares the same space, and they conflict when calling `AddSerializedFile`. That's why I proposed this option, because I didn't find any other way to figure out how to avoid this conflict.

I was just thinking now that an alternative could be to check if the `sentencepiece` module is available, import the `sentencepiece_pb.py` from that package, and avoid conflicts. We could then delete our "sentencepiece_pb_new.py" file. But that means it will only work when the `sentencepiece` module is available.

Let me know if you prefer this option and I can modify this PR accordingly.

I force-pushed a cleaner solution, I check if `sentencepiece` is available and if it's the case, we use that instead of our proto version, so we avoid conflicts.