Missing `vocab_file` Attribute When Using Custom SentencePiece Models
Author: teleprint-me
Pull request: no
Labels: name: Feature request

name: Good Difficult Issue

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.1.69-1-lts-x86_64-with-glibc2.38
- Python version: 3.11.6
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Steps to Reproduce

1. Import the SentencePiece library:

   ```python
   import sentencepiece as spm
   ```

2. Verify the SentencePiece version:

   ```python
   spm.__version__  # Ensure it is 0.1.98 or later
   ```

3. Attempt to create a `LlamaConverter` instance with a loaded SentencePiece tokenizer:

   ```python
   from transformers.convert_slow_tokenizer import LlamaConverter

   # Define the path to your SentencePiece model
   spm_model_path = "path/to/your/tokenizer.model"

   # Load the SentencePiece tokenizer
   spm_tokenizer = spm.SentencePieceProcessor()
   spm_tokenizer.Load(spm_model_path)

   # Create a LlamaConverter instance with the loaded tokenizer
   llama_converter = LlamaConverter(spm_tokenizer)
   ```

4. Observe the error raised due to the missing `vocab_file` attribute.

### Expected Behavior

To address this issue and make it easier for users to work with custom SentencePiece models, the following modifications are proposed:

1. **Modify `Converter` Class**:
   - Update the `Converter` class to accept a file path to the SentencePiece model when initializing an instance of the class.
   - Instantiate the SentencePiece tokenizer and load the model from the provided file path within the `Converter` class.

   ```python
   import sentencepiece as spm

   class Converter:
       def __init__(self, file_path_to_tokenizer):
           self.file_path = file_path_to_tokenizer
           self.original_tokenizer = spm.SentencePieceProcessor()
           self.original_tokenizer.Load(self.file_path)

       def converted(self) -> Tokenizer:
           raise NotImplementedError()
   ```

2. **Update `SpmConverter` Class**:
   - Update the `SpmConverter` class to use the `file_path` attribute instead of the non-existent `model_file` attribute when loading the SentencePiece model.
   - Ensure compatibility with the modifications made in the `Converter` class.

   ```python
   from transformers.convert_slow_tokenizer import Converter

   class SpmConverter(Converter):
       def __init__(self, file_path_to_tokenizer):
           requires_backends(self, "protobuf")
           super().__init__(file_path_to_tokenizer)

           # Load the SentencePiece model using the file path
           with open(self.file_path, "rb") as f:
               m.ParseFromString(f.read())
   ```

### Expected Outcome

With these proposed modifications, users should be able to create `Converter` instances with their custom SentencePiece models by providing the file path, and the `SpmConverter` class should correctly load the model using the specified file path, resolving the issue of missing attributes and ensuring compatibility with Transformers.

Comments:
Thanks for opening this! 
The conversion scripts should not always rely on sentencepiece as some models do not use it. ( Bert for example)
However it is true that supporting a kind of direct conversion could / would help. 

Would you like to open a PR to help using the converter class for converting spm models to fast tokenizers ? 

Sure, I'll start looking into this today. I appreciate your suggestion on the PR, and I'll consider how to integrate these changes without enforcing SentencePiece dependency where not necessary. Looking forward to any further feedback or suggestions.

@ArthurZucker I think it's important to note that this is not a feature request. It's a bug. `SentencePieceProcessor` does not have a `vocab_file` attribute. Any source code that relies on this will fail as a result. I'm only familiar with the newer version(s) of `sentencepiece`, so I'm not sure if this was the case in previous versions.

```python
>>> fast_tokenizer = llama_converter.converted()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    fast_tokenizer = llama_converter.converted()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/valerie/transformers/src/transformers/convert_slow_tokenizer.py", line 530, in converted
    tokenizer = self.tokenizer(self.proto)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/valerie/transformers/src/transformers/convert_slow_tokenizer.py", line 1181, in tokenizer
    _, merges = SentencePieceExtractor(self.original_tokenizer.vocab_file).extract(vocab_scores)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'SentencePieceProcessor' object has no attribute 'vocab_file'
```

Got it, it's more of a bug on our side I think. We need to update the converter. 

Would you like to open a PR? I am a bit low on bandwidth for this currently 

I unfortunately do not have the time or resources to contribute at the moment. I'm spread thin as it is.

I did look into it and it's unfortunately not a simple fix. 

There's probably a workaround or a hack that could be implemented, but the entire design is assuming a certain pattern that doesn't exist.

I was looking into this when I had time to work on the convert.py for llama.cpp and was attempting to figure out how to support the vocab for huggingface fast tokenizers.

After studying the tokenizers source code, I saw that the implementation is based on the factory pattern which seems to be extended by a bridge or facade pattern. This isn't normally an issue, but there are multiple steps in the pipeline that coerce the user to remotely access the vocabulary.

That's when I dug a bit deeper and learned about the conversion and tokenizer scripts which don't seem to be documented very well or at all.

I'm working on other personal projects at the moment which have a higher priority for me. I'm more than happy to bounce ideas off one another. I'm not sure when I'll be able to dedicate more time into redesigning/fixing the implementation though.

These are just some of my observations and notes I made throughout the process. 

Hey @teleprint-me, sorry for the very slow reply here, we're catching up on the backlog.

The root cause is that `LlamaConverter` (and the other classes in `convert_slow_tokenizer.py`) are internal helpers that expect a *transformers* tokenizer instance, which carries a `.vocab_file` attribute, see https://github.com/huggingface/transformers/blob/150eb7c9ed4091294c829fa0e9466b090cb0f87f/src/transformers/convert_slow_tokenizer.py#L697-L708
Passing a raw `spm.SentencePieceProcessor` was never the supported entry point, which is why you hit the `AttributeError`, so this isn't really a converter bug.

What you actually want (building a fast tokenizer straight from a bare `.model` file) is now supported natively: put your `tokenizer.model` in a directory and call `AutoTokenizer.from_pretrained(<that dir>)`; I verified this returns a working full-vocab (32k) fast tokenizer on `hf-internal-testing/llama-tokenizer`. The whole slow-tokenizer/conversion stack was also rewritten since you filed this (#40936), and there's now an explicit `convert_from_spm` hook for exactly this direct-from-SentencePiece path.