How can I generate sentencepiece file or vocabulary from tokenizers?
Author: darwinharianto
Pull request: no

After I make custom tokenizer using Tokenizers library, I could load it into XLNetTokenizerFast using
```
tokenizer = Tokenizer.from_file("unigram.json")
tok = XLNetTokenizerFast(tokenizer_object=tokenizer)
```

After I called
```
tok .save_vocabulary("ss")
```

it throws an error since I didnt load XLNetTokenizerFast using spm file. I believe save_vocabulary is looking for vocab_file parameter.

Is there any way to save_vocab after loading it from XLNetTokenizer?



Comments:
Could you try `save_pretrained` instead?

It throws
```
Traceback (most recent call last):
  File "~/pretrain.py", line 26, in <module>
    tokenizer.save_pretrained('./test_Spm')
  File ~/anaconda3/envs/deltalake/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1958, in save_pretrained
    save_files = self._save_pretrained(
  File "~/anaconda3/envs/deltalake/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 555, in _save_pretrained
    vocab_files = self.save_vocabulary(save_directory, filename_prefix=filename_prefix)
  File "~/anaconda3/envs/deltalake/lib/python3.9/site-packages/transformers/models/xlnet/tokenization_xlnet_fast.py", line 232, in save_vocabulary
    if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
  File "~/anaconda3/envs/deltalake/lib/python3.9/posixpath.py", line 374, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

It looked for vocab file (spm), but since I initiated it with tokenizer_object, there is no vocab_file

Could you share your `unigram.json` file or mention how you obtained it so that I can reproduce the issue? Thank you!

Here it is

[unigram.json.zip](https://github.com/huggingface/transformers/files/6839324/unigram.json.zip)


Edit:

```
from transformers import AutoTokenizer, XLNetTokenizerFast

tokenizer = AutoTokenizer.from_pretrained('xlnet-base-cased')
print(tokenizer.save_pretrained("./dump"))
tokenizer = XLNetTokenizerFast(tokenizer_file='./dump/tokenizer.json')
print(tokenizer.save_pretrained("./dump"))
```
Above Throws an error

```

from transformers import AutoTokenizer, XLNetTokenizerFast, BertTokenizerFast

tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
print(tokenizer.save_pretrained("./dump"))
tokenizer = BertTokenizerFast(tokenizer_file='./dump/tokenizer.json')
print(tokenizer.save_pretrained("./dump"))

```
Above does not throw an error

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Sorry, is there any progress on this?

@SaulLu can you give this one a look?

Thank you very much for reporting this problem @darwinharianto . 

I have the impression that it linked to the problem reported in [this issue](https://github.com/huggingface/transformers/issues/12762) and that I had started to work on in this [PR](https://github.com/huggingface/transformers/pull/12806). As the cleanest looking fix requires quite a bit of work, I had put it on hold. I'll try to work on it again at the beginning of the week.

@darwinharianto, to answer the question in the issue title, at the moment it is not possible to transform a fast tokenizer initialized only with a `tokenizer_object` into an spm file compatible with the `SentencePiece` library. The conversion is only supported in the opposite direction.

Why did you want the vocabulary? Because the vocabulary can be found in the `tokenizer.json` file or by doing: 
```
tok.get_vocab()
```


Since I tried to make my own XLNet tokenizer, i wanted to check if the saved format for vocabulary is the same as the published models.
I thought the fastest way would be comparing saved vocab file from huggingface model hub with mine.
Just for sanity check.

Duly noted! If it's to do a sanity check, would it be ok to compare the files of the fast version of the tokenizers (in particular the `tokenizer.json` file)? 

To retrieve these files for the new tokenizer `tok` you made, the following command should work: 
```
tok.save_pretrained("./dump", legacy_format=False)
```

For information, the vocabulary will be visible in the `tokenizer.json` file.

Thanks! it works.

One more question, I can see this
```
                "type": "Precompiled",
                "precompiled_charsmap": 
```
under normalizers, when I tried to save a pretrained tokenizer.

My custom tokenizer doesn't have this attribute. Is this normal?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I'm sorry, I realize that I never answered your last question. 

This type of `Precompiled` normalizer is only used to recover the normalization operation which would be contained in a file generated by the sentencepiece library. If you have ever created your tokenizer with the tokenizers library it is perfectly normal that you do not have this type of normalization. Nevertheless, if you want to have an equivalent normalization for your tokenizer, it is generally possible to build it with the tokenizers library but it requires to know exactly which normalizations you want to apply. :slightly_smiling_face: 

> @darwinharianto, to answer the question in the issue title, at the moment it is not possible to transform a fast tokenizer initialized only with a `tokenizer_object` into an spm file compatible with the `SentencePiece` library. The conversion is only supported in the opposite direction.
> 
> Why did you want the vocabulary? Because the vocabulary can be found in the `tokenizer.json` file or by doing:
> 
> ```
> tok.get_vocab()
> ```

Hi @SaulLu,

Sorry to open this old thread, I noticed that you mentioned transferring from spm tokenizer to a huggingface one is easy, but I could not find any function which does that for me. I would be grateful if you could share any piece of code to help me with that.
(Just to give you a quick background, I trained a SPM tokenizer and would like to use it in huggingface, but I have .vocab and .model for it, and huggingface expect a .json file) 

Thank you,
Soheila

Hello @SoheilaSamiee, I'm encountering a similar problem. If you've found a solution, could you please share it with me? Thank you!

Hi, @nandinimundra, questions like these are best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

cc @ArthurZucker 

We indeed don't seem to have a good piece of documentation on how to properly convert a `sentencepiece` based model to `transformers` / `tokenizer` format. I'll try to create one in the coming month! 🤗 

Thankyou @ArthurZucker. The problem was resolved by using https://github.com/huggingface/tokenizers/blob/main/bindings/python/scripts/sentencepiece_extractor.py I managed to get the vocab.json and merges.txt files. Then, by using https://huggingface.co/docs/tokenizers/api/models#tokenizers.models.BPE, I was able to produce the tokenizer.json file.

@amyeroberts sure, Noted.
thanks

@nandinimundra your solution does not work. I tried but the vocab.json is not correct.

> Thankyou @ArthurZucker. The problem was resolved by using https://github.com/huggingface/tokenizers/blob/main/bindings/python/scripts/sentencepiece_extractor.py I managed to get the vocab.json and merges.txt files. Then, by using https://huggingface.co/docs/tokenizers/api/models#tokenizers.models.BPE, I was able to produce the tokenizer.json file.
> 
> @amyeroberts sure, Noted. thanks



I want to convert huggingface tokenizer into sentencepiece format (.model / .spm). let's say bert tokenizer.
I need it to initialize MarianTokenizer (it requires .model files) any suggestion?

+1 one the @RRaphaell message 

@rraphaell @ooddaa this has been requested, but I don't think we have an automatic way to do it! In `transformers` we usually prefer fast tokenizer classes written in Rust, which usually use `tokenizer.json`, rather than the original `sentencepiece` model files in `.model/.spm` format. As a result, we have lots of code for converting **to** fast `tokenizer.json`, by initializing tokenizers using the `from_slow` argument, but not a lot for converting **from** `tokenizer.json` (there is no `from_fast` argument!)

If someone wants to investigate this and try writing a converter, let us know! I believe the KerasNLP project was also interested, because they can already load safetensors files, so adding this would allow them to just grab any Transformers checkpoint and load it as a full KerasNLP model + tokenizer.

➕ on @Rocketknight1's answer here! happy to have a MarianTokenizerFast
cc @itazap maybe!