Inconsistent saving of tokenizer with custom code from HF hub vs. local directory
Author: mtm-cai
Pull request: no
Labels: name: bug

name: Remote code

### System Info

- `transformers` version: 4.47.1
- Platform: Linux-5.15.0-130-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.25.2
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: no
- GPU type: NVIDIA RTX A6000

### Who can help?

@ArthurZucker or @itazap 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1GPySa5JEclxzlVZbI1mg3ajWzjv3mf42?usp=sharing

When loading a tokenizer that requires custom code from HF hub and saving it, the custom Python file is not saved. In contrast, loading the same tokenizer from a local clone of the repository and saving it behaves as expected, including the custom Python file.

**Aditional context**
The output during the `save_pretrained` method lists an `added_tokens.json` file, which does not exist after saving and is not part of the original repository files. This happens regardless of where the tokenizer was loaded from.

### Expected behavior

When saving the tokenizer loaded from the HF hub, the custom Python file should be consistently included, ensuring complete functionality of the tokenizer.

Comments:
cc @arthurzucker for tokenizers but let me know if you want to take this, since it's a custom code saving issue

Will let you handle it @Rocketknight1 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Still needs to be fixed.

On Sun, 9 Mar 2025, 08:06 github-actions[bot], ***@***.***>
wrote:

> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/35597#issuecomment-2708726082>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BOLEA2Z46E5IXJCWUERO4F32TPZBZAVCNFSM6AAAAABU4Q3JXOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBYG4ZDMMBYGI>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
> [image: github-actions[bot]]*github-actions[bot]* left a comment
> (huggingface/transformers#35597)
> <https://github.com/huggingface/transformers/issues/35597#issuecomment-2708726082>
>
> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/35597#issuecomment-2708726082>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BOLEA2Z46E5IXJCWUERO4F32TPZBZAVCNFSM6AAAAABU4Q3JXOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBYG4ZDMMBYGI>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


hey @mtm-cai thanks for your patience on this! You mentioned this is needed for "complete functionality of the tokenizer", are you experiencing a discrepancy in the tokenizer functionality when loaded from the hub? If so, can you please provide a reproducible code snippet? Thanks 😊 

edit:
took a deeper look and here's what is happening under the hood:

1. Load model from hub:
    - the model is saved into your local cache, including the custom tokenizer `py` file. So, when you save the tokenizer into a given directory, it doesn't need to "duplicate" the `.py` file there.

2.  Download model locally:
     - the model is downloaded explicitly to a given directory, nothing is saved in your local cache, and so a copy is downloaded to the directory, unlike when loading from the hub.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.