Unable to resume model training with a locally downloaded model and trust remote code
Author: conceptofmind
Pull request: no
Labels: name: bug

### System Info

requires-python = ">=3.12"
dependencies = [
    "accelerate>=1.13.0",
    "datasets>=4.8.5",
    "sentence-transformers>=5.4.1",
    "torch>=2.11.0",
    "transformers>=5.8.0",
]

### Who can help?

@tomaszcichy98 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When downloading a model locally using:
```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="voyageai/voyage-4-nano",
    local_dir="/e/data1/datasets/playground/mmlaion/shared/enrico/models/voyage-4-nano",
)
```
I am unable to resume training as there are missing files due to custom code not being saved on disk.
```python
    model = SentenceTransformer(
        base_model_path, 
        trust_remote_code=trust_remote_code
    )
```
I get the error:
```
[rank1]: Traceback (most recent call last):
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank1]:     main()
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank1]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank1]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank1]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank1]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank1]:     super().__init__(
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank1]:     modules, self.module_kwargs = self._load_modules(
[rank1]:                                   ^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank1]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank1]:     module = module_class.load(
[rank1]:              ^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank1]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank1]:     self.model = self._load_model(
[rank1]:                  ^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank1]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank1]:     model_class = get_class_from_dynamic_module(
[rank1]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank1]:     final_module = get_cached_module_file(
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank1]:     resolved_module_file = cached_file(
[rank1]:                            ^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank1]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank1]:     raise OSError(
[rank1]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank3]: Traceback (most recent call last):
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank3]:     main()
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank3]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank3]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank3]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank3]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank3]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank3]:     super().__init__(
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank3]:     modules, self.module_kwargs = self._load_modules(
[rank3]:                                   ^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank3]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank3]:     module = module_class.load(
[rank3]:              ^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank3]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank3]:     self.model = self._load_model(
[rank3]:                  ^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank3]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank3]:     model_class = get_class_from_dynamic_module(
[rank3]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank3]:     final_module = get_cached_module_file(
[rank3]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank3]:     resolved_module_file = cached_file(
[rank3]:                            ^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank3]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank3]:     raise OSError(
[rank3]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank2]: Traceback (most recent call last):
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank2]:     main()
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank2]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank2]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank2]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank2]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank2]:     super().__init__(
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank2]:     modules, self.module_kwargs = self._load_modules(
[rank2]:                                   ^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank2]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank2]:     module = module_class.load(
[rank2]:              ^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank2]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank2]:     self.model = self._load_model(
[rank2]:                  ^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank2]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank2]:     model_class = get_class_from_dynamic_module(
[rank2]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank2]:     final_module = get_cached_module_file(
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank2]:     resolved_module_file = cached_file(
[rank2]:                            ^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank2]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank2]:     raise OSError(
[rank2]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank0]:     main()
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank0]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank0]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank0]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank0]:     super().__init__(
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank0]:     modules, self.module_kwargs = self._load_modules(
[rank0]:                                   ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank0]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank0]:     module = module_class.load(
[rank0]:              ^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank0]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank0]:     self.model = self._load_model(
[rank0]:                  ^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank0]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank0]:     model_class = get_class_from_dynamic_module(
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank0]:     final_module = get_cached_module_file(
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank0]:     resolved_module_file = cached_file(
[rank0]:                            ^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank0]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank0]:     raise OSError(
[rank0]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
```
This currently happens with the models:
https://huggingface.co/voyageai/voyage-4-nano
https://huggingface.co/jinaai/jina-embeddings-v5-text-small-retrieval

### Expected behavior

I should be able to resume model training.

Comments:
the issue is that ur custom code cant be saved or gets overwritten via the downloads right? so cant you refactor those custom portions to ensure that the downloads dont overwrite it?

Tom has noted that this might be a regression of this PR: https://github.com/huggingface/transformers/pull/15379


hi, i'd love to work on this issue. 

From reading the traceback, it looks like the root cause is that _save_checkpoint in trainer.py doesn't copy over the custom remote code files (e.g. modeling_qwen3_bidirectional.py) when saving a checkpoint for a model loaded with trust_remote_code=True. So when resuming, the dynamic module loader can't find those files in the checkpoint directory.
My proposed fix would be to detect trust_remote_code usage at checkpoint save time and copy any custom .py files from the original model directory into the checkpoint folder. Does that approach sound right to you? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
> 
> Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale