Can not find best model after training.
Author: ILG2021
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The error is:
`Could not locate the best model at whisper-large-v2-fil/checkpoint-2000/pytorch_model.bin, if you are running a distributed training on multiple nodes, you should activate `--save_on_each_node`.`

Because the checkpoint is using safetensor now, but the trainer tried to find pytorch_model.bin. So it can not find.

### Expected behavior

works regular.

Comments:
Hey, could you share the training command you used? (a reproducer for training) as well as the full traceback? 🤗 


> `Could not locate the best model at whisper-large-v2-fil/checkpoint-2000/pytorch_model.bin, if you are running a distributed training on multiple nodes, you should activate `--save_on_each_node`.`

I use the finetune whisper notebook in this blog:
https://huggingface.co/blog/fine-tune-whisper

We need the call you used, and the full traceback 😉 

I had the same issue, i fixed with instead of this

model.save_pretrained(output_dir)

i've use

model.save_pretrained(output_dir, safe_serialization=False)

And also on the training arguments, i 've added   save_safetensors=False which by default is True

training_args = TrainingArguments(
        output_dir=output_dir,
        overwrite_output_dir=True,
        num_train_epochs=epochs, 
        per_device_train_batch_size=batch_size,
        learning_rate=2e-5, 
        weight_decay=0.02, 
        max_grad_norm=1.0,
        save_steps=30,
        save_total_limit=2,
        save_strategy="steps",
        warmup_steps=1000, 
        gradient_accumulation_steps=1,
        logging_dir = output_dir + '/logs',
        logging_steps=30,
        save_safetensors=False
 )







> I had the same issue, i fixed with instead of this
> 
> model.save_pretrained(output_dir)
> 
> i've use
> 
> model.save_pretrained(output_dir, safe_serialization=False)
> 
> And also on the training arguments, i 've added save_safetensors=False which by default is True
> 
> training_args = TrainingArguments( output_dir=output_dir, overwrite_output_dir=True, num_train_epochs=epochs, per_device_train_batch_size=batch_size, learning_rate=2e-5, weight_decay=0.02, max_grad_norm=1.0, save_steps=30, save_total_limit=2, save_strategy="steps", warmup_steps=1000, gradient_accumulation_steps=1, logging_dir = output_dir + '/logs', logging_steps=30, save_safetensors=False )

Thanks for you sharing. That's the problem.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.