[WIP] Ignore checkpoint error
Author: Bfault
Pull request: yes

What does this PR do?

Having a flag to load the checkpoints until the checkpoint is valid or no more checkpoints are available

Fixes #31283

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)


Comments:
> Thanks! Overall this looks very well done. If possible can we add a test under `tests/trainer/test_trainer.py`?
> 
> Otherwise a few nits just to make sure we don't barrage our users with too many messages (that they generally won't read).
> 
> cc @amyeroberts :)

I was trying to add test but It's a bit complicated to simulate the corruption of checkpoints.
I tried to run a regression train on a thread and wait the "saving model to" in the logger to stop the first thread, but I'm not familiar with logging

This test should work but I found out that it'll be more complex to catch all errors concerning the checkpoint file.
For now I don't pass the test because it does not catch the case where optimizer.pt is corrupted.
I can resolve this by making the same for try loop around the `inner_training_loop` call but using 4 times the same for try loop look a bit ugly. I try to make a wrapper but it make uglier.
Any suggestion ?

> @Bfault I agree, not pretty but I think we need to blanket `exception`. What we can do though is keep that exception stored and raise it at the end to give a trace, potentially.

raise it even if one of the checkpoints work ?

1. The `_load_from_checkpoint` does not load all file in checkpoint, only: `config.json` `model.safetensors`
2. the `TrainerState.load_from_json` in `train` do: `trainer_state.json`

3. the `deepspeed_load_checkpoint` in `_inner_training_loop`  do: Idk yet use deepspeed lib
4. the `self._load_optimizer_and_scheduler` in `_inner_training_loop` do: `optimizer.pt` and `scheduler.pt`
5. the `self._load_rng_state` in `_inner_training_loop` do: `rng_state.pth`
`training_args.bin` seems to be saved but not used during training

I think there is an issue with doing like this,
if the checkpoint that passes 1 is not passing 2 it'll keep going with the checkpoint of 2 except with config.json and model.safetensors loaded with the checkpoint that passed 1.

A dry run of all loading functions may be better ?
I don't know if just loading is time consuming or not

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.