Question about resum_from_checkpoint in run_translation_no_trainer.py
Author: danielDigitalArt
Pull request: no

### System Info

- `transformers` version: 4.29.0.dev0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.6
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.12.1+cpu (False)
- Tensorflow version (GPU?): 2.10.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

Hi, i stumbled across this function, while i was debugging my own code which is a changed version of the no_trainer version.
While my code crashes with a cuda error, i found this part interesting as it would help me in getting faster to the evaluation process (where my error occurs).

However it seems a bit weird as i was reading this function
[https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/run_translation_no_trainer.py#LL594C4-L614C66](https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/run_translation_no_trainer.py#LL594C4-L614C66)

What seems a bit wird to me is, that first "resume_from_checkpoint" could be either None (default) or a checkpoint like "epoch_5" or "step_1000". The parameter Definition says it should be of Type String or daults to None [https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/run_translation_no_trainer.py#L266-L271](https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/run_translation_no_trainer.py#L266-L271)

The first check is if the argument was set, to go into the resume block. 
```
if args.resume_from_checkpoint:
```
From my understanding the following values would work:
"step_1234" or any other string,
"" an empty string.
True

Now the next check will look if it is not none or not empty
```
        if args.resume_from_checkpoint is not None or args.resume_from_checkpoint != "":
```
So everything in the if block will be executed in these instances:
is: True/False
is: "step_1000" or any other non empty string.
But if it is True, the later part can not work here as it searches either for "epoch_*" or "step_*", in one of my testruns it whowed when using True it will search for the folder "True" which then fails. While it might be perfectly finde to set it to "" i also found some examples about using the normal trainer, where resume was simply set to true like so:
```trainer.train(resume_from_checkpoint = True)```

My suggestion here would be to first check if it is not None, then if it is not True:
```
if args.resume_from_checkpoint:
    args.resume_from_checkpoint != "" and args.resume_from_checkpoint is not True
```
So if it is set to True or an empty string, it will search for a folder with either "step_" or "epoch_" as a name and use this folder.

Another difficulty i found in my project was, that i wanted to save the checkpoints into a subfolder, saving it was not the problem but loading it here, because the script will only replace "epoch_" or "step_" to fetch the step or epoch number. In Order to find the folders in the subfolder i changed the check like so:
```
                dirs = [os.path.join(args.checkpoint_dir, f.name) for f in os.scandir(f"{args.checkpoint_dir}") if f.is_dir()]
...
            import re
            # First the correct epoch is detected here, later the process will skip training until reaching the correct step.
            if "epoch" in training_difference:
                repl = re.search(r'epoch_(\d+)', training_difference).group()
                starting_epoch = int(repl.replace("epoch_", "")) + 1
```
Same check would have to be done in the "step_" part.

Maybe the args.checkpoint_dir can be omitted, so the user can specify the path when setting the argument (True or "" would then not work anymore)

At last i have a question about this part:
```
accelerator.load_state(checkpoint)
```
This is only defined in the block when ```args.resume_from_checkpoint``` is not empty, is it not needed to let the accelerator always load the appropriate checkpoint? In the example it would not use accelerator.load_state when loading it from the latest chekpoint which it discovered by itself.

Edit:
fixed file links

Comments:
cc @muellerzr 

@danielDigitalArt it's not needed unless we resume from a checkpoint, yes. If you would like to open a PR with your suggestions, that would be welcome as they do make sense to me as an inclusion. Very keen observation

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.