Repository: huggingface/accelerate

Issue:
`FileNotFoundError` from `load_state` caused the `input_dir` is joined twice
### System Info

```Shell
- `Accelerate` version: 0.22.0
- Platform: Linux-4.15.0-194-generic-x86_64-with-glibc2.27
- Python version: 3.9.15
- Numpy version: 1.24.4
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- PyTorch XPU available: False
- PyTorch NPU available: False
- System RAM: 251.84 GB
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
```


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [X] My own task or dataset (give details below)

### Reproduction

In https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L2879, `input_dir`  is added twice to `folders`, resulting in a  `FileNotFoundError`. Replacing the last line with `input_dir =folders[-1]` solved the issue for me.

```
        elif self.project_configuration.automatic_checkpoint_naming:
            # Pick up from automatic checkpoint naming
            input_dir = os.path.join(self.project_dir, "checkpoints")
            folders = [os.path.join(input_dir, folder) for folder in os.listdir(input_dir)]

            def _inner(folder):
                return list(map(int, re.findall(r"[\/]?([0-9]+)(?=[^\/]*$)", folder)))[0]

            folders.sort(key=_inner)
            input_dir = os.path.join(input_dir, folders[-1])
``` 



### Expected behavior

Quick fix.


Discussion:
@qfournier can you try installing from the latest `pypi` version `0.23.0`? I think this was fixed as the logic is now:
```python
            if (
                self.project_configuration.total_limit is not None
                and (len(folders) + 1 > self.project_configuration.total_limit)
                and self.is_main_process
            ):

                def _inner(folder):
                    return list(map(int, re.findall(r"[\/]?([0-9]+)(?=[^\/]*$)", folder)))[0]

                folders.sort(key=_inner)
                logger.warning(
                    f"Deleting {len(folders) + 1 - self.project_configuration.total_limit} checkpoints to make room for new checkpoint."
                )
                for folder in folders[: len(folders) + 1 - self.project_configuration.total_limit]:
                    shutil.rmtree(folder)
```
So it's all done solely on the main process
Thanks for the quick reply! Unfortunately, `accelerate==0.23.0` did not solve the issue, which seems to have the same behavior:  https://github.com/huggingface/accelerate/blob/48d96319e0033fb8c8979072d97edf3995639029/src/accelerate/accelerator.py#L2879
@muellerzr note that the issue is different from https://github.com/huggingface/accelerate/issues/1982 as it is related to the ˋinput_dir` and not the process.
Can you provide a full script of how you are running this and receiving the error? 
@muellerzr I cannot share my full code, but here is the logic:

```python
# Accelerator object
project_config = ProjectConfiguration(project_dir=cfg.trainer.dir, automatic_checkpoint_naming=True, total_limit=5)
accelerator = Accelerator(project_config=project_config)

# Get the model, optimzier, scheduler, dataloaders, and loss
model = get_model(**cfg.model, **cfg.tokenizer)
optimizer = get_optimizer(model, **cfg.optimizer)
scheduler = get_scheduler(optimizer, **cfg.scheduler)
train_dataloader = get_dataloader(**cfg.tokenizer, **cfg.dataset.train, **cfg.trainer.train)
eval_dataloader = get_dataloader(**cfg.tokenizer, **cfg.dataset.validation, **cfg.trainer.validation)
loss_fn = get_loss(**cfg.tokenizer, **cfg.trainer.train)

# Prepare the model, optimzier, scheduler, and dataloaders for distributed training
train_dataloader, eval_dataloader, model, optimizer, scheduler = accelerator.prepare(
    train_dataloader,
    eval_dataloader,
    model,
    optimizer,
    scheduler,
)


if cfg.trainer.resume and os.path.exists(cfg.trainer.dir):
    accelerator.load_state()
    skipped_train_dataloader = accelerator.skip_first_batches(train_dataloader, metrics.num_steps_in_epoch)

for x, y, pad_mask in train_dataloader if skipped_train_dataloader is None else skipped_train_dataloader:
    outputs = model(x, pad_mask)
    train_loss = loss_fn(outputs.view(-1, cfg.tokenizer.vocab_size), y.view(-1))
    accelerator.backward(train_loss)
    optimizer.step()
    scheduler.step()
    optimizer.zero_grad()
    accelerator.save_state()
```

The first time I run the code, the folder `cfg.trainer.dir` will be created. Let's say there are 10 checkpoints (`checkpoint_0` to `checkpoint_9`). The second time I run the code with `cfg.trainer.resume=True`, the model will call `accelerator.load_state()`:

```python
    def load_state(self, input_dir: str = None, **load_model_func_kwargs):
        if input_dir is not None:
            # Check if folder exists
            input_dir = os.path.expanduser(input_dir)
            if not os.path.isdir(input_dir):
                raise ValueError(f"Tried to find {input_dir} but folder does not exist")
        elif self.project_configuration.automatic_checkpoint_naming:
            # Pick up from automatic checkpoint naming
            input_dir = os.path.join(self.project_dir, "checkpoints")
            folders = [os.path.join(input_dir, folder) for folder in os.listdir(input_dir)]

            def _inner(folder):
                return list(map(int, re.findall(r"[\/]?([0-9]+)(?=[^\/]*$)", folder)))[0]

            folders.sort(key=_inner)
            # ADD PRINT HERE
            print(f"input_dir: {input_dir}")
            print(f"folders[-1]: {folders[-1]}")
            print(f"os.path.join(input_dir, folders[-1]) : {os.path.join(input_dir, folders[-1])}")
            input_dir = os.path.join(input_dir, folders[-1])
        else:
            raise ValueError("No input_dir provided and automatic checkpoint naming is disabled.")
        logger.info(f"Loading states from {input_dir}")
```

The three prints that I have added give (real output from my script):

```
input_dir: outputs/toy/checkpoints
folders[-1]: outputs/toy/checkpoints/checkpoint_9
os.path.join(input_dir, folders[-1]) : outputs/toy/checkpoints/outputs/toy/checkpoints/checkpoint_9
[...]
FileNotFoundError: [Errno 2] No such file or directory: 'outputs/toy/checkpoints/outputs/toy/checkpoints/checkpoint_9/pytorch_model.bin'
```
Yes, I think this is indeed a bug. At first, I was astonished why this was not discovered earlier. I think the reason is that it only occurs with relative paths, not absolute, because of this behavior of `os.path.join`:

```python
>>> import os
>>> os.path.join("/tmp/foo", "/tmp/foo/bar.txt")
'/tmp/foo/bar.txt'
```

So with absolute paths, this double-joining of `input_dir` is not actually an issue. I'll create a PR to fix the issue.

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -2876,7 +2876,7 @@ def _inner(folder):
                 return list(map(int, re.findall(r"[\/]?([0-9]+)(?=[^\/]*$)", folder)))[0]
 
             folders.sort(key=_inner)
-            input_dir = os.path.join(input_dir, folders[-1])
+            input_dir = folders[-1]
         else:
             raise ValueError("No input_dir provided and automatic checkpoint naming is disabled.")
         logger.info(f"Loading states from {input_dir}")


Test patch:
diff --git a/tests/test_state_checkpointing.py b/tests/test_state_checkpointing.py
--- a/tests/test_state_checkpointing.py
+++ b/tests/test_state_checkpointing.py
@@ -19,6 +19,8 @@
 import shutil
 import tempfile
 import unittest
+import uuid
+from contextlib import contextmanager
 
 import pytest
 import torch
@@ -201,6 +203,71 @@ def test_can_resume_training(self):
             self.assertEqual(opt_state1, opt_state3)
             self.assertEqual(ground_truth_rands, test_rands)
 
+    def test_can_resume_training_checkpoints_relative_path(self):
+        # See #1983
+        # This test is like test_can_resume_training but uses a relative path for the checkpoint and automatically
+        # infers the checkpoint path when loading.
+        @contextmanager
+        def temporary_relative_directory():
+            # This is equivalent to tempfile.TemporaryDirectory() except that it returns a relative path
+            rand_dir = f"test_path_{uuid.uuid4()}"
+            os.mkdir(rand_dir)
+            try:
+                yield rand_dir
+            finally:
+                shutil.rmtree(rand_dir)
+
+        with temporary_relative_directory() as tmpdir:
+            set_seed(42)
+            model = DummyModel()
+            optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)
+            train_dataloader, valid_dataloader = dummy_dataloaders()
+            project_config = ProjectConfiguration(automatic_checkpoint_naming=True)
+
+            # Train baseline
+            accelerator = Accelerator(project_dir=tmpdir, project_config=project_config)
+            model, optimizer, train_dataloader, valid_dataloader = accelerator.prepare(
+                model, optimizer, train_dataloader, valid_dataloader
+            )
+            # Save initial
+            accelerator.save_state()
+            (a, b) = model.a.item(), model.b.item()
+            opt_state = optimizer.state_dict()
+            ground_truth_rands = train(3, model, train_dataloader, optimizer, accelerator)
+            (a1, b1) = model.a.item(), model.b.item()
+            opt_state1 = optimizer.state_dict()
+
+            # Train partially
+            set_seed(42)
+            model = DummyModel()
+            optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)
+            train_dataloader, valid_dataloader = dummy_dataloaders()
+            project_config = ProjectConfiguration(iteration=1, automatic_checkpoint_naming=True)
+            accelerator = Accelerator(project_dir=tmpdir, project_config=project_config)
+            model, optimizer, train_dataloader, valid_dataloader = accelerator.prepare(
+                model, optimizer, train_dataloader, valid_dataloader
+            )
+            accelerator.load_state()  # <= infer the directory automatically
+            (a2, b2) = model.a.item(), model.b.item()
+            opt_state2 = optimizer.state_dict()
+            self.assertEqual(a, a2)
+            self.assertEqual(b, b2)
+            self.assertEqual(opt_state, opt_state2)
+
+            test_rands = train(2, model, train_dataloader, optimizer, accelerator)
+            # Save everything
+            accelerator.save_state()
+
+            # Load everything back in and make sure all states work
+            accelerator.load_state(os.path.join(tmpdir, "checkpoints", "checkpoint_1"))
+            test_rands += train(1, model, train_dataloader, optimizer, accelerator)
+            (a3, b3) = model.a.item(), model.b.item()
+            opt_state3 = optimizer.state_dict()
+            self.assertEqual(a1, a3)
+            self.assertEqual(b1, b3)
+            self.assertEqual(opt_state1, opt_state3)
+            self.assertEqual(ground_truth_rands, test_rands)
+
     def test_invalid_registration(self):
         t = torch.tensor([1, 2, 3])
         t1 = torch.tensor([2, 3, 4])