Race Condition when using Sagemaker Checkpointing and Model Repository
Author: DougTrajano
Pull request: no

### System Info

transformers version: 4.26.0
huggingface_hub version: 0.12.0
Platform: SageMaker
pytorch version: 1.10.2+cuda113

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Looks like we have a race condition when using the SageMaker Checkpointing feature together with Model Repository (`push_to_hub=True` in the TrainingArguments).

Basically, SageMaker creates temporary files inside the checkpoint dir. When using a Model Repository, these files are mapped in git, which raises an error FileNotFoundError when the file is deleted by SageMaker later. 

I tested several executions and always fails, except when I used another output_dir path such as `./output`, which isn't a SageMaker local checkpoint directory.

## Reproduction

## train.py

```python
...

trainer_args = TrainingArguments(
    output_dir="opt/ml/checkpoints",
    overwrite_output_dir=True if get_last_checkpoint(
        "opt/ml/checkpoints"
    ) is not None else False,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=self.args.early_stopping_patience+1,
    load_best_model_at_end=True,
    push_to_hub=True,
    hub_token=self.env.HUGGINGFACE_HUB_TOKEN,
    hub_model_id=self.args.hub_model_id,
    hub_strategy="checkpoint",
    metric_for_best_model="f1",
    num_train_epochs=self.args.num_train_epochs,
    seed=self.args.seed
)

trainer = Trainer(
    model=self.model,
    args=trainer_args,
    train_dataset=self.dataset["train"],
    eval_dataset=self.dataset[self.args.eval_dataset],
    tokenizer=self.tokenizer,
    compute_metrics=lambda p: compute_metrics(p, threshold=self.args.threshold),
    callbacks=[
        EarlyStoppingCallback(early_stopping_patience=self.args.early_stopping_patience)
    ] if self.args.early_stopping_patience is not None else None
)

# check if checkpoint existing if so continue training
last_checkpoint = get_last_checkpoint("opt/ml/checkpoints")
if last_checkpoint is not None:
    _logger.info(f"Resuming training from checkpoint: {last_checkpoint}")

trainer.train(resume_from_checkpoint=last_checkpoint)

...
```

## SageMaker Estimator

```python
...

import logging
from sagemaker.huggingface import HuggingFace

checkpoint_s3_uri = f"s3://{bucket_name}/{prefix}/checkpoints"

instance_type = "ml.g4dn.xlarge"

estimator = HuggingFace(
    entry_point="train.py",
    source_dir="ml",
    base_job_name=params.mlflow_experiment_name,
    container_log_level=logging.DEBUG,
    role=params.sagemaker_execution_role_arn,
    sagemaker_session=sagemaker_session,
    py_version="py38",
    pytorch_version="1.10.2",
    transformers_version="4.17.0",
    instance_count=1,
    instance_type=instance_type,
    use_spot_instances=True,
    max_wait=10800,
    max_run=10800,
    checkpoint_s3_uri=checkpoint_s3_uri,
    checkpoint_local_path="/opt/ml/checkpoints",
    environment={
        "MLFLOW_TRACKING_URI": params.mlflow_tracking_uri,
        "MLFLOW_EXPERIMENT_NAME": params.mlflow_experiment_name,
        "MLFLOW_TRACKING_USERNAME": params.mlflow_tracking_username,
        "MLFLOW_TRACKING_PASSWORD": params.mlflow_tracking_password,
        "MLFLOW_TAGS": params.mlflow_tags,
        "MLFLOW_RUN_ID": mlflow.active_run().info.run_id,
        "MLFLOW_FLATTEN_PARAMS": "True",
        "HF_MLFLOW_LOG_ARTIFACTS": "True",
        "HUGGINGFACE_HUB_TOKEN": params.huggingface_hub_token
    },
    hyperparameters={
        "push_to_hub": "True",
        "hub_model_id": f"dougtrajano/{params.mlflow_experiment_name}",
        "num_train_epochs": params.num_train_epochs,
        "early_stopping_patience": params.early_stopping_patience,
        "batch_size": params.batch_size,
        "seed": params.seed,
        "concat_validation_set": "True",
        "eval_dataset": "test"
   }
)

estimator.fit(inputs, wait=False)

```

Full code is available in [DougTrajano/ToChiquinho](https://github.com/DougTrajano/ToChiquinho)

## Logs

The file that raises the error always has "sagemaker-uploading" or "sagemaker-uploaded" in its name.

```log
Traceback (most recent call last):
  File ""train.py"", line 29, in <module>
    experiment.run()
  File ""/opt/ml/code/experiments/toxic_comment_classification.py"", line 199, in run
    trainer.train(resume_from_checkpoint=last_checkpoint)
  File ""/opt/conda/lib/python3.8/site-packages/transformers/trainer.py"", line 1543, in train
    return inner_training_loop(
  File ""/opt/conda/lib/python3.8/site-packages/transformers/trainer.py"", line 1883, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File ""/opt/conda/lib/python3.8/site-packages/transformers/trainer.py"", line 2135, in _maybe_log_save_evaluate"
1676172693307,"self._save_checkpoint(model, trial, metrics=metrics)
  File ""/opt/conda/lib/python3.8/site-packages/transformers/trainer.py"", line 2279, in _save_checkpoint"
1676172693307,"self._push_from_checkpoint(output_dir)
  File ""/opt/conda/lib/python3.8/site-packages/transformers/trainer.py"", line 3443, in _push_from_checkpoint"
1676172693308,"_, self.push_in_progress = self.repo.push_to_hub(
  File ""/opt/conda/lib/python3.8/site-packages/huggingface_hub/repository.py"", line 1366, in push_to_hub"
1676172693308,"self.git_add(auto_lfs_track=True)
  File ""/opt/conda/lib/python3.8/site-packages/huggingface_hub/repository.py"", line 1046, in git_add"
1676172693308,"tracked_files = self.auto_track_large_files(pattern)
  File ""/opt/conda/lib/python3.8/site-packages/huggingface_hub/repository.py"", line 970, in auto_track_large_files"
1676172693308,"size_in_mb = os.path.getsize(path_to_file) / (1024 * 1024)
  File ""/opt/conda/lib/python3.8/genericpath.py"", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError
[Errno 2] No such file or directory: '/opt/ml/checkpoints/toxic-comment-classification-2023-02-12-03-19-37-149/model/last-checkpoint/special_tokens_map.json.sagemaker-uploading'
3%|▎         | 1408/42240 [03:37<1:45:03,  6.48it/s]
2023-02-12 03:31:34,706 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code.
2023-02-12 03:31:34,706 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 1 from exiting process.
2023-02-12 03:31:34,707 sagemaker-training-toolkit ERROR    Reporting training FAILURE
2023-02-12 03:31:34,707 sagemaker-training-toolkit ERROR    ExecuteUserScriptError:
ExitCode 1
ErrorMessage ""FileNotFoundError
 [Errno 2] No such file or directory: '/opt/ml/checkpoints/toxic-comment-classification-2023-02-12-03-19-37-149/model/last-checkpoint/special_tokens_map.json.sagemaker-uploading'
 3%|▎         | 1408/42240 [03:37<1:45:03,  6.48it/s]""
Command ""/opt/conda/bin/python3.8 train.py --batch_size 8 --early_stopping_patience 5 --eval_dataset test --hub_model_id dougtrajano/toxic-comment-classification --num_train_epochs 30 --push_to_hub True --seed 1993"""
1676172695312,"2023-02-12 03:31:34,707 sagemaker-training-toolkit ERROR    Encountered exit_code 1
```

## Proposed Solution

In my opinion, the issue happens because SageMaker doesn't use a good sync mechanism for the checkpoint folder, but I don't know if they will change it because of this :( However, I think that there're some options from our side we can do.

One of the possible solutions I thought of is to add `*.sagemaker-uploading` and `*.sagemaker-uploaded` in the `.gitignore` file in the `Trainer().init_repo()` when we know that we are running inside SageMaker.

https://github.com/huggingface/transformers/blob/c836f77266be9ace47bff472f63caf71c0d11333/src/transformers/trainer.py#L3357-L3398

Additionally, we need to add the `--exclude-standard` in the git-lfs-files command called inside the `auto_track_large_files()` function.

I tested it by adding the following code between the `Trainer()` object creation and the execution of the `Trainer().train()` function.

```python
with open(os.path.join(trainer.repo.local_dir, ".gitignore"), "a") as f:
    f.write("\n*.sagemaker-uploading")
    f.write("\n*.sagemaker-uploaded")
trainer.repo.git_add(".gitignore")
trainer.repo.git_commit("Add *.sagemaker patterns to .gitignore.")
```

and in the [huggingface/huggingface_hub](https://github.com/huggingface/huggingface_hub).

```python
def patched_files_to_be_staged(
    pattern: str = ".", folder: Union[str, Path, None] = None
) -> List[str]:
    """
    Returns a list of filenames that are to be staged.

    Args:
        pattern (`str` or `Path`):
            The pattern of filenames to check. Put `.` to get all files.
        folder (`str` or `Path`):
            The folder in which to run the command.

    Returns:
        `List[str]`: List of files that are to be staged.
    """
    try:
        # --exclude-standard
        p = run_subprocess("git ls-files --exclude-standard -mo".split() + [pattern], folder)
        if len(p.stdout.strip()):
            files = p.stdout.strip().split("\n")
        else:
            files = []
    except subprocess.CalledProcessError as exc:
        raise EnvironmentError(exc.stderr)

    _logger.debug(f"Files to be staged: {files}")
    return files

# Monkey patching huggingface_hub.repository.files_to_be_staged
from huggingface_hub import repository
repository.files_to_be_staged = patched_files_to_be_staged
```

<details>
  <summary>files_to_be_staged() without --exclude-standard arg</summary>

  2023-02-13 11:32:32 :: DEBUG :: train :: patched_files_to_be_staged :: Files to be staged: ['.gitattributes.sagemaker-uploaded', '.gitignore.sagemaker-uploaded', 'README.md.sagemaker-uploaded', 'config.json', 'config.json.sagemaker-uploaded', 'last-checkpoint/config.json', 'last-checkpoint/config.json.sagemaker-uploaded', 'last-checkpoint/optimizer.pt', 'last-checkpoint/optimizer.pt.sagemaker-uploading', 'last-checkpoint/pytorch_model.bin', 'last-checkpoint/pytorch_model.bin.sagemaker-uploading', 'last-checkpoint/rng_state.pth', 'last-checkpoint/rng_state.pth.sagemaker-uploaded', 'last-checkpoint/rng_state.pth.sagemaker-uploading', 'last-checkpoint/scheduler.pt', 'last-checkpoint/scheduler.pt.sagemaker-uploading', 'last-checkpoint/special_tokens_map.json', 'last-checkpoint/special_tokens_map.json.sagemaker-uploaded', 'last-checkpoint/special_tokens_map.json.sagemaker-uploading', 'last-checkpoint/tokenizer.json', 'last-checkpoint/tokenizer.json.sagemaker-uploaded', 'last-checkpoint/tokenizer_config.json', 'last-checkpoint/tokenizer_config.json.sagemaker-uploaded', 'last-checkpoint/trainer_state.json', 'last-checkpoint/trainer_state.json.sagemaker-uploaded', 'last-checkpoint/training_args.bin', 'last-checkpoint/training_args.bin.sagemaker-uploaded', 'last-checkpoint/vocab.txt', 'last-checkpoint/vocab.txt.sagemaker-uploaded', 'pytorch_model.bin', 'pytorch_model.bin.sagemaker-uploading', 'special_tokens_map.json', 'special_tokens_map.json.sagemaker-uploaded', 'tokenizer.json', 'tokenizer.json.sagemaker-uploading', 'tokenizer_config.json', 'tokenizer_config.json.sagemaker-uploaded', 'training_args.bin', 'training_args.bin.sagemaker-uploaded', 'vocab.txt']

</details>

<details>
  <summary>files_to_be_staged() with --exclude-standard arg</summary>

  2023-02-13 11:42:35 :: DEBUG :: train :: patched_files_to_be_staged :: Files to be staged: ['config.json', 'last-checkpoint/config.json', 'last-checkpoint/optimizer.pt', 'last-checkpoint/pytorch_model.bin', 'last-checkpoint/rng_state.pth', 'last-checkpoint/scheduler.pt', 'last-checkpoint/special_tokens_map.json', 'last-checkpoint/tokenizer.json', 'last-checkpoint/tokenizer_config.json', 'last-checkpoint/trainer_state.json', 'last-checkpoint/training_args.bin', 'last-checkpoint/vocab.txt', 'pytorch_model.bin', 'special_tokens_map.json', 'tokenizer.json', 'tokenizer_config.json', 'training_args.bin', 'vocab.txt']

</details>

If you agree with this solution, I'll be very happy to submit a PR to implement this.

## Some links

- [Run training on Amazon SageMaker](https://huggingface.co/docs/sagemaker/train)
- [Renate/file.py at main · awslabs/Renate](https://github.com/awslabs/Renate/blob/main/src/renate/utils/file.py#L98-L116)

### Expected behavior

I expected that I can use SageMaker Checkpointing with Model Repository with no errors.

Comments:
Thanks for diving into this and offering solutions! I think your plan sounds sensible, woudl you like to open a PR with it?

> Thanks for diving into this and offering solutions! I think your plan sounds sensible, woudl you like to open a PR with it?

yeah! I'll do that and submit a PR soon.

**Update:** I merged https://github.com/huggingface/huggingface_hub/pull/1339. Will not be released before a few weeks but you can always install it from git if it's urgent :) 

> **Update:** I merged [huggingface/huggingface_hub#1339](https://github.com/huggingface/huggingface_hub/pull/1339). Will not be released before a few weeks but you can always install it from git if it's urgent :)

no problem, I monkey-patched it in my training script. :)

If someone also needs it before the new release. The monkey patch code is in the issue description.