Repository: huggingface/transformers

Issue:
training LayouttLM 1 epoch in distributed more results in error
Not sure whether this issue should be posted here or rather in the pytorch repo, please let me know if it is not a transformer issue.

When training LayoutLM with the Trainer in distributed mode for only one epoch, with setting `load_best_model_at_end` to `True`, I get an error when the model is loaded at the end. According to the error message, the config.json file for the model cannot be found although it is there. This issue does **not** arise when not training in distributed mode or when training in distributed mode for more than one epoch.

```
import os

import torch
from transformers import EvaluationStrategy, Trainer
from transformers.training_args import TrainingArguments
from transformers import (
    LayoutLMConfig,
    LayoutLMForTokenClassification,
)

training_args = TrainingArguments(
    output_dir="output_dir",  # output directory
    do_train=True,
    do_eval=False,
    do_predict=False,
    evaluation_strategy=EvaluationStrategy.EPOCH,
    num_train_epochs=1,  # total # of training epochs
    per_device_train_batch_size=8,  # batch size per device during training
    per_device_eval_batch_size=8,  # batch size for evaluation
    weight_decay=0.0005,  # strength of weight decay
    learning_rate=0.00000001,
    logging_steps=0,  # it logs when running evaluation so no need to log on step interval
    save_steps=0,
    seed=42,
    overwrite_output_dir=True,
    save_total_limit=10,
    load_best_model_at_end=True,
    metric_for_best_model="f1",
    greater_is_better=True,  # higher f1 score is better
    fp16=True,
    local_rank=-1,
    gradient_accumulation_steps=2,
    warmup_steps=300,
)

model_dir = "layoutlm_pretrained_model"
train_dataset = []
validation_dataset = []

config = LayoutLMConfig.from_pretrained(
    os.path.join(model_dir, "config.json"), num_labels=64, cache_dir=None
)
model = LayoutLMForTokenClassification.from_pretrained(
    model_dir,
    from_tf=bool(".ckpt" in model_dir),
    config=config,
    cache_dir=None,
)

device = torch.device("cuda")
model.train().to(device)

trainer = Trainer(
    model=model,  # the instantiated 🤗 Transformers model to be trained
    args=training_args,  # training arguments, defined above
    train_dataset=train_dataset,  # training dataset
    eval_dataset=validation_dataset,  # validation dataset
)

trainer.train()
```


Discussion:
Hi! Do you mind posting the error message with the stacktrace? Thank you!

Pinging @sgugger 
Here you go:
```
Training completed. Do not forget to share your model on huggingface.co/models =)
Loading best model from /mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8 (score: 0.00027643400138217).
Saving model checkpoint to /mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8
Configuration saved in /mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8/config.json
404 Client Error: Not Found for url: 
https://huggingface.co//mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8/resolve/main/config.json
2021-03-11 12:17:57 ERROR    layoutlm_model_training_script Can't load config for '/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8'. Make sure that:
- '/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8' is a correct model identifier listed on '
https://huggingface.co/models
'
- or '/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8' is the correct path to a directory containing a config.json file
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 399, in get_config_dict
    resolved_config_file = cached_path(
  File "/usr/local/lib/python3.8/dist-packages/transformers/file_utils.py", line 1077, in cached_path
    output_path = get_from_cache(
  File "/usr/local/lib/python3.8/dist-packages/transformers/file_utils.py", line 1215, in get_from_cache
    r.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: 
https://huggingface.co//mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8/resolve/main/config.json
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 50, in <module>
    train_model(
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 260, in train_model
    trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 868, in train
    self.model = self.model.from_pretrained(self.state.best_model_checkpoint)
  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 948, in from_pretrained
    config, model_kwargs = cls.config_class.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 360, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 418, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for '/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8'. Make sure that:
- '/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8' is a correct model identifier listed on '
https://huggingface.co/models
'
- or '/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8' is the correct path to a directory containing a config.json file
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 399, in get_config_dict
    resolved_config_file = cached_path(
  File "/usr/local/lib/python3.8/dist-packages/transformers/file_utils.py", line 1077, in cached_path
    output_path = get_from_cache(
  File "/usr/local/lib/python3.8/dist-packages/transformers/file_utils.py", line 1215, in get_from_cache
    r.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: 
https://huggingface.co//mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8/resolve/main/config.json
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 50, in <module>
    train_model(
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 260, in train_model
    trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 868, in train
    self.model = self.model.from_pretrained(self.state.best_model_checkpoint)
  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 948, in from_pretrained
    config, model_kwargs = cls.config_class.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 360, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 418, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for '/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8'. Make sure that:
- '/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8' is a correct model identifier listed on '
https://huggingface.co/models
'
- or '/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8' is the correct path to a directory containing a config.json file
```


And what is inside the folder `/mnt/pipeline/636d29be-97f9-458a-8174-70ae425a2a6a/pytorch_model/checkpoint-8`?
config.json
optimizer.pt
pytorch_model.bin
sheduler.pt
trainer_state.json
training_args.bin

Everything that is needed to load the model.
I checked the config file, it looks entirely normal.
Oh, I think I know why: it's possible the process 1 arrived at that line before the process 0 finished its save and since there is no barrier, it failed loading the model since it wasn't there yet. Will make a fix for that.
That makes sense, judging by our logs process 1 wasn't finished yet, as we had a log of saving a checkpoint after the error message from process 0. I cannot share the logs since the pod they were one is already gone...

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -1131,6 +1131,12 @@ def train(
 
         logger.info("\n\nTraining completed. Do not forget to share your model on huggingface.co/models =)\n\n")
         if self.args.load_best_model_at_end and self.state.best_model_checkpoint is not None:
+            # Wait for everyone to get here so we are sur the model has been saved by process 0.
+            if is_torch_tpu_available():
+                xm.rendezvous("load_best_model_at_end")
+            elif self.args.local_rank != -1:
+                dist.barrier()
+
             logger.info(
                 f"Loading best model from {self.state.best_model_checkpoint} (score: {self.state.best_metric})."
             )


Test patch: