config.json not found!
Author: beyondguo
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.0-1063-azure-x86_64-with-glibc2.10
- Python version: 3.8.3
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes
```


### Who can help?

@sgugger 
I am training a NER model following tutorial:
```python
from transformers import TrainingArguments

args = TrainingArguments(
    "saved_models_bert-finetuned-ner-100examples-with-aug",
    
    learning_rate=2e-5,
    num_train_epochs=100,
    weight_decay=0.01,
    per_device_train_batch_size = 32,
    per_device_eval_batch_size = 32,
    
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end = True,  
    metric_for_best_model = 'f1'
)

from transformers import Trainer

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=new_training_dataset,
    eval_dataset=tokenized_datasets["validation"].select(range(100)),
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    tokenizer=tokenizer,
)

trainer.train()
```

Then I got this error:
```shell
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [28], in <cell line: 14>()
      1 from transformers import Trainer
      3 trainer = Trainer(
      4     model=model,
      5     args=args,
   (...)
     11     tokenizer=tokenizer,
     12 )
---> 14 trainer.train()

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:1512, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1509     self.control.should_training_stop = True
   1511 self.control = self.callback_handler.on_epoch_end(args, self.state, self.control)
-> 1512 self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1514 if DebugOption.TPU_METRICS_DEBUG in self.args.debug:
   1515     if is_torch_tpu_available():
   1516         # tpu-comment: Logging debug metrics for PyTorch/XLA (compile, execute times, ops, etc.)

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:1628, in Trainer._maybe_log_save_evaluate(self, tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1625     self._report_to_hp_search(trial, epoch, metrics)
   1627 if self.control.should_save:
-> 1628     self._save_checkpoint(model, trial, metrics=metrics)
   1629     self.control = self.callback_handler.on_save(self.args, self.state, self.control)

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:1700, in Trainer._save_checkpoint(self, model, trial, metrics)
   1697     self.store_flos()
   1699 output_dir = os.path.join(run_dir, checkpoint_folder)
-> 1700 self.save_model(output_dir, _internal_call=True)
   1701 if self.deepspeed:
   1702     # under zero3 model file itself doesn't get saved since it's bogus! Unless deepspeed
   1703     # config `stage3_gather_16bit_weights_on_model_save` is True
   1704     self.deepspeed.save_checkpoint(output_dir)

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:2128, in Trainer.save_model(self, output_dir, _internal_call)
   2125             self.deepspeed.save_checkpoint(output_dir)
   2127 elif self.args.should_save:
-> 2128     self._save(output_dir)
   2130 # Push to the Hub when `save_model` is called by the user.
   2131 if self.args.push_to_hub and not _internal_call:

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:2180, in Trainer._save(self, output_dir, state_dict)
   2178         torch.save(state_dict, os.path.join(output_dir, WEIGHTS_NAME))
   2179 else:
-> 2180     self.model.save_pretrained(output_dir, state_dict=state_dict)
   2181 if self.tokenizer is not None:
   2182     self.tokenizer.save_pretrained(output_dir)

File /opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py:1352, in PreTrainedModel.save_pretrained(self, save_directory, save_config, state_dict, save_function, push_to_hub, max_shard_size, **kwargs)
   1350 # Save the config
   1351 if save_config:
-> 1352     model_to_save.config.save_pretrained(save_directory)
   1354 # Save the model
   1355 if state_dict is None:

File /opt/conda/lib/python3.8/site-packages/transformers/configuration_utils.py:440, in PretrainedConfig.save_pretrained(self, save_directory, push_to_hub, **kwargs)
    437 # If we save using the predefined names, we can load using `from_pretrained`
    438 output_config_file = os.path.join(save_directory, CONFIG_NAME)
--> 440 self.to_json_file(output_config_file, use_diff=True)
    441 logger.info(f"Configuration saved in {output_config_file}")
    443 if push_to_hub:

File /opt/conda/lib/python3.8/site-packages/transformers/configuration_utils.py:805, in PretrainedConfig.to_json_file(self, json_file_path, use_diff)
    794 def to_json_file(self, json_file_path: Union[str, os.PathLike], use_diff: bool = True):
    795     """
    796     Save this instance to a JSON file.
    797 
   (...)
    803             is serialized to JSON file.
    804     """
--> 805     with open(json_file_path, "w", encoding="utf-8") as writer:
    806         writer.write(self.to_json_string(use_diff=use_diff))

FileNotFoundError: [Errno 2] No such file or directory: 'saved_models_bert-finetuned-ner-100examples-with-aug/checkpoint-6/config.json'
```

This is so wired! From my understanding, the config.json file should be written, so such error shouldn't occur.


### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am not sure if this can be reproduced in another machine. btw, I am using A100.

### Expected behavior

```shell
A normal training...
```


Comments:
Sorry to issue here, I have already asked this question in the forum but haven't receive any response.

This means the folders were not created. Are you sure you point to locations where the Python script can write?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.