Saving Models Broke
Author: johnml1135
Pull request: no

### System Info

I get this error when building a hugginface NLLB model on a ClearML docker image as per this bug in my repo (https://github.com/sillsdev/machine.py/issues/14):

```
 50% 500/1000 [08:59<09:03,  1.09s/it][INFO|trainer.py:2904] 2023-05-24 13:04:36,643 >> Saving model checkpoint to /root/machine/builds/646e3f95cf5823db7b5edd92/model/checkpoint-500
Traceback (most recent call last):
  File "/root/.clearml/venvs-builds/3.8/code/untitled.py", line 11, in <module>
    run(args)
  File "/usr/local/lib/python3.8/dist-packages/machine/jobs/build_nmt_engine.py", line 56, in run
    job.run(check_canceled)
  File "/usr/local/lib/python3.8/dist-packages/machine/jobs/nmt_engine_build_job.py", line 54, in run
    model_trainer.train(check_canceled=check_canceled)
  File "/usr/local/lib/python3.8/dist-packages/machine/translation/huggingface/hugging_face_nmt_model_trainer.py", line 263, in train
    train_result = self._trainer.train(resume_from_checkpoint=ckpt)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1664, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2019, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2308, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2365, in _save_checkpoint
    self.save_model(output_dir, _internal_call=True)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2866, in save_model
    self._save(output_dir)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2922, in _save
    self.model.save_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 1734, in save_pretrained
    model_to_save.config.save_pretrained(save_directory)
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 457, in save_pretrained
    self.to_json_file(output_config_file, use_diff=True)
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 850, in to_json_file
    writer.write(self.to_json_string(use_diff=use_diff))
  File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 836, in to_json_string
    return json.dumps(config_dict, indent=2, sort_keys=True) + "\n"
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.8/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 353, in _iterencode_dict
    items = sorted(dct.items())
TypeError: '<' not supported between instances of 'str' and 'int'
```

Here is some analysis -> The normal dict has, among other things:
```
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  }
```
But after being trained (and possible ClearML doing something), it becomes:
```
  "id2label": {
    0: "LABEL_0",
    1: "LABEL_1",
    "0": "LABEL_0",
    "1": "LABEL_1"
  }
```
Which causes the sorting to break (and the above error).

**Ideas:**
* There were no labels passed to it, but labels are auto-created based upon this code:
  * If there are no id2label mapping, num_labels is set to 2: https://github.com/huggingface/transformers/blob/f67dac97bdc63874f2288546b3fa87e69d2ea1c8/src/transformers/configuration_utils.py#L319-L331
  * If num_labels is 2, then labels the above labels are created: https://github.com/huggingface/transformers/blob/f67dac97bdc63874f2288546b3fa87e69d2ea1c8/src/transformers/configuration_utils.py#L418-L421
  * Likely this code got called twice - and in between the int's got converted to strings due to label stuff happening.

What is a good path forward?



### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This is pretty custom and would not work without all the proper licenses - but to reproduce:
* Spin up the docker-compose environment from: https://github.com/sillsdev/serval
* Setup a clearml docker agent using a docker image made from the master branch from: https://github.com/sillsdev/machine.py
* Run the NmtBatch E2E test from: https://github.com/sillsdev/serval
* The error occurs in the docker container in clearml.

### Expected behavior

* Don't crash.  
* Likely, don't auto-create labels if none are given (the case here)
* Or, if you are going to convert all the numbers to strings when saving to a dictionary, account for the id2label fields properly, such as always use strings or convert back to int's when loading from a dict, etc. 

Comments:
I tested out a work around - specifically adding empty id2label's etc. and it worked.
```
AutoConfig.from_pretrained(model_name, label2id={}, id2label={}, num_labels=0)
```
This probably should have a longer term fix - possibly both in not auto-creating meaningless labels and making the save/restore not cause saving conflicts with int/str dict keys.

What is a reproducer for the first issue?

Hey @johnml1135, what version of clearml were you using?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I believe the most recent version of ClearML - though the source of the error can be seen in the code refrenced.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I hit this exact stack trace as well using the following docker setup:

`docker image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-training:2.1.0-transformers4.36.0-gpu-py310-cu121-ubuntu20.04`
`torch==2.3.0`
`transformers==4.40.2`
`accelerate==0.30.1`

Per @johnml1135's suggestion, simply commenting out the `label2id` and `id2label` references in the configs got everything working.

Also encountered with exact same issue while using Clearml. Specifically, when training model in remote execution mode with several gpus. Interesting that training in clearml interactive session works fine, the trainer saves model without any issues.

Also fixed with manually setting the mappings.

@johnml1135 thanks for saving hours of debugging time.