trainer.train(resume_from_checkpoint=True) failed
Author: John-Lin98
Pull request: no

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.28.1
- Platform: Linux-4.15.0-209-generic-x86_64-with-glibc2.27
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger
When I call trainer.train() to continue training a llama-7B model from a checkpoint, I encounter the following issue:
![image](https://github.com/huggingface/transformers/assets/65805703/e121ff01-601f-4891-b5ca-02cbf1d1dab0)
![image](https://github.com/huggingface/transformers/assets/65805703/ffb3c1d0-d880-45da-891a-fbb94417e6c0)
![image](https://github.com/huggingface/transformers/assets/65805703/0721553f-1d9c-4d5b-9204-4c0e5119c402)

And I'm not sure why this problem is occurring. Here is the code I'm running:
![image](https://github.com/huggingface/transformers/assets/65805703/0f6feafa-4779-4065-bfbe-152abb30d7a4)


### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

RuntimeError: Trying to resize storage that is not resizable

Here is the code I'm running:

```
def train():
    global local_rank

    parser = transformers.HfArgumentParser(
        (ModelArguments, DataArguments, TrainingArguments))
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
    local_rank = training_args.local_rank
    model = transformers.AutoModelForCausalLM.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=training_args.cache_dir,
    ).half()
    tokenizer = transformers.AutoTokenizer.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=training_args.cache_dir,
        model_max_length=training_args.model_max_length,
        padding_side="right",
        use_fast=False,
    )
    tokenizer.pad_token = tokenizer.unk_token

    data_module = make_supervised_data_module(tokenizer=tokenizer,
                                              data_args=data_args)
    trainer = Trainer(model=model,
                      tokenizer=tokenizer,
                      args=training_args,
                      **data_module)

    if list(pathlib.Path(training_args.output_dir).glob("checkpoint-*")):
        trainer.train(resume_from_checkpoint=True)
    else:
        trainer.train()
    trainer.save_state()
    trainer.save_model(training_args.output_dir)
```

### Expected behavior

I don't encounter any checkpoint import error when I continue training from a checkpoint.

Comments:
The saved checkpoint is corrupted somehow. I don't know what could be the reason for it since I don't know how it was saved in the first place.

sorry for not showing the training parameters earlier. In fact, I used the trainer's automatic checkpoint saving method based on the number of steps. Here, it is set to save every 1200 steps：
![image](https://github.com/huggingface/transformers/assets/65805703/db83cecd-8af8-4d2f-a45f-f597e177f970)

Here is the directory where I save my checkpoints：
![image](https://github.com/huggingface/transformers/assets/65805703/6f2e2c0b-8f43-4dce-8fb5-599ffdb5f408)




This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.