Error pushing Mixtral fine-tune to hub
Author: RonanKMcGovern
Pull request: no

### System Info

- `transformers` version: 4.36.1
- Platform: Linux-5.4.0-155-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, 2x A6000
- Using distributed or parallel set-up in script?: 

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2",
    cache_dir=cache_dir
)

# Apply an adapter:
from peft import PeftModel
model = PeftModel.from_pretrained(
    model,
    adapter_dir,
)

model = model.merge_and_unload() # merge adapters with the base model.

model.push_to_hub(new_model, token=True, max_shard_size="10GB",safe_serialization=True)
```

Leads to:
```
SafetensorError                           Traceback (most recent call last)
Cell In[20], line 1
----> 1 model.push_to_hub(new_model, token=True, max_shard_size="10GB",safe_serialization=True)

File /usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:871, in PushToHubMixin.push_to_hub(self, repo_id, use_temp_dir, commit_message, private, token, max_shard_size, create_pr, safe_serialization, revision, commit_description, **deprecated_kwargs)
    868 files_timestamps = self._get_files_timestamps(work_dir)
    870 # Save all files.
--> 871 self.save_pretrained(work_dir, max_shard_size=max_shard_size, safe_serialization=safe_serialization)
    873 return self._upload_modified_files(
    874     work_dir,
    875     repo_id,
   (...)
    881     commit_description=commit_description,
    882 )

File /usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:2376, in PreTrainedModel.save_pretrained(self, save_directory, is_main_process, state_dict, save_function, push_to_hub, max_shard_size, safe_serialization, variant, token, save_peft_format, **kwargs)
   2372 for shard_file, shard in shards.items():
   2373     if safe_serialization:
   2374         # At some point we will need to deal better with save_function (used for TPU and other distributed
   2375         # joyfulness), but for now this enough.
-> 2376         safe_save_file(shard, os.path.join(save_directory, shard_file), metadata={"format": "pt"})
   2377     else:
   2378         save_function(shard, os.path.join(save_directory, shard_file))

File /usr/local/lib/python3.10/dist-packages/safetensors/torch.py:281, in save_file(tensors, filename, metadata)
    250 def save_file(
    251     tensors: Dict[str, torch.Tensor],
    252     filename: Union[str, os.PathLike],
    253     metadata: Optional[Dict[str, str]] = None,
    254 ):
    255     """
    256     Saves a dictionary of tensors into raw bytes in safetensors format.
    257 
   (...)
    279     ```
    280     """
--> 281     serialize_file(_flatten(tensors), filename, metadata=metadata)

SafetensorError: Error while serializing: IoError(Os { code: 28, kind: StorageFull, message: "No space left on device" })
```

Even though I'm only using 31% of 600 GB of disk space locally.
```

### Expected behavior

Typically, safetensors push successfully.

Comments:
cc @Narsil 

As the error mentions, `... StorageFull, message: "No space left on device" })` you don't have enough space on the device. Make sur to read the stacktrace it usuallu helps a lot 🤗 

@ArthurZucker that's true - but it doesn't seem consistent with the space that @RonanKMcGovern is reporting is left on the device. If the model is the same size as [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/tree/main), then we'd expect ~100GB required for the safetensor shards. Let's say it's 190GB if all of the currently used memory (600 * 0.31) is just the downloaded checkpoint.  I'm not sure of the additional requirements PEFT applies - is it expected to increase the memory requirements of the model by more than 2x? 


Yeah, that's what I found @amyeroberts . Originally I tried with only 200 GB of total disk space, so it seemed plausible I was out of space if I had saved another copy of the model, but with 600 GB of space, that seemed more than enough and yet I still got the error.

BTW, the adapter is merged to the model, so I would have thought the PEFT portion is no longer relevant at that point.

@RonanKMcGovern Could you try with a small model that will fit many times into your memory and see whether you still see a large explosion in the memory footprint? 

Yes. I have run this script for many models like Llama 7B , openchat 3.5, deepseek models etc.

In this case I solved the issue with using upload files to hf.

> @ArthurZucker that's true - but it doesn't seem consistent with the space that @RonanKMcGovern is reporting is left on the device. If the model is the same size as [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/tree/main), then we'd expect ~100GB required for the safetensor shards. Let's say it's 190GB if all of the currently used memory (600 * 0.31) is just the downloaded checkpoint. I'm not sure of the additional requirements PEFT applies - is it expected to increase the memory requirements of the model by more than 2x?

Do you mean Nvidia's graphics memory or do you mean computer memory or disk storage?

Hi @zysNLP I meant disk storage.

But yes, perhaps that's the issue - insufficient space on the gpu to push. Although I did previously run LoRA bf16 fine-tuning and inference.

Closing this out, I haven't been able to test it, but I think the error is GPU out of memory, not disk - probably what Arthur was initially implying.

I was on 2x A6000s so the VRAM is definitely close to the total Mixtral model size, and trying to push a loaded model may have been too much. I assume that was the issue, if I find different later, I'll reopen this.

Hmm, I having the same issue with DeepSeek Coder 33B now. Seems like a safetensors issue when serializing (I've tried to do this with the model on cpu and on gpu...):
```
SafetensorError                           Traceback (most recent call last)
Cell In[7], line 1
----> 1 model.push_to_hub(new_model, token=True, max_shard_size="10GB",safe_serialization=True)

File /usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:871, in PushToHubMixin.push_to_hub(self, repo_id, use_temp_dir, commit_message, private, token, max_shard_size, create_pr, safe_serialization, revision, commit_description, **deprecated_kwargs)
    868 files_timestamps = self._get_files_timestamps(work_dir)
    870 # Save all files.
--> 871 self.save_pretrained(work_dir, max_shard_size=max_shard_size, safe_serialization=safe_serialization)
    873 return self._upload_modified_files(
    874     work_dir,
    875     repo_id,
   (...)
    881     commit_description=commit_description,
    882 )

File /usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:2376, in PreTrainedModel.save_pretrained(self, save_directory, is_main_process, state_dict, save_function, push_to_hub, max_shard_size, safe_serialization, variant, token, save_peft_format, **kwargs)
   2372 for shard_file, shard in shards.items():
   2373     if safe_serialization:
   2374         # At some point we will need to deal better with save_function (used for TPU and other distributed
   2375         # joyfulness), but for now this enough.
-> 2376         safe_save_file(shard, os.path.join(save_directory, shard_file), metadata={"format": "pt"})
   2377     else:
   2378         save_function(shard, os.path.join(save_directory, shard_file))

File /usr/local/lib/python3.10/dist-packages/safetensors/torch.py:281, in save_file(tensors, filename, metadata)
    250 def save_file(
    251     tensors: Dict[str, torch.Tensor],
    252     filename: Union[str, os.PathLike],
    253     metadata: Optional[Dict[str, str]] = None,
    254 ):
    255     """
    256     Saves a dictionary of tensors into raw bytes in safetensors format.
    257 
   (...)
    279     ```
    280     """
--> 281     serialize_file(_flatten(tensors), filename, metadata=metadata)

SafetensorError: Error while serializing: IoError(Os { code: 28, kind: StorageFull, message: "No space left on device" })
```

I don't recall this issue before and I have pushed many models of this size.

@RonanKMcGovern Could you test with a small model and check the memory utilization? Does this still occur if `safe_serialization=False` in the push_to_hub call? 

Yes, working fine with smaller models, suggesting it is indeed out of memory somehow. I'll revert the next time I run a larger model again trying to push with safetensors false.

Ok, I found the issue. The shard size I had set for pushing to hub was larger than my docker container size... I set my container size to larger than the shard size to push and everything works. It just happens that for bigger models I was using bigger shards, hence why I had issues for larger models.

Thanks for the help. I didn't appreciate how this worked.