Training doesn't end properly but stops the machine. With no error message.
Author: ghost
Pull request: no

### System Info

- `transformers` version: 4.37.0.dev0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.11.5
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.0
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@LysandreJik @stas00 @sgugger 


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. I used own remote local machine via AnyDesk.
2. there happens some warning messages like "You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding." and "The input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in torch.float16."
3. Training goes well, but after the train, train loss is shown and suddenly the machine stops. All the programs include visual studio turn off and got disconnected from the machine. I think it automatically restarts. Saving isn't performed right, and output isn't saved right either. To be specific, .json files such as config.json or tokenizer.json and README.md are all broken. They do exist but all broken. I cannot check the safetensor file. There is nothing saved in the output_dir which is declared in training_params. 
4. To add, I ran the code with commenting out trainer.model.save_pretrained line, but still crashes. But surprisingly, .json files related to tokenizer is saved properly, not broken. PC crash still happens but by commenting out saving model line, broken file problem is partially solved.
5. Here is the code

```python
import transformers
from transformers import (BitsAndBytesConfig, AutoModelForCausalLM, AutoTokenizer, GenerationConfig, TrainingArguments, logging)
import torch
import os
from datasets import load_dataset, concatenate_datasets
import json
from peft import LoraConfig
from trl import SFTTrainer, DataCollatorForCompletionOnlyLM


def main():
	base_model = "mistralai/Mistral-7B-Instruct-v0.2"
	new_model = "Mistral-7B-Instruct-v0.2_newmodel"

	compute_dtype = getattr(torch, "float16")

	quant_config = BitsAndBytesConfig(
		load_in_4bit=True,
		bnb_4bit_quant_type="nf4",
		bnb_4bit_compute_dtype=compute_dtype,
		bnb_4bit_use_double_quant=False
	)

	tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code = True, padding_side = "right")
	model = AutoModelForCausalLM.from_pretrained(base_model, quantization_config = quant_config, attn_implementation = "flash_attention_2", device_map = {"": 0})
	model.config.use_cache = False
	model.config.pretraining_tp = 1

	if tokenizer.pad_token is None:
		tokenizer.pad_token = tokenizer.eos_token

	
	train_dataset = load_dataset('json', data_files = './dataset/mixed_train.json', split = 'train')
	eval_dataset = load_dataset('json', data_files = './dataset/mixed_val.json', split = 'train')
	print(f"train dataset size: {len(train_dataset)}, eval dataset size: {len(eval_dataset)}")

	training_params = TrainingArguments(
		output_dir="./FT_newmodel",
		num_train_epochs=1,
		per_device_train_batch_size=2,
		per_device_eval_batch_size= 1,
		evaluation_strategy='steps',
		eval_steps=25,
		gradient_accumulation_steps=4,
		optim="paged_adamw_32bit",
		logging_steps=25,
		learning_rate=2e-5,
		weight_decay=0.001,
		fp16=False,
		bf16=False,
		max_grad_norm=0.3,
		max_steps=-1,
		warmup_ratio=0.03,
		group_by_length=True,
		lr_scheduler_type="constant",
		report_to="tensorboard"
	)

	peft_config = LoraConfig(
		lora_alpha=16,
		lora_dropout=0.1,
		r=64,
		bias="none",
		target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "lm_head",
    	],
		task_type="CAUSAL_LM"
	)

	trainer = SFTTrainer(
		model = model,
		tokenizer = tokenizer,
		train_dataset = train_dataset,
		eval_dataset = eval_dataset,
		dataset_text_field= "text",
		args = training_params, 
		peft_config = peft_config,
		max_seq_length = 512,
		packing = False,
		neftune_noise_alpha = 5
	)

	trainer.train()
	trainer.model.save_pretrained(new_model)
	trainer.tokenizer.save_pretrained(new_model)


if __name__ == "__main__":
	print("Training starts")
	main()
	print("Training ended")

```
6. Funny part is that actually the machine stops after "Training ended" is printed. There is no error message, machine just stops. I really can't figure out the problem. Please help..

### Expected behavior

I just want it to end properly by saving the fine-tuned model.

Comments:
Hey 🤗  thanks for opening an issue! 
You should check [for OOM ](https://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer) issues and whether or not this is what crached your machine. If yes, you should try to use float16 or bfloat16 for training rather than full precision. Also check your memory usage during training. 

As this is related to custom code, could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help! We try to keep the github issues for bugs/feature requests 🤗 

Thanks!

Thanks for the kind comment! For OOM issue, do i have to check hard disk memory? If so, more than 300GB is left, but i guess you got the point. 
Since I'm using windows I installed bitsandbytes library compiled for windows, made by individual not official, I thought of some unknown incompatibility like thing happened, so I made new WSL2 Linux environment and run the code, which led me to "Error out of memory at line 380 in file /mmfs1/gscratch/zlab/timdettmers/git/bitsandbytes/csrc/pythonInterface.c" Does this indicate something??

+ You mean deleting this issue and upload at forum instead? If so, I'll delete this and reupload this to forum. Waiting for your answer! Thank you so much.


Memory issue is both RAM and DRAM (cpu and GPU) depending on the data processing (CPU) and the model running (GPU?) 
No need to delete the issue. 
Yes this indicates that you ran out of memory. What is the environnement that you are using? 

Now I'm using Intel(R) Xeon(R) Gold 6326 CPU, and NVIDIA GeForce RTX 4090 GPU. I'm checking the resource used in the training session, dedicated GPU memory is used about 23.5/24GB and shared GPU memory is used about 5GB/128GB. I didn't check the CPU resource. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.