RuntimeError "Some tensors share memory" occurred when saving checkpoints during LoRA
Author: Yuta555
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.10.0-26-cloud-amd64-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:
- Using distributed or parallel set-up in script?: 

### Who can help?

@muellerzr @pacman

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm fine-tuning Llama 2 7B (ForSequenceClassification) with LoRA as shown below, then got an error when I set target_modules as all linear layers for LoRA while didn't get the error when setting only q and v.
What will cause this error to occur and how can I avoid it? If I can follow this error message to avoid the problem, how can I use `save_model` in the trainer setting?

Error:
```
RuntimeError: 
            Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: [{'base_model.model.score.lora_A.weight', 'base_model.model.score.modules_to_save.lora_A.weight'}, {'base_model.model.score.modules_to_save.lora_B.weight', 'base_model.model.score.lora_B.weight'}].
            A potential way to correctly save your model is to use `save_model`.
            More information at https://huggingface.co/docs/safetensors/torch_shared_tensors
```

Code:
```
# Build a model and set configuration for LoRA
model = AutoModelForSequenceClassification.from_pretrained(
    model_name_or_path,
    torch_dtype=torch.bfloat16,
    num_labels=len(label2id),
    id2label=id2label,
    label2id=label2id,
    device_map="auto",
    use_flash_attention_2=True,
)
if getattr(model.config, "pad_token_id") is None:
    model.config.pad_token_id = model.config.eos_token_id

# Pik up all linear layer 
model_modules = str(model.modules)
pattern = r'\((\w+)\): Linear'
linear_layer_names = re.findall(pattern, model_modules)

names = []
# Print the names of the Linear layers
for name in linear_layer_names:
    names.append(name)
target_modules = list(set(names))

print(f"target modules for LoRA: {target_modules}")

config = LoraConfig(
    task_type="SEQ_CLS",
    inference_mode=False,
    r=16,
    lora_alpha=16,
    lora_dropout=0.1,
    target_modules=target_modules,
)

model = get_peft_model(model, config)`

# Set Trainer Arguments
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=8,    # increase by 2x for every 2x decrease in batch size
    #num_train_epochs=NUM_EPOCHS,
    learning_rate=LR,
    bf16=True,
    warmup_steps=WARMUP_STEPS,
    logging_steps=5, # For check
    save_steps=5, # For check
    save_safetensors=True,
    evaluation_strategy="steps",
    max_steps=9, # For check
    remove_unused_columns=False,    
    label_names=["labels"],           
    group_by_length=True,              
    lr_scheduler_type=LR_SCHEDULER,     
    ddp_find_unused_parameters=False,
    report_to="wandb",
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    greater_is_better=True,    
)

data_collator = DataCollatorWithPadding(tokenizer, padding="longest")
#metrics = evaluate.load("accuracy")
acc = evaluate.load('accuracy')

# Added f1, precision and recall as reference
# Set average method as "macro" since dataset is imbalanced but consider each class eqaully important
f1 = evaluate.load('f1', average='macro')
pre = evaluate.load('precision', average='macro')
rec = evaluate.load('recall', average='macro')

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    #return metrics.compute(predictions=predictions, references=labels)
    results = {}
    results.update(acc.compute(predictions=predictions, references=labels))
    results.update(f1.compute(predictions=predictions, references=labels, average="macro"))
    results.update(pre.compute(predictions=predictions, references=labels, average="macro"))
    results.update(rec.compute(predictions=predictions, references=labels, average="macro"))
    return results

trainer = Trainer(
    args=training_args,
    model=model,
    train_dataset=train_data["train"],
    eval_dataset=train_data["test"].train_test_split(test_size=0.05)['test'],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
    callbacks=[],
)

trainer.train()
```



### Expected behavior

Save checkpoints successfully without the error above.

Comments:
cc @LysandreJik, should we do what we did similar for accelerate where we just warned instead? 

What we did in accelerate: https://github.com/huggingface/accelerate/pull/2136 cc @SunMarc 

cc @Narsil also

I've also hit this, and it seems like a backwards incompatible change from `4.33.2`.  I appreciate that the error message links to a [page](https://huggingface.co/docs/safetensors/torch_shared_tensors) with clear instructions.  But the instructions say to replace `save_file` with `save_model` which would need to be done [deep in the Trainer class](https://github.com/huggingface/transformers/blob/df5c5c62ae253055336f5bb0828ca8e3e15ab6bd/src/transformers/trainer.py#L2893) for those of us using `Trainer`.

Looking at the code I see there's a workaround to set `save_safetensors=False`.  Seems to work, but then we lose whatever benefits safetensors are supposed to give us.  (TF compatibility - I'm sure somebody cares, but nobody I know.)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey! Could you try again with main, I think the compatibility has been taken care of! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@muellerzr @ArthurZucker I also met this problem while saving the adapter for the auto-gptq quantized models. Any solutions? 

Not really sure, would recommend you to open a new issue and ping @SunMarc there!

I've also come across this problem and so it would be good to reopen it. In my case it is occuring when using a third party repo for post-merge training (DAM), and I would like to be able to use that training system without forking it. Can we reopen this? Thanks!

NO, there is not point in re-opening as the provided script was fixed. 
You should open a new issue with a reproducer! This way we can focus on your issue 🤗 

> NO, there is not point in re-opening as the provided script was fixed. You should open a new issue with a reproducer! This way we can focus on your issue 🤗

@ArthurZucker Could you please point to the script that solves this problem? :)

The script from the issue now works AFAIK!