[`Trainer`] Force `is_model_parallel` when model is loaded in multiple GPUs using `accelerate`
Author: younesbelkada
Pull request: yes

# What does this PR do?

When using the Trainer on a multi-GPU environment, users currently apply a patch that leads to some bugs.
Before running a training they [need to call](https://github.com/huggingface/peft/issues/205#issuecomment-1491455711):
```python
setattr(model, 'model_parallel', True)
setattr(model, 'is_parallelizable', True)
```

Which can lead to unexpected bugs on some models, such as T5, that has the `parallelize` API that is still in place, thus when forcing `model_parallel` to be `True`, calls that API, which is deprecated and should not be maintained.

Script to reproduce:

```python
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoModelForSeq2SeqLM, AutoTokenizer, Trainer, TrainingArguments, DataCollatorForLanguageModeling

from peft import prepare_model_for_int8_training,LoraConfig, get_peft_model

causal_lm_model_id = "facebook/opt-350m"

model = AutoModelForCausalLM.from_pretrained(
    causal_lm_model_id,
    load_in_8bit=True,
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained(causal_lm_model_id)
model = prepare_model_for_int8_training(model)

# setattr(model, 'model_parallel', True)
# setattr(model, 'is_parallelizable', True)  

config = LoraConfig(
    r=16, 
    lora_alpha=32, 
    target_modules=["q_proj", "v_proj"], 
    lora_dropout=0.05, 
    bias="none", 
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, config)

data = load_dataset("Abirate/english_quotes")
data = data.map(lambda samples: tokenizer(samples["quote"]), batched=True)

trainer = Trainer(
    model=model,
    train_dataset=data["train"],
    args=TrainingArguments(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        warmup_steps=2,
        max_steps=3,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=1,
        output_dir="outputs",
    ),
    data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
)

trainer.is_model_parallel = True

model.config.use_cache = False
trainer.train()
```

cc @sgugger 

Related: https://github.com/huggingface/peft/issues/205


Comments:
Could you elaborate why is such a patch needed and what is the goal of your PR? Cause all of this seems very hacky.

These hacks were needed because `self.place_model_on_device` [needs to be set to `True`](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L461-L469) in order for the `Trainer` to work correctly on a multi-GPU environment, i.e. with a model that has been loaded across multiple GPUs (so we're talking about Naive PP here). Otherwise users will encounter device mismatch between model's input/output.

Moreover, modifying `place_model_on_device` directly on `TrainingArguments` seems to not work, as this argument seems to not be on the `__init__` of that class, and also it seems to me that it is better to not touch this attribute as it is a property method: https://github.com/huggingface/transformers/blob/9419f144ad6d5464afc3c9c65a23c6940f8dd9c2/src/transformers/training_args.py#L1801

That is why I preferred to introduce a new argument to avoid modifying what is already in place and modify directly what is needed to be edited, without having to modify the model's internals (forcing `model_parallel` to `True` on T5 models will call the deprecated `parallelize` API that leads to some bugs)

_The documentation is not available anymore as the PR was closed or merged._

Or you could just analyze the device map of the model and determine if there are several GPUs used. It would be cleaner and not require the user to learn the 97th training argument.

Ahh yes good point!