Repository: huggingface/trl

Issue:
Unable to use `ref_model=None` with DPOTrainer
Hi,

While trying to run DPO with `ref_model=None` ([PR #640](https://github.com/huggingface/trl/pull/640)), I ran into a couple of issues.

Following the Stack-Llama-2 example, `dpo_llama2.py` runs ok with all the default settings. However, this example loads the active and ref model separately and passes both to the `DPOTrainer`. 

If I understand correctly, setting `ref_model=None`, should avoid copying the model and instead disable the adapters to compute the reference model score. But it seems like [this check](https://github.com/huggingface/trl/blob/d3bbee3ab84d4d5dcb413003bf93541df5b4e54c/trl/trainer/dpo_trainer.py#L116) consistently fails and the model gets copied in any case. This also produces the following warning with no updates to the loss:

```python
 UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
```

To reproduce, replace https://github.com/huggingface/trl/blob/d3bbee3ab84d4d5dcb413003bf93541df5b4e54c/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py#L141C5-L146C6 with `model_ref=None`.

---

Libraries:

```
accelerate               0.21.0
bitsandbytes             0.41.1
peft                     0.5.0
torch                    2.0.1
torchvision              0.15.2
transformers             4.33.0.dev0
trl                      0.6.1.dev0
```

Any insights on this would be greatly appreciated!


Discussion:
Hi @tannonk 
Thanks for the issue, 
I am suspecting the model does not set the input to `requires_grad=True` as expected, can you try to call:

```python
model.enable_input_require_grads()
```

Right before passing the model to the trainer?
More details on what the method does: https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L1224C16-L1224C16
Hi @younesbelkada

Thanks for the response! I tried out your suggestion but unfortunately it doesn't seem to fix the issue.
@tannonk so i can do the PR now if that is ok?

Resolving patch:
diff --git a/trl/trainer/dpo_trainer.py b/trl/trainer/dpo_trainer.py
--- a/trl/trainer/dpo_trainer.py
+++ b/trl/trainer/dpo_trainer.py
@@ -29,7 +29,7 @@
 
 
 if is_peft_available():
-    from peft import get_peft_model, prepare_model_for_int8_training
+    from peft import PeftModel, get_peft_model, prepare_model_for_int8_training
 
 
 class DPOTrainer(Trainer):
@@ -113,7 +113,7 @@ def __init__(
                 model = prepare_model_for_int8_training(model)
             model = get_peft_model(model, peft_config)
 
-        self.is_peft_model = getattr(model, "is_peft_model", False)
+        self.is_peft_model = is_peft_available() and isinstance(model, PeftModel)
 
         if ref_model:
             self.ref_model = ref_model
@@ -197,10 +197,7 @@ def __init__(
             )
 
         if self.ref_model is None:
-            if not hasattr(
-                self.accelerator.unwrap_model(self.model).pretrained_model,
-                "disable_adapter",
-            ):
+            if not hasattr(self.accelerator.unwrap_model(self.model), "disable_adapter"):
                 raise ValueError(
                     "You are using a `peft` version that does not support `disable_adapter`. Please update your `peft` version to the latest version."
                 )
@@ -347,7 +344,7 @@ def get_batch_metrics(
         ) = self.concatenated_forward(model, batch)
         with torch.no_grad():
             if self.ref_model is None:
-                with self.accelerator.unwrap_model(self.model).pretrained_model.disable_adapter():
+                with self.accelerator.unwrap_model(self.model).disable_adapter():
                     (
                         reference_chosen_logps,
                         reference_rejected_logps,
@@ -415,7 +412,7 @@ def get_batch_samples(self, model, batch: Dict[str, torch.LongTensor]) -> Tuple[
         )
 
         if self.ref_model is None:
-            with self.accelerator.unwrap_model(self.model).pretrained_model.disable_adapter():
+            with self.accelerator.unwrap_model(self.model).disable_adapter():
                 reference_output = self.model.generate(
                     batch["prompt_input_ids"],
                     attention_mask=batch["prompt_attention_mask"],


Test patch: