internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: apple/coremltools

Issue:
Typo in error message text. of->or
## 🐞Describe the bug
typo in error message "Dimension cannot be None of -1"
it should be "Dimension cannot be None or -1"
`of` should be replaced with `or`

## Trace
code line https://github.com/apple/coremltools/blob/master/coremltools/converters/mil/input_types.py#L296

## To Reproduce
Try to convert TF mobilenet with shape `(-1, 224, 224,3)`

- If a python script can reproduce the error, please paste the code snippet
```
# Paste code snippet here
```
- If applicable, please attach the source model
    - If the model cannot be shared publicly, please attach it via filing a bug report at https://developer.apple.com/bug-reporting/  and provide the reference number here
- If it is a model conversion issue and the conversion succeeds, however, if there is a numerical mismatch between the original and the coreml model, please paste script used for comparison.

## System environment (please complete the following information):
 - coremltools version  (e.g., 3.0b5): 4.0b3
 - OS (e.g., MacOS, Linux): Linux
 - macOS version (if applicable):
 - XCode version (if applicable):
 - How you install python (anaconda, virtualenv, system):
 - python version (e.g. 3.7): 3.7
 - any other relevant information:
     - e.g. keras version if using keras conversion etc.

## Additional context
Add any other context about the problem here.



Discussion:
Related PR #858 (where the bug was introduced)

I asked @bhushan23 to look at the issue but no reply
@apivovarov  The API for the coremltools 4.0b3 is changed,
now you need to use ct.RangeDim() instead if -1.
Please reopen the issue if this doesn't solve your issue.

Resolving patch:
diff --git a/coremltools/converters/mil/input_types.py b/coremltools/converters/mil/input_types.py
--- a/coremltools/converters/mil/input_types.py
+++ b/coremltools/converters/mil/input_types.py
@@ -293,7 +293,7 @@ def __init__(self, shape, default=None):
         shape = list(shape)
         for idx, s in enumerate(shape):
             if s is None or s == -1:
-                msg = 'Dimension cannot be None of -1. Use ' +\
+                msg = 'Dimension cannot be None or -1. Use ' +\
                         'ct.RangeDim for runtime determined dimension. ' +\
                         'Dim {}: {} ' +\
                         'See https://coremltools.readme.io/docs/flexible-inputs'


Test patch:
Repository: apple/coremltools

Issue:
cuda tensor parameter fail to convert to numpy in InternalTorchIRGraph 
## 🐞Describe the bug
- If the input parameter type to a traced model is tensor.cuda(),  ct.convert fails with the below error
- Torch

## Trace
```
  File "/home/josh/anaconda3/lib/python3.7/site-packages/coremltools/converters/mil/frontend/torch/internal_graph.py", line 180, in __init__
    value = param.detach().numpy()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

```
Note, possible fix:

replace
`    value = param.detach().numpy()
`
with
`    value = param.cpu().detach().numpy()
`


## System environment (please complete the following information):
 - coremltools version:  4.0b1
 - OS: Linux
 - How you install python: anaconda
 - python version: 3.7.6



## Additional context
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/coremltools/converters/mil/frontend/torch/internal_graph.py b/coremltools/converters/mil/frontend/torch/internal_graph.py
--- a/coremltools/converters/mil/frontend/torch/internal_graph.py
+++ b/coremltools/converters/mil/frontend/torch/internal_graph.py
@@ -246,7 +246,7 @@ def __init__(
 
             # Add params
             for name, param in params_dict.items():
-                value = param.detach().numpy()
+                value = param.detach().cpu().numpy()
                 self.params[name] = value
 
             # Add inputs


Test patch:
Repository: huggingface/trl

Issue:
adap_kl_ctrl boolean is not used 
Hello,

Thanks for this implementation, is it possible that your kl adaptation is always on ?
https://github.com/lvwerra/trl/blob/750f5fd5329bb81c79b00243c4c8923ac14981d5/trl/ppo.py#L92

Best,
Thibaud


Discussion:
Yes, that is true - well spotted! I'll add it as a TODO.

Resolving patch:
diff --git a/trl/ppo.py b/trl/ppo.py
--- a/trl/ppo.py
+++ b/trl/ppo.py
@@ -102,9 +102,12 @@ def __init__(self, model, ref_model, **ppo_params):
         self.model = model
         self.optimizer = Adam(model.parameters(), lr=self.ppo_params['lr'])
 
-        self.kl_ctl = AdaptiveKLController(self.ppo_params['init_kl_coef'],
-                                           self.ppo_params['target'],
-                                           self.ppo_params['horizon'])
+        if self.ppo_params['adap_kl_ctrl']:
+            self.kl_ctl = AdaptiveKLController(self.ppo_params['init_kl_coef'],
+                                               self.ppo_params['target'],
+                                               self.ppo_params['horizon'])
+        else:
+            self.kl_ctl = FixedKLController(self.ppo_params['init_kl_coef'])
 
 
     def step(self, query, response, scores):


Test patch:
Repository: huggingface/trl

Issue:
[Bug] when using accelerator + deepspeed, AcceleratorState class instantiated twice with different config
When run `accelerate launch` (with deepspeed), the following errors will appear:

<img width="1344" alt="image" src="https://user-images.githubusercontent.com/3998421/228439528-a81196e7-68f0-404c-8ddd-cf7791d0882e.png">

I don't know why this error occurred. It was working fine without using deepspeed. Two places of initialization:

- The second initialization occurs during `AutoModelForCausalLMWithValueHead.from_pretrained(pretrained_model)`: https://github.com/lvwerra/trl/blob/237eb9c6a5b1f90f6ccb674269bd7a33533e4bf7/trl/models/modeling_base.py#L256
- The second initialization occurs during `PPOTrainer()`: https://github.com/lvwerra/trl/blob/237eb9c6a5b1f90f6ccb674269bd7a33533e4bf7/trl/trainer/ppo_trainer.py#L181

Current workaround: reset AcceleratorState after from_pretrained
```python
    model = AutoModelForCausalLMWithValueHead.from_pretrained(pretrained_model)
    AcceleratorState._reset_state()
```



Discussion:
After make `accelerate launch` work, If it is fp16 training, you also need to modify the mini_batch_dict in `step()`, because datasets do not support fp16 and will report the following error:

<img width="766" alt="截屏2023-03-29 14 16 07" src="https://user-images.githubusercontent.com/3998421/228444410-58bf705e-fee6-4582-8d9c-2ad4d53620b1.png">

Current workaround:
https://github.com/lvwerra/trl/blob/237eb9c6a5b1f90f6ccb674269bd7a33533e4bf7/trl/trainer/ppo_trainer.py#L595

```python
mini_batch_dict = {
            "queries": queries,
            "responses": responses,
            "logprobs": all_logprobs.to(torch.float32),
            "values": values.to(torch.float32),
            "rewards": rewards,
            "masks": masks,
        }
```

Resolving patch:
diff --git a/trl/trainer/ppo_trainer.py b/trl/trainer/ppo_trainer.py
--- a/trl/trainer/ppo_trainer.py
+++ b/trl/trainer/ppo_trainer.py
@@ -592,11 +592,12 @@ def step(
         rewards, non_score_reward = self.compute_rewards(scores, all_logprobs, ref_logprobs, masks)
         timing["time/ppo/compute_rewards"] = time.time() - t
 
+        # upcast to float32 to avoid dataset issues
         mini_batch_dict = {
             "queries": queries,
             "responses": responses,
-            "logprobs": all_logprobs,
-            "values": values,
+            "logprobs": all_logprobs.to(torch.float32),
+            "values": values.to(torch.float32),
             "rewards": rewards,
             "masks": masks,
         }


Test patch:
Repository: huggingface/trl

Issue:
t5-sentiment example collapses on master
I just rerun the t5-sentiment example and rn on master it has negative kl divergence in new version and does not learn in general.
This seems not have happened since the v0.4.1 release.

The t5-sentiment.py script itself does not seem to be the culprit as i tested master with it reverted to v0.4.1 version and the behavior is identical. 

![image](https://user-images.githubusercontent.com/17483828/228250281-c6ab4de0-e689-4243-9658-39b36f199351.png)



Discussion:
@younesbelkada can you reproduce? negative KL is very suspicious!
I was able to reproduce, will investigate!
The culprit seems to be b5cce0d13e95c8e21eb9d57177930253d9092a02 
This PR introduced batched generation on the t5 example, as it can be observed on the [wandb log](https://wandb.ai/distill-bloom/trl/runs/kbj6ts85?workspace=user-) the kl is negative, I can confirm the KL was always positive before that commit

Resolving patch:
diff --git a/examples/sentiment/scripts/t5-sentiment.py b/examples/sentiment/scripts/t5-sentiment.py
--- a/examples/sentiment/scripts/t5-sentiment.py
+++ b/examples/sentiment/scripts/t5-sentiment.py
@@ -148,8 +148,7 @@ def collater(data):
     response_tensors = ppo_trainer.generate(
         query_tensors, return_prompt=False, length_sampler=output_length_sampler, **generation_kwargs
     )
-    response_tensors = [r[1:] for r in response_tensors]
-    batch["response"] = tokenizer.batch_decode(response_tensors)
+    batch["response"] = tokenizer.batch_decode([r[1:] for r in response_tensors])
 
     # Compute sentiment score
     texts = [q + r for q, r in zip(batch["query"], batch["response"])]
diff --git a/trl/trainer/ppo_trainer.py b/trl/trainer/ppo_trainer.py
--- a/trl/trainer/ppo_trainer.py
+++ b/trl/trainer/ppo_trainer.py
@@ -417,7 +417,8 @@ def _generate_batched(
         outputs = []
 
         padding_side_default = self.tokenizer.padding_side
-        self.tokenizer.padding_side = "left"
+        if not self.is_encoder_decoder:
+            self.tokenizer.padding_side = "left"
 
         # in case we have fewer examples than bs
         batch_size = min(len(query_tensors), batch_size)
@@ -444,7 +445,11 @@ def _generate_batched(
             generations = self.accelerator.unwrap_model(self.model).generate(**padded_inputs, **generation_kwargs)
 
             for generation, mask in zip(generations, padded_inputs["attention_mask"]):
-                output = generation[(1 - mask).sum() :]  # remove padding
+                if not self.is_encoder_decoder:
+                    output = generation[(1 - mask).sum() :]  # remove padding
+                else:
+                    output = generation
+
                 if not return_prompt and not self.is_encoder_decoder:
                     output = output[(mask).sum() :]  # remove prompt
                 outputs.append(output)


Test patch:
Repository: huggingface/trl

Issue:
Unavailable not pass Dataset, because of warning issue
If I would not pass Dataset to PPOTrainer, than it cause a warning:
![image](https://github.com/lvwerra/trl/assets/57448108/b135d184-7dd0-4ec4-98cf-6bfec4d20660)

It lead to error:
![image](https://github.com/lvwerra/trl/assets/57448108/ba309d26-5f45-488b-9746-8cd96fee4935)




Discussion:

Resolving patch:
diff --git a/trl/trainer/ppo_trainer.py b/trl/trainer/ppo_trainer.py
--- a/trl/trainer/ppo_trainer.py
+++ b/trl/trainer/ppo_trainer.py
@@ -233,9 +233,9 @@ def __init__(
             self.dataloader = self.prepare_dataloader(self.dataset, data_collator)
         elif self.dataset is None and self.accelerator.num_processes > 1:
             warnings.warn(
-                "No dataset is provided. In a multi-GPU setting, this will lead to an error. You should",
-                " prepare your dataloader yourself with `dataloader = ppo_trainer.accelerator.prepare(dataloader)`",
-                " and using `torch.utils.data.DataLoader`, or pass a dataset to the `PPOTrainer`. Please ",
+                "No dataset is provided. In a multi-GPU setting, this will lead to an error. You should"
+                " prepare your dataloader yourself with `dataloader = ppo_trainer.accelerator.prepare(dataloader)`"
+                " and using `torch.utils.data.DataLoader`, or pass a dataset to the `PPOTrainer`. Please "
                 " refer to the documentation for more details.",
                 UserWarning,
             )


Test patch:
Repository: huggingface/trl

Issue:
SFTTrainer: passing a PeftModel as a model argument without passing the config breaks
[https://github.com/lvwerra/trl/blob/main/trl/trainer/sft_trainer.py#L127C36-L127C47](https://github.com/lvwerra/trl/blob/main/trl/trainer/sft_trainer.py#L127C36-L127C47) Requires that you pass a PEFT config even though you've already instantiated the PeftModel yourself, otherwise it tries ignores the if and tries to build a regular transformers model at https://github.com/lvwerra/trl/blob/main/trl/trainer/sft_trainer.py#L148 taking your PeftModel as argument, which makes no sense.

I suggest adding a check at https://github.com/lvwerra/trl/blob/main/trl/trainer/sft_trainer.py#L147 to make sure that not isinstance(model, PeftModel):

```python
elif not isinstance(model, PreTrainedModel) and not isinstance(model, PeftModel):
```




Discussion:
This would let the user of the library instantiate their own peftmodel, which looks like it's not possible right now

Resolving patch:
diff --git a/trl/trainer/sft_trainer.py b/trl/trainer/sft_trainer.py
--- a/trl/trainer/sft_trainer.py
+++ b/trl/trainer/sft_trainer.py
@@ -144,7 +144,7 @@ def __init__(
 
             if callbacks is None:
                 callbacks = [PeftSavingCallback]
-        elif not isinstance(model, PreTrainedModel):
+        elif not isinstance(model, (PreTrainedModel, PeftModel)):
             model = AutoModelForCausalLM.from_pretrained(model)
 
         if tokenizer is None:


Test patch:
Repository: huggingface/trl

Issue:
DPO evaluation error---tensors on two devices
Hi!

Thanks for the awesome codebase. I ran the DPO example `trl/examples/dpo.py` but encountered an error at the evaluation step: `Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!`

Here is a [colab notebook](https://colab.research.google.com/drive/11AVym7U3gkTn_qTfrnSDtA_AO5_zJkkd?usp=sharing) that shows this problem. To expose the problem faster, I set `training_args.eval_steps = 1`.

To solve it, a hotfix can be adding `.to(self.accelerator.device)` at a few places in [concatenated_forward](https://github.com/lvwerra/trl/blob/main/trl/trainer/dpo_trainer.py#L288-L296):

```python
all_logits = model(
    concatenated_batch["concatenated_input_ids"].to(self.accelerator.device),
    attention_mask=concatenated_batch["concatenated_attention_mask"].to(self.accelerator.device),
).logits.to(torch.float32)
all_logps = self._get_batch_logps(
    all_logits,
    concatenated_batch["concatenated_labels"].to(self.accelerator.device),
    average_log_prob=False,
)
```

However, I am not sure why the trainer does not handle the device change automatically. If this hotfix is fine, I can submit a pull request. Otherwise, I'm also happy to learn how to address this problem more generically.

Tianlin





Discussion:
Hi @liutianlin0121 
Thanks very much for the clean reproducer, this sounds like definitely the right fix. Would you mind contributing and opening a PR so that the community can benefit from the fix🙏 
Thanks very much!
also cc @kashif 

Resolving patch:
diff --git a/trl/trainer/dpo_trainer.py b/trl/trainer/dpo_trainer.py
--- a/trl/trainer/dpo_trainer.py
+++ b/trl/trainer/dpo_trainer.py
@@ -204,7 +204,7 @@ def concatenated_inputs(self, batch: Dict[str, Union[List, torch.LongTensor]]) -
                         pad_to_length(batch[k], max_length, pad_value=pad_value),
                     ),
                     dim=0,
-                )
+                ).to(self.accelerator.device)
         return concatenated_batch
 
     def dpo_loss(


Test patch:
Repository: huggingface/trl

Issue:
StackLLAMA examples uses sentiment model as reward model
The script to train the StackLLAMA model uses a sentiment pipeline for training the reward model while it should use the model that was trained before. 

https://github.com/lvwerra/trl/blob/main/examples/stack_llama/scripts/rl_training.py#L216


Discussion:
The "sentiment" pipeline is not training but an inference pipeline that outputs the scalar reward. It does use the reward model trained from the previous step.
Understood! Does it help if I add a comment to explain? 
Feel free to add a comment to clarify. Closing the issue.

Resolving patch:
diff --git a/examples/research_projects/stack_llama/scripts/rl_training.py b/examples/research_projects/stack_llama/scripts/rl_training.py
--- a/examples/research_projects/stack_llama/scripts/rl_training.py
+++ b/examples/research_projects/stack_llama/scripts/rl_training.py
@@ -92,6 +92,7 @@ class ScriptArguments:
 
 train_dataset = load_dataset("lvwerra/stack-exchange-paired", data_dir="data/rl", split="train")
 train_dataset = train_dataset.select(range(100000))
+
 # We then define the arguments to pass to the sentiment analysis pipeline.
 # We set `return_all_scores` to True to get the sentiment score for each token.
 sent_kwargs = {
@@ -207,9 +208,9 @@ def collator(data):
     optimizer=optimizer,
 )
 
-# We then build the sentiment analysis pipeline, passing the model name and the
-# sentiment analysis pipeline arguments. Let's also make sure to set the device
-# to the same device as the PPOTrainer.
+# We then build the sentiment analysis pipeline using our reward model, passing the
+# model name and the sentiment analysis pipeline arguments. Let's also make sure to
+# set the device to the same device as the PPOTrainer.
 device = ppo_trainer.accelerator.device
 if ppo_trainer.accelerator.num_processes == 1:
     device = 0 if torch.cuda.is_available() else "cpu"  # to avoid a ` pipeline` bug
@@ -251,7 +252,7 @@ def collator(data):
     )
     batch["response"] = tokenizer.batch_decode(response_tensors, skip_special_tokens=True)
 
-    # Compute sentiment score
+    # Compute reward score (using the sentiment analysis pipeline)
     texts = [q + r for q, r in zip(batch["query"], batch["response"])]
     pipe_outputs = sentiment_pipe(texts, **sent_kwargs)
     rewards = [torch.tensor(output[0]["score"] - script_args.reward_baseline) for output in pipe_outputs]


Test patch:
Repository: huggingface/trl

Issue:
Tutorial script missing eval_batch_size
Following this tutorial https://huggingface.co/blog/dpo-trl, I ran into out of memory issues with the DPO script. The problem was that the script missed copying per_device_eval_batch_size to Training Arguments [here](https://github.com/huggingface/trl/blob/main/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py#L166C12-L166C12). This makes it use the default size of 8 for eval_batch_size instead of 1 as set in this script. The below addition fixes the issue.

```
training_args = TrainingArguments(
        per_device_train_batch_size=script_args.per_device_train_batch_size,
-->        per_device_eval_batch_size=script_args.per_device_eval_batch_size,
```


Discussion:
Would you mind opening a PR?
Happy to do it, but I am getting a permission error. This is my first time using github (though I am a long time git user). I created a personal access token (classic) on github and used that as the password (with my username), but that didn't work. Do I need a specific access token for the trl repo?

```
git push -u origin rahuljha/fix_arg_setting
Username for 'https://github.com': rahuljha
Password for 'https://rahuljha@github.com': 
remote: Permission to huggingface/trl.git denied to rahuljha.
fatal: unable to access 'https://github.com/huggingface/trl.git/': The requested URL returned error: 403
```
Reading through some more docs, looks like I need to first fork the repo. Will do this and try creating a pull request.

Resolving patch:
diff --git a/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py b/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py
--- a/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py
+++ b/examples/research_projects/stack_llama_2/scripts/dpo_llama2.py
@@ -164,6 +164,7 @@ def return_prompt_and_responses(samples) -> Dict[str, str]:
     # 4. initialize training arguments:
     training_args = TrainingArguments(
         per_device_train_batch_size=script_args.per_device_train_batch_size,
+        per_device_eval_batch_size=script_args.per_device_eval_batch_size,
         max_steps=script_args.max_steps,
         logging_steps=script_args.logging_steps,
         save_steps=script_args.save_steps,


Test patch:
Repository: huggingface/trl

Issue:
Bug in dpo_trainer.py, prediction_step function
I see the following code in dpo_trainer.py's prediction_step method:
```python
... ...
logits_dict = {
    "logits_test/chosen": metrics["logits_test/chosen"],
    "logits_test/rejected": metrics["logits_test/rejected"],
}
... ...
```
But the keys and values are defined in get_batch_metrics function:
```python
... ...
prefix = "eval_" if train_eval == "eval" else ""
metrics[f"{prefix}rewards/chosen"] = chosen_rewards.cpu().numpy().mean()
metrics[f"{prefix}rewards/rejected"] = rejected_rewards.cpu().numpy().mean()
metrics[f"{prefix}rewards/accuracies"] = reward_accuracies.cpu().numpy().mean()
metrics[f"{prefix}rewards/margins"] = (chosen_rewards - rejected_rewards).cpu().numpy().mean()
metrics[f"{prefix}logps/rejected"] = policy_rejected_logps.detach().cpu().numpy().mean()
metrics[f"{prefix}logps/chosen"] = policy_chosen_logps.detach().cpu().numpy().mean()
metrics[f"{prefix}logits/rejected"] = policy_rejected_logits.detach().cpu().numpy().mean()
metrics[f"{prefix}logits/chosen"] = policy_chosen_logits.detach().cpu().numpy().mean()
... ...
```

That is, the variable metrics doesn't have "logits_test/chosen"、"logits_test/rejected".

I have run this code. For transformers framework, if the compute_metrics method is not defined, prediction_loss_only will be set to True and errors will be ignored. But if prediction_loss_only is forced to False, an error is indeed occurred.
So should we use the keys defined in get_batch_metrics when constructing logits_dict?


Discussion:
Referring to @kashif for DPO related questions.
indeed @Emperorizzis looks like a typo bug let me double check and make a PR

Resolving patch:
diff --git a/trl/trainer/dpo_trainer.py b/trl/trainer/dpo_trainer.py
--- a/trl/trainer/dpo_trainer.py
+++ b/trl/trainer/dpo_trainer.py
@@ -472,8 +472,8 @@ def prediction_step(
 
         # logits for the chosen and rejected samples from model
         logits_dict = {
-            "logits_test/chosen": metrics["logits_test/chosen"],
-            "logits_test/rejected": metrics["logits_test/rejected"],
+            "eval_logits/chosen": metrics["eval_logits/chosen"],
+            "eval_logits/rejected": metrics["eval_logits/rejected"],
         }
         logits = tuple(v for k, v in logits_dict.items() if k not in ignore_keys)
         logits = torch.stack(logits).mean(axis=1)


Test patch: