Repository: huggingface/trl
Pull request: [`core`] Add warning when negative KL
# What does this PR do?

Fixes: #235 

The warning message will be displayed only once, if there is a repetition, as described in the official documentation of `warnings` package: https://docs.python.org/2/library/warnings.html / https://stackoverflow.com/questions/22661745/why-only-one-warning-in-a-loop

cc @lvwerra 

Problem statement:
Add warning when KL-divergence is negative
A common failure mode of the training is that the generation kwargs are not correctly set which leads to negative KL-divergence which is almost always a precursor for failed training. We should at least show a warning when this happens.

Hints from issue discussion:

Commit message: fix

Patch to review:
diff --git a/trl/trainer/ppo_trainer.py b/trl/trainer/ppo_trainer.py
index dec8eefd32..66da3415b1 100644
--- a/trl/trainer/ppo_trainer.py
+++ b/trl/trainer/ppo_trainer.py
@@ -985,6 +985,14 @@ def record_step_stats(self, kl_coef: float, **data):
 
         mean_non_score_reward = masked_mean(data["non_score_reward"], mask)
 
+        if mean_kl.item() < 0.0:
+            # warn users
+            warnings.warn(
+                f"KL divergence is starting to become negative: {mean_kl.item()} - this might be a precursor for failed training."
+                " sometimes this happens because the generation kwargs are not correctly set. Please make sure"
+                " that the generation kwargs are set correctly, or review your training hyperparameters."
+            )
+
         stats = {
             "objective/kl": mean_kl,
             "objective/kl_dist": kl_list,


Code review comments:
diff_hunk: @@ -993,6 +985,14 @@ def record_step_stats(self, kl_coef: float, **data):
 
         mean_non_score_reward = masked_mean(data["non_score_reward"], mask)
 
+        if mean_kl.item() < 0.0:
+            # warn users
+            warnings.warn(
+                f"KL divergence is starting to become negative: {mean_kl.item()} - this might be a precursor for failed training."
path: trl/trainer/ppo_trainer.py
text: @user1:
for better readability :) 
```suggestion
                f"KL divergence is starting to become negative: {mean_kl.item():.2f} - this might be a precursor for failed training."
```