Repository: huggingface/open-r1

Issue:
Format reward problem
The format reward seems to have some problems.

I tested the current reward function using the following code. The input is a model-generated output after training for 100 steps:  

```python
import re

inputs = "<think>\nI will count each distinct object in the image:\n1. Purple scooter\n2. Red bicycle\n3. Green motorcycle\n4. Gray sedan\n5. Yellow school bus\n6. Small green double-decker bus\n7. Small red car\n8. Small purple car\n9. Small gray dirt bike\n\nThere are 9 distinct objects in total.\n</think>\n<answer>9</answer>"

def format_reward(completions):
    """Reward function that checks if the completion follows a specific format."""
    pattern = r"<think>.*?</think>\s*<answer>.*?</answer>"
    matches = [re.match(pattern, content) for content in completions]
    return [1.0 if match else 0.0 for match in matches]
```

However, this function incorrectly returns `0` for valid inputs.  

The issue occurs because the `.*?` wildcard does not match newlines unless `re.DOTALL` is enabled.  Updating the regex as follows ensures that the entire string matches the expected format:  

```python
pattern = r"<think>.*?</think>\s*<answer>.*?</answer>"
matches = [re.match(pattern, content, re.DOTALL) for content in completions]
```

I valid this format reward in the VLM setting with Qwen2-VL. See https://github.com/Deep-Agent/R1-V/issues/19

Discussion:
@asirgogogo: great！！！@brishtiteveja: Related node: Please also check whether you are using cuda 12.4 (as it is shown as Caution in ReadMe). I was initially using cuda 12.1 and then switched to 12.4@qgallouedec: Closed by #238

Solution patch:
diff --git a/src/open_r1/rewards.py b/src/open_r1/rewards.py
index f92da19..f33b564 100644
--- a/src/open_r1/rewards.py
+++ b/src/open_r1/rewards.py
@@ -51,9 +51,9 @@ def accuracy_reward(completions, solution, **kwargs):
 
 def format_reward(completions, **kwargs):
     """Reward function that checks if the completion has a specific format."""
-    pattern = r"^<think>.*?</think><answer>.*?</answer>$"
+    pattern = r"^<think>.*?</think>\s*<answer>.*?</answer>$"
     completion_contents = [completion[0]["content"] for completion in completions]
-    matches = [re.match(pattern, content) for content in completion_contents]
+    matches = [re.match(pattern, content, re.DOTALL | re.MULTILINE) for content in completion_contents]
     return [1.0 if match else 0.0 for match in matches]
 
 


Test patch:
diff --git a/tests/test_rewards.py b/tests/test_rewards.py
index 0ff8a10..473a076 100644
--- a/tests/test_rewards.py
+++ b/tests/test_rewards.py
@@ -97,6 +97,13 @@ class TestRewards(unittest.TestCase):
             rewards = get_cosine_scaled_reward(**test_params)(completion, [solution])
             self.assertAlmostEqual(rewards[0], expected_reward, places=2)
 
+    def test_format_reward_specific_multiline(self):
+        """Test format_reward with a specific multiline input."""
+        inputs = "<think>\nI will count each distinct object in the image:\n1. Purple scooter\n2. Red bicycle\n3. Green motorcycle\n4. Gray sedan\n5. Yellow school bus\n6. Small green double-decker bus\n7. Small red car\n8. Small purple car\n9. Small gray dirt bike\n\nThere are 9 distinct objects in total.\n</think>\n<answer>9</answer>"
+        completion = [[{"content": inputs}]]
+        rewards = format_reward(completion)
+        self.assertEqual(rewards[0], 1.0)
+
 
 if __name__ == "__main__":
     unittest.main()