Repository: huggingface/open-r1

Issue:
accuracy_reward: difference in ordering of arguments in verify?
The [example usage](https://github.com/huggingface/Math-Verify?tab=readme-ov-file#example-usage) of `verify` in `math_verify` mentions the importance of arguments order (gold first, then answer).

The verification performed in `accuracy_reward` (see [here](https://github.com/huggingface/open-r1/blob/9915e06f1e2e5cdf580b706110734e2a24146a4d/src/open_r1/rewards.py#L74)) uses a different ordering (answer first, then gold). Is this normal? Shouldn't it be the opposite?

I am unsure if this is a bug, or if I am misunderstanding part of the code. 


Discussion:
+1. I have the same question here.



Patch:
diff --git a/src/open_r1/rewards.py b/src/open_r1/rewards.py
index 4a4e4b4d6..76797868c 100644
--- a/src/open_r1/rewards.py
+++ b/src/open_r1/rewards.py
@@ -20,7 +20,7 @@
 import math
 import re
 from functools import partial, update_wrapper
-from typing import Callable, Dict
+from typing import Callable, Dict, Optional
 
 from latex2sympy2_extended import NormalizationConfig
 from math_verify import LatexExtractionConfig, parse, verify
@@ -38,7 +38,7 @@
     AsyncSandbox = None
 
 
-def accuracy_reward(completions, solution, **kwargs):
+def accuracy_reward(completions: list[list[dict[str, str]]], solution: list[str], **kwargs) -> list[Optional[float]]:
     """Reward function that checks if the completion is the same as the ground truth."""
     contents = [completion[0]["content"] for completion in completions]
     rewards = []
@@ -46,7 +46,6 @@ def accuracy_reward(completions, solution, **kwargs):
         gold_parsed = parse(
             sol,
             extraction_mode="first_match",
-            extraction_config=[LatexExtractionConfig()],
         )
         if len(gold_parsed) != 0:
             # We require the answer to be provided in correct latex (no malformed operators)
@@ -69,15 +68,15 @@ def accuracy_reward(completions, solution, **kwargs):
                 ],
                 extraction_mode="first_match",
             )
-            # Reward 1 if the content is the same as the ground truth, 0 otherwise
+            # Compute binary rewards if verifiable, `None` otherwise to skip this example
             try:
-                reward = float(verify(answer_parsed, gold_parsed))
+                reward = float(verify(gold_parsed, answer_parsed))
             except Exception as e:
                 print(f"verify failed: {e}, answer: {answer_parsed}, gold: {gold_parsed}")
-                reward = 0.0
+                reward = None
         else:
-            # If the gold solution is not parseable, we reward 1 to skip this example
-            reward = 1.0
+            # If the gold solution is not parseable, we assign `None` to skip this example
+            reward = None
             print("Failed to parse gold solution: ", sol)
         rewards.append(reward)
 


Test patch:
diff --git a/tests/test_rewards.py b/tests/test_rewards.py
index 51b544f82..3a9df6100 100644
--- a/tests/test_rewards.py
+++ b/tests/test_rewards.py
@@ -82,7 +82,13 @@ def test_accuracy_reward_wrong_answer(self):
         """Test accuracy_reward with an incorrect answer."""
         completion = [[{"content": r"\boxed{\frac{64}{400}}"}]]
         solution = [r"\frac{63}{400}"]
+        rewards = accuracy_reward(completion, solution)
+        self.assertEqual(rewards[0], 0.0)
 
+    def test_accuracy_reward_wrong_answer_no_latex(self):
+        """Test accuracy_reward with an incorrect answer and gold solution with no latex."""
+        completion = [[{"content": r"\boxed{3}"}]]
+        solution = ["6"]
         rewards = accuracy_reward(completion, solution)
         self.assertEqual(rewards[0], 0.0)