Repository: camel-ai/camel
Pull request #2133: update: add float type comparison
Author: login: GitHoobar
Labels: name: enhancement
description: New feature or request
name: Loong
Created: 2025-04-07T15:53:47Z | Merged: 2025-04-08T00:20:13Z

## Description

for #2132 

## Checklist

Go over all the following points, and put an `x` in all the boxes that apply.

- [ ] I have read the [CONTRIBUTION](https://github.com/camel-ai/camel/blob/master/CONTRIBUTING.md) guide (**required**)
- [ ] I have linked this PR to an issue using the Development section on the right sidebar or by adding `Fixes #issue-number` in the PR description (**required**)
- [ ] I have checked if any dependencies need to be added or updated in `pyproject.toml` and `uv lock`
- [ ] I have updated the tests accordingly (*required for a bug fix or a new feature*)
- [ ] I have updated the documentation if needed:
- [ ] I have added examples if this is a new feature

If you are unsure about any of these, don't hesitate to ask. We are here to help!



Diff:
diff --git a/camel/verifiers/python_verifier.py b/camel/verifiers/python_verifier.py
index 94d20759fd..8181f83429 100644
--- a/camel/verifiers/python_verifier.py
+++ b/camel/verifiers/python_verifier.py
@@ -20,7 +20,7 @@
 import sys
 import tempfile
 import venv
-from typing import List, Optional, Tuple
+from typing import Any, List, Optional, Tuple
 
 from camel.extractors.base import BaseExtractor
 from camel.logger import get_logger
@@ -51,6 +51,7 @@ def __init__(
         extractor: Optional[BaseExtractor] = None,
         timeout: Optional[float] = 30.0,
         required_packages: Optional[List[str]] = None,
+        float_tolerance: Optional[float] = 1e-05,
         **kwargs,
     ):
         r"""Initializes the PythonVerifier.
@@ -63,11 +64,14 @@ def __init__(
             required_packages (Optional[List[str]], optional): A list of
                 packages to install in the virtual environment.
                 (default: :obj:`None`)
+            float_tolerance (Optional[float], optional): The tolerance for
+                floating point comparisons. (default: :obj:`1e-05`)
         """
         # TODO: Use CAMEL's Interpreter to execute the code
         super().__init__(extractor=extractor, timeout=timeout, **kwargs)
         self.venv_path: Optional[str] = None
         self.required_packages = required_packages or []
+        self.float_tolerance = float_tolerance
 
         if os.name == 'nt':  # Windows
             self.bin_dir = 'Scripts'
@@ -272,7 +276,13 @@ async def _verify_implementation(
                         result="",
                         error_message=f"Ground truth evaluation error: {e}",
                     )
-                if sol_val == gt_val:
+
+                if self.float_tolerance is not None:
+                    equal = self._is_equal_with_tolerance(sol_val, gt_val)
+                else:
+                    equal = sol_val == gt_val
+
+                if equal:
                     return VerificationResult(
                         status=VerificationOutcome.SUCCESS,
                         result=str(sol_val),
@@ -281,9 +291,18 @@ async def _verify_implementation(
                     return VerificationResult(
                         status=VerificationOutcome.FAILURE,
                         result=str(sol_val),
-                        error_message="Output mismatch: "
-                        f"{sol_val} != {gt_val}",
+                        error_message=(
+                            "Values not equal"
+                            + (
+                                " (with float tolerance "
+                                f"{self.float_tolerance})"
+                                if self.float_tolerance is not None
+                                else ""
+                            )
+                            + f": {sol_val} != {gt_val}"
+                        ),
                     )
+
             else:
                 return VerificationResult(
                     status=VerificationOutcome.SUCCESS,
@@ -333,17 +352,21 @@ async def _verify_implementation(
                             error_message="Ground truth evaluation error:"
                             f"{e}",
                         )
-                    if sol_val == gt_val:
+                    if self.float_tolerance is not None:
+                        equal = self._is_equal_with_tolerance(sol_val, gt_val)
+                    else:
+                        equal = sol_val == gt_val
+
+                    if equal:
                         return VerificationResult(
-                            status=VerificationOutcome.SUCCESS,
-                            result=sol_out,
+                            status=VerificationOutcome.SUCCESS, result=sol_out
                         )
                     else:
                         return VerificationResult(
                             status=VerificationOutcome.FAILURE,
                             result=sol_out,
-                            error_message="Output mismatch: "
-                            f"{sol_val} != {gt_val}",
+                            error_message=f"Output mismatch: {sol_val} "
+                            f"!= {gt_val}",
                         )
                 else:
                     # Fallback: string comparison
@@ -456,3 +479,64 @@ def _is_expression(self, code: str) -> bool:
                 return True
             except Exception:
                 return False
+
+    def _is_equal_with_tolerance(self, a: Any, b: Any) -> bool:
+        r"""Compares two Python objects for equality with optional float
+        tolerance.
+
+        This method recursively compares nested structures (lists, tuples,
+        sets, and dictionaries) and applies floating point tolerance when
+        comparing numerical values. If no float tolerance is set, a runtime
+        error is raised.
+
+        Args:
+            a (Any): First value to compare.
+            b (Any): Second value to compare.
+
+        Returns:
+            bool: True if the values are considered equal within the
+                specified float tolerance; False otherwise.
+
+        Raises:
+            RuntimeError: If float tolerance is not set (i.e., None).
+        """
+        if self.float_tolerance is None:
+            raise RuntimeError(
+                "Can't compare with tolerance if tolerance is None."
+            )
+        if isinstance(a, (int, float)) and isinstance(b, (int, float)):
+            return abs(float(a) - float(b)) <= self.float_tolerance
+        if isinstance(a, list) and isinstance(b, list):
+            return len(a) == len(b) and all(
+                self._is_equal_with_tolerance(x, y) for x, y in zip(a, b)
+            )
+        if isinstance(a, tuple) and isinstance(b, tuple):
+            return len(a) == len(b) and all(
+                self._is_equal_with_tolerance(x, y) for x, y in zip(a, b)
+            )
+        if isinstance(a, set) and isinstance(b, set):
+            if len(a) != len(b):
+                return False
+            # Need to check both directions to ensure proper matching
+            # Create a copy of b to track matched elements
+            b_copy = list(b)
+            for x in a:
+                found_match = False
+                for i, y in enumerate(b_copy):
+                    if self._is_equal_with_tolerance(x, y):
+                        found_match = True
+                        # Remove the matched element to prevent double-matching
+                        b_copy.pop(i)
+                        break
+                if not found_match:
+                    return False
+            return True
+        if isinstance(a, dict) and isinstance(b, dict):
+            if set(a.keys()) != set(b.keys()):
+                return False
+            return all(self._is_equal_with_tolerance(a[k], b[k]) for k in a)
+        logger.warning(
+            f"Falling back to simple comparison without "
+            f"tolerance for {a} and {b}."
+        )
+        return a == b  # fallback
diff --git a/test/verifiers/test_python_verifier.py b/test/verifiers/test_python_verifier.py
index 394200fc8f..2b548869ad 100644
--- a/test/verifiers/test_python_verifier.py
+++ b/test/verifiers/test_python_verifier.py
@@ -151,3 +151,91 @@ async def test_python_verifier_with_numpy(python_verifier):
     assert result.status == VerificationOutcome.SUCCESS
     assert "[1 2 3]" in result.result
     await python_verifier.cleanup()
+
+
+@pytest.mark.parametrize(
+    "a,b,tol,expected",
+    [
+        (0.123456, 0.123457, 1e-5, True),
+        (0.123456, 0.123467, 1e-5, False),
+        ([1.0, 2.0], [1.0, 2.000001], 1e-5, True),
+        ([1.0, 2.0], [1.0, 2.1], 1e-5, False),
+        ({'a': 1.0, 'b': 2.0}, {'a': 1.0000001, 'b': 2.0}, 1e-5, True),
+        ({'a': 1.0, 'b': 2.0}, {'a': 1.0001, 'b': 2.0}, 1e-5, False),
+        ((1.0, 2.0), (1.0, 2.000001), 1e-5, True),
+        ((1.0, 2.0), (1.0, 2.01), 1e-5, False),
+        ({1.0, 2.0}, {2.000001, 1.0}, 1e-5, True),
+        ({1.0, 2.0}, {2.1, 1.0}, 1e-5, False),
+        ({1.0, 1.000001}, {1.000002, 2.0}, 1e-5, False),
+        ({"x": [1.0, 2.0]}, {"x": [1.000001, 2.0]}, 1e-5, True),
+        ({"x": [1.0, 2.0]}, {"x": [1.1, 2.0]}, 1e-5, False),
+    ],
+)
+def test_is_equal_with_tolerance(a, b, tol, expected):
+    r"""Test the function calculating whether two floats are equal
+    given tolerance"""
+    verifier = PythonVerifier(float_tolerance=tol)
+    assert verifier._is_equal_with_tolerance(a, b) == expected
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "script,expected_output,tol,should_pass",
+    [
+        # Float scalar
+        ("0.123456", "0.123457", 1e-5, True),
+        ("0.123456", "0.123467", 1e-5, False),
+        # Lists
+        ("[1.0, 2.0]", "[1.0, 2.000001]", 1e-5, True),
+        ("[1.0, 2.0]", "[1.0, 2.1]", 1e-5, False),
+        # Dicts
+        ("{'a': 1.0, 'b': 2.0}", "{'a': 1.000001, 'b': 2.0}", 1e-5, True),
+        ("{'a': 1.0, 'b': 2.0}", "{'a': 1.1, 'b': 2.0}", 1e-5, False),
+        # Sets
+        ("{1.0, 2.0}", "{1.0, 2.000001}", 1e-5, True),
+        ("{1.0, 2.0}", "{1.0, 2.1}", 1e-5, False),
+        # Tuples
+        ("(1.0, 2.0)", "(1.0, 2.000001)", 1e-5, True),
+        ("(1.0, 2.0)", "(1.0, 2.1)", 1e-5, False),
+        # Mixed nesting
+        ("{'a': [1.0, 2.0]}", "{'a': [1.000001, 2.0]}", 1e-5, True),
+        ("{'a': [1.0, 2.0]}", "{'a': [1.1, 2.0]}", 1e-5, False),
+        # Same as above, but as code blocks
+        ("print(0.123456)", "0.123457", 1e-5, True),
+        ("print(0.123456)", "0.123467", 1e-5, False),
+        ("print([1.0, 2.0])", "[1.0, 2.000001]", 1e-5, True),
+        ("print([1.0, 2.0])", "[1.0, 2.1]", 1e-5, False),
+        (
+            "print({'a': 1.0, 'b': 2.0})",
+            "{'a': 1.000001, 'b': 2.0}",
+            1e-5,
+            True,
+        ),
+        ("print({'a': 1.0, 'b': 2.0})", "{'a': 1.1, 'b': 2.0}", 1e-5, False),
+        ("print({1.0, 2.0})", "{1.0, 2.000001}", 1e-5, True),
+        ("print({1.0, 2.0})", "{1.0, 2.1}", 1e-5, False),
+        # This test should fail with the simplified all(any(...)) logic
+        ("print({1.0, 1.000001})", "{1.000002, 2.0}", 1e-5, False),
+    ],
+)
+async def test_verify_with_float_tolerance(
+    script, expected_output, tol, should_pass
+):
+    r"""Test verifier end-to-end with float tolerance"""
+    verifier = PythonVerifier(float_tolerance=tol)
+    await verifier.setup(uv=True)
+
+    result = await verifier.verify(
+        solution=script, reference_answer=expected_output
+    )
+
+    if should_pass:
+        assert (
+            result.status == VerificationOutcome.SUCCESS
+        ), f"Expected success, got {result}"
+    else:
+        assert (
+            result.status == VerificationOutcome.FAILURE
+        ), f"Expected failure, got {result}"
+
+    await verifier.cleanup()


Reviews:
user: login: old-hallerite
body: Thanks @GitHoobar. Left some comments.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T15:57:30Z
user: login: old-hallerite
body: Looks almost good to go, just one more change.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T19:22:00Z
user: login: old-hallerite
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-07T20:39:27Z
user: login: Wendong-Fan
body: thanks @GitHoobar ! Left some comments below
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T23:45:03Z
user: login: old-hallerite
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-08T00:04:14Z
user: login: Wendong-Fan
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-08T00:15:13Z
user: login: Wendong-Fan
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T00:19:08Z

Review comments:
diff_hunk: @@ -272,6 +272,25 @@ async def _verify_implementation(
                         result="",
                         error_message=f"Ground truth evaluation error: {e}",
                     )
+
+                # Direct float comparison after evaluation
+                if isinstance(sol_val, float) or isinstance(gt_val, float):
path: camel/verifiers/python_verifier.py
user: login: old-hallerite
body: we should only do this if one of them is float AND the other is a number (float or int), else this will fail. 
created_at: 2025-04-07T15:56:37Z
author_association: COLLABORATOR
diff_hunk: @@ -272,6 +272,25 @@ async def _verify_implementation(
                         result="",
                         error_message=f"Ground truth evaluation error: {e}",
                     )
+
+                # Direct float comparison after evaluation
+                if isinstance(sol_val, float) or isinstance(gt_val, float):
+                    if abs(
+                        float(sol_val) - float(gt_val)
+                    ) <= 1e-08 + 1e-05 * abs(float(gt_val)):
path: camel/verifiers/python_verifier.py
user: login: old-hallerite
body: the tolerance should just be a number that the user can set in the `__init__`.
created_at: 2025-04-07T15:57:14Z
author_association: COLLABORATOR
diff_hunk: @@ -51,6 +51,8 @@ def __init__(
         extractor: Optional[BaseExtractor] = None,
         timeout: Optional[float] = 30.0,
         required_packages: Optional[List[str]] = None,
+        enable_float_comparison: bool = False,
+        float_tolerance: float = 1e-05,
path: camel/verifiers/python_verifier.py
user: login: old-hallerite
body: Why not just `float_tolerance: Optional[float]`?

Then, you don't need a flag to check whether to use the tolerance. Instead, if it is `None`, we do not tolerate any deviation.
created_at: 2025-04-07T19:20:53Z
author_association: COLLABORATOR
diff_hunk: @@ -63,11 +64,14 @@ def __init__(
             required_packages (Optional[List[str]], optional): A list of
                 packages to install in the virtual environment.
                 (default: :obj:`None`)
+            float_tolerance (Optional[float], optional): The tolerance for
+            floating point comparisons. (default: :obj:`1e-05`)
path: camel/verifiers/python_verifier.py
user: login: Wendong-Fan
body: docstring format
```suggestion
            float_tolerance (Optional[float], optional): The tolerance for
                floating point comparisons. (default: :obj:`1e-05`)
```
created_at: 2025-04-07T22:39:40Z
author_association: MEMBER
diff_hunk: @@ -456,3 +479,50 @@ def _is_expression(self, code: str) -> bool:
                 return True
             except Exception:
                 return False
+
+    def _is_equal_with_tolerance(self, a, b) -> bool:
+        r"""Compares two Python objects for equality with optional float
+            tolerance.
+
+        This method recursively compares nested structures (lists, tuples,
+        sets, and dictionaries) and applies floating point tolerance when
+        comparing numerical values. If no float tolerance is set, a runtime
+        error is raised.
+
+        Args:
+            a: First value to compare.
+            b: Second value to compare.
+
+        Returns:
+            bool: True if the values are considered equal within the
+            specified float tolerance; False otherwise.
+
+        Raises:
+            RuntimeError: If float tolerance is not set (i.e., None).
+        """
path: camel/verifiers/python_verifier.py
user: login: Wendong-Fan
body: ```suggestion
        r"""Compares two Python objects for equality with optional float
        tolerance.

        This method recursively compares nested structures (lists, tuples,
        sets, and dictionaries) and applies floating point tolerance when
        comparing numerical values. If no float tolerance is set, a runtime
        error is raised.

        Args:
            a: First value to compare.
            b: Second value to compare.

        Returns:
            bool: True if the values are considered equal within the
                specified float tolerance; False otherwise.

        Raises:
            RuntimeError: If float tolerance is not set (i.e., None).
        """
```
created_at: 2025-04-07T22:40:20Z
author_association: MEMBER
diff_hunk: @@ -456,3 +479,50 @@ def _is_expression(self, code: str) -> bool:
                 return True
             except Exception:
                 return False
+
+    def _is_equal_with_tolerance(self, a, b) -> bool:
+        r"""Compares two Python objects for equality with optional float
+            tolerance.
+
+        This method recursively compares nested structures (lists, tuples,
+        sets, and dictionaries) and applies floating point tolerance when
+        comparing numerical values. If no float tolerance is set, a runtime
+        error is raised.
+
+        Args:
+            a: First value to compare.
+            b: Second value to compare.
+
+        Returns:
+            bool: True if the values are considered equal within the
+            specified float tolerance; False otherwise.
+
+        Raises:
+            RuntimeError: If float tolerance is not set (i.e., None).
+        """
+        if self.float_tolerance is None:
+            raise RuntimeError(
+                "Can't compare with tolerance if tolerance is None."
+            )
+        if isinstance(a, (int, float)) and isinstance(b, (int, float)):
+            return abs(float(a) - float(b)) <= self.float_tolerance
+        if isinstance(a, list) and isinstance(b, list):
+            return len(a) == len(b) and all(
+                self._is_equal_with_tolerance(x, y) for x, y in zip(a, b)
+            )
+        if isinstance(a, tuple) and isinstance(b, tuple):
+            return len(a) == len(b) and all(
+                self._is_equal_with_tolerance(x, y) for x, y in zip(a, b)
+            )
+        if isinstance(a, set) and isinstance(b, set):
+            if len(a) != len(b):
+                return False
+            return all(
+                any(self._is_equal_with_tolerance(x, y) for y in b) for x in a
+            )
+        if isinstance(a, dict) and isinstance(b, dict):
+            if set(a.keys()) != set(b.keys()):
+                return False
+            return all(self._is_equal_with_tolerance(a[k], b[k]) for k in a)
path: camel/verifiers/python_verifier.py
user: login: Wendong-Fan
body: seems it only checked if each element in set a had at least one matching element in set b, but didn't account for multiple elements in a potentially matching the same element in b? Also the check didn't consider the sequence
created_at: 2025-04-07T22:52:17Z
author_association: MEMBER
diff_hunk: @@ -456,3 +479,50 @@ def _is_expression(self, code: str) -> bool:
                 return True
             except Exception:
                 return False
+
+    def _is_equal_with_tolerance(self, a, b) -> bool:
+        r"""Compares two Python objects for equality with optional float
+            tolerance.
+
+        This method recursively compares nested structures (lists, tuples,
+        sets, and dictionaries) and applies floating point tolerance when
+        comparing numerical values. If no float tolerance is set, a runtime
+        error is raised.
+
+        Args:
+            a: First value to compare.
+            b: Second value to compare.
+
+        Returns:
+            bool: True if the values are considered equal within the
+            specified float tolerance; False otherwise.
+
+        Raises:
+            RuntimeError: If float tolerance is not set (i.e., None).
+        """
+        if self.float_tolerance is None:
+            raise RuntimeError(
+                "Can't compare with tolerance if tolerance is None."
+            )
+        if isinstance(a, (int, float)) and isinstance(b, (int, float)):
+            return abs(float(a) - float(b)) <= self.float_tolerance
+        if isinstance(a, list) and isinstance(b, list):
+            return len(a) == len(b) and all(
+                self._is_equal_with_tolerance(x, y) for x, y in zip(a, b)
+            )
+        if isinstance(a, tuple) and isinstance(b, tuple):
+            return len(a) == len(b) and all(
+                self._is_equal_with_tolerance(x, y) for x, y in zip(a, b)
+            )
+        if isinstance(a, set) and isinstance(b, set):
+            if len(a) != len(b):
+                return False
+            return all(
+                any(self._is_equal_with_tolerance(x, y) for y in b) for x in a
+            )
+        if isinstance(a, dict) and isinstance(b, dict):
+            if set(a.keys()) != set(b.keys()):
+                return False
+            return all(self._is_equal_with_tolerance(a[k], b[k]) for k in a)
path: camel/verifiers/python_verifier.py
user: login: old-hallerite
body: I don't quite understand what you mean
created_at: 2025-04-08T00:04:14Z
author_association: COLLABORATOR
diff_hunk: @@ -456,3 +479,50 @@ def _is_expression(self, code: str) -> bool:
                 return True
             except Exception:
                 return False
+
+    def _is_equal_with_tolerance(self, a, b) -> bool:
+        r"""Compares two Python objects for equality with optional float
+            tolerance.
+
+        This method recursively compares nested structures (lists, tuples,
+        sets, and dictionaries) and applies floating point tolerance when
+        comparing numerical values. If no float tolerance is set, a runtime
+        error is raised.
+
+        Args:
+            a: First value to compare.
+            b: Second value to compare.
+
+        Returns:
+            bool: True if the values are considered equal within the
+            specified float tolerance; False otherwise.
+
+        Raises:
+            RuntimeError: If float tolerance is not set (i.e., None).
+        """
+        if self.float_tolerance is None:
+            raise RuntimeError(
+                "Can't compare with tolerance if tolerance is None."
+            )
+        if isinstance(a, (int, float)) and isinstance(b, (int, float)):
+            return abs(float(a) - float(b)) <= self.float_tolerance
+        if isinstance(a, list) and isinstance(b, list):
+            return len(a) == len(b) and all(
+                self._is_equal_with_tolerance(x, y) for x, y in zip(a, b)
+            )
+        if isinstance(a, tuple) and isinstance(b, tuple):
+            return len(a) == len(b) and all(
+                self._is_equal_with_tolerance(x, y) for x, y in zip(a, b)
+            )
+        if isinstance(a, set) and isinstance(b, set):
+            if len(a) != len(b):
+                return False
+            return all(
+                any(self._is_equal_with_tolerance(x, y) for y in b) for x in a
+            )
+        if isinstance(a, dict) and isinstance(b, dict):
+            if set(a.keys()) != set(b.keys()):
+                return False
+            return all(self._is_equal_with_tolerance(a[k], b[k]) for k in a)
path: camel/verifiers/python_verifier.py
user: login: Wendong-Fan
body: check: https://github.com/camel-ai/camel/pull/2133/commits/5a7dd4bcc127d827a6ae758dbd402844d3ea1596#diff-61a0e70c4af3fd35cd485a6bae9e12aa34c4f9d4c49f8400d98e02482be8709dR217
created_at: 2025-04-08T00:19:08Z
author_association: MEMBER

PR comments:
user: login: lightaime
created_at: 2025-04-07T16:16:00Z
body: Thanks Rishabh. A couple feedback:

1. The float tolerance maybe is not needed to enable by default. Only if the user set it as true. Otherwise, there maybe some edge cases
2. What if the gt is a list of floating points?
author_association: MEMBER
user: login: lightaime
created_at: 2025-04-07T22:46:09Z
body: Should we set it as None by default to avoid unexpected behaviors?
author_association: MEMBER
user: login: lightaime
created_at: 2025-04-08T00:23:42Z
body: > Should we set it as None by default to avoid unexpected behaviors?

Why ignored my comments and merged 😂
author_association: MEMBER
user: login: Wendong-Fan
created_at: 2025-04-08T00:24:37Z
body: > > Should we set it as None by default to avoid unexpected behaviors?
> 
> Why ignored my comments and merged 😂

oh, my bad.. hot fix soon
author_association: MEMBER