Repository: numpy/numpy
Pull request: ENH: assert_array_less should report max violations instead of max differences 
Fixes #23894 

Problem statement:
ENH: `assert_array_less` should report max violations instead of max differences
### Proposed new feature or change:

When `numpy.testing.assert_array_less` fails, it reports a "Max absolute difference" and a "Max relative difference". These metrics are the max element-wise differences between the two tested arrays. These make sense to report when testing array equality, but they don't seem useful when asserting that one array is element-wise less than another. It would make more sense to report the maximum values by which the less-than constraint is violated. "Max absolute violation" could be calculated as:

```
np.maximum(array1 - array2, 0).max()
```

when `npt.assert_array_less(array1, array2)` fails. "Max relative violation" could be calculated similarly.

I'd be happy to implement this if people agree. Thanks!

Hints from issue discussion:
Right the difference itself isn't meaningful for those, so I would agree that it should report the difference only for violating terms.
Not sure if it does or doesn't, but we can change the wording in that case to be clearer.
It reports the max difference of all terms, not just violating terms. Here's an example:

```python
from numpy.testing import assert_array_less
import numpy as np

assert_array_less([0., 3., 0.], [1000., 2.5, 1.])
```
which prints:
```
Arrays are not less-ordered

Mismatched elements: 1 / 3 (33.3%)
Max absolute difference: 1000.
Max relative difference: 1.
 x: array([0., 3., 0.])
 y: array([1000. ,    2.5,    1. ])
```

The only violation here is 3.0 !< 2.5, but it reports "Max absolute difference: 1000."
Hi may I check if this issue is open for contribution? I would like to give it a go if it is open
We don't assign issues. When you submit a PR, refer back to this with a `#23894` (without the code formatting) so github will create a link to this issue and others will know you are working on it. @allenlawrence94 did you get anywhere with
>  I'd be happy to implement this if people agree.

I don't see a linked PR.
@mattip - no I haven't been able to work on this yet. @ellaella12 - feel free to take it if you like!

Commit message: Update docstrings

Patch to review:
diff --git a/numpy/testing/_private/utils.py b/numpy/testing/_private/utils.py
index 28dd656c4a4d..93ea8c030364 100644
--- a/numpy/testing/_private/utils.py
+++ b/numpy/testing/_private/utils.py
@@ -474,8 +474,8 @@ def assert_almost_equal(actual, desired, decimal=7, err_msg='', verbose=True):
     Arrays are not almost equal to 9 decimals
     <BLANKLINE>
     Mismatched elements: 1 / 2 (50%)
-    Max absolute difference: 6.66669964e-09
-    Max relative difference: 2.85715698e-09
+    Max absolute difference among violations: 6.66669964e-09
+    Max relative difference among violations: 2.85715698e-09
      x: array([1.         , 2.333333333])
      y: array([1.        , 2.33333334])
 
@@ -740,6 +740,7 @@ def func_assert_same_pos(x, y, func=isnan, hasval='nan'):
             return
 
         val = comparison(x, y)
+        invalids = np.logical_not(val)
 
         if isinstance(val, bool):
             cond = val
@@ -767,29 +768,42 @@ def func_assert_same_pos(x, y, func=isnan, hasval='nan'):
                     if np.issubdtype(x.dtype, np.unsignedinteger):
                         error2 = abs(y - x)
                         np.minimum(error, error2, out=error)
-                    max_abs_error = max(error)
+
+                    reduced_error = error[invalids]
+                    max_abs_error = max(reduced_error)
                     if getattr(error, 'dtype', object_) == object_:
-                        remarks.append('Max absolute difference: '
-                                       + str(max_abs_error))
+                        remarks.append(
+                            'Max absolute difference among violations: '
+                            + str(max_abs_error))
                     else:
-                        remarks.append('Max absolute difference: '
-                                       + array2string(max_abs_error))
-
+                        remarks.append(
+                            'Max absolute difference among violations: '
+                            + array2string(max_abs_error))
+                        
                     # note: this definition of relative error matches that one
                     # used by assert_allclose (found in np.isclose)
                     # Filter values where the divisor would be zero
                     nonzero = bool_(y != 0)
-                    if all(~nonzero):
+                    nonzero_and_invalid = np.logical_and(invalids, nonzero)
+                    
+                    if all(~nonzero_and_invalid):
                         max_rel_error = array(inf)
                     else:
-                        max_rel_error = max(error[nonzero] / abs(y[nonzero]))
-                    if getattr(error, 'dtype', object_) == object_:
-                        remarks.append('Max relative difference: '
-                                       + str(max_rel_error))
-                    else:
-                        remarks.append('Max relative difference: '
-                                       + array2string(max_rel_error))
-
+                        nonzero_invalid_error = error[nonzero_and_invalid]
+
+                        broadcasted_y = np.broadcast_to(y, error.shape)
+                        nonzero_invalid_y = broadcasted_y[nonzero_and_invalid]
+                        max_rel_error = max(nonzero_invalid_error 
+                                            / abs(nonzero_invalid_y))
+
+                    if getattr(error, 'dtype', object_) == object_: 
+                        remarks.append(
+                            'Max relative difference among violations: '
+                            + str(max_rel_error))
+                    else:               
+                        remarks.append(
+                            'Max relative difference among violations: '
+                            + array2string(max_rel_error))
             err_msg += '\n' + '\n'.join(remarks)
             msg = build_err_msg([ox, oy], err_msg,
                                 verbose=verbose, header=header,
@@ -870,8 +884,8 @@ def assert_array_equal(x, y, err_msg='', verbose=True, *, strict=False):
     Arrays are not equal
     <BLANKLINE>
     Mismatched elements: 1 / 3 (33.3%)
-    Max absolute difference: 4.4408921e-16
-    Max relative difference: 1.41357986e-16
+    Max absolute difference among violations: 4.4408921e-16
+    Max relative difference among violations: 1.41357986e-16
      x: array([1.      , 3.141593,      nan])
      y: array([1.      , 3.141593,      nan])
 
@@ -983,8 +997,8 @@ def assert_array_almost_equal(x, y, decimal=6, err_msg='', verbose=True):
     Arrays are not almost equal to 5 decimals
     <BLANKLINE>
     Mismatched elements: 1 / 3 (33.3%)
-    Max absolute difference: 6.e-05
-    Max relative difference: 2.57136612e-05
+    Max absolute difference among violations: 6.e-05
+    Max relative difference among violations: 2.57136612e-05
      x: array([1.     , 2.33333,     nan])
      y: array([1.     , 2.33339,     nan])
 
@@ -1080,8 +1094,8 @@ def assert_array_less(x, y, err_msg='', verbose=True):
     Arrays are not less-ordered
     <BLANKLINE>
     Mismatched elements: 1 / 3 (33.3%)
-    Max absolute difference: 1.
-    Max relative difference: 0.5
+    Max absolute difference among violations: 0.
+    Max relative difference among violations: 0.
      x: array([ 1.,  1., nan])
      y: array([ 1.,  2., nan])
 
@@ -1092,8 +1106,8 @@ def assert_array_less(x, y, err_msg='', verbose=True):
     Arrays are not less-ordered
     <BLANKLINE>
     Mismatched elements: 1 / 2 (50%)
-    Max absolute difference: 2.
-    Max relative difference: 0.66666667
+    Max absolute difference among violations: 1.
+    Max relative difference among violations: 0.33333333
      x: array([1., 4.])
      y: array(3)
 
diff --git a/numpy/testing/tests/test_utils.py b/numpy/testing/tests/test_utils.py
index 0aaa508ee5d2..0bed157cff17 100644
--- a/numpy/testing/tests/test_utils.py
+++ b/numpy/testing/tests/test_utils.py
@@ -89,16 +89,27 @@ def foo(t):
     def test_0_ndim_array(self):
         x = np.array(473963742225900817127911193656584771)
         y = np.array(18535119325151578301457182298393896)
-        assert_raises(AssertionError, self._assert_func, x, y)
+        
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(x, y)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+        in msg)
 
         y = x
         self._assert_func(x, y)
 
-        x = np.array(43)
-        y = np.array(10)
-        assert_raises(AssertionError, self._assert_func, x, y)
+        x = np.array(4395065348745.5643764887869876)
+        y = np.array(0)
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(x, y)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+                'Max absolute difference among violations: 4.39506535e+12\n'
+                'Max relative difference among violations: inf'
+                in msg)
 
-        y = x
+        x = y
         self._assert_func(x, y)
 
     def test_generic_rank3(self):
@@ -190,6 +201,23 @@ def __ne__(self, other):
         self._test_not_equal(a, b)
         self._test_not_equal(b, a)
 
+        with pytest.raises(AssertionError) as exc_info:
+            self._test_equal(a, b)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 2 (50%)\n'
+                'Max absolute difference among violations: 1.\n'
+                'Max relative difference among violations: 0.5'
+                in msg)
+        
+        c = np.array([0., 2.9]).view(MyArray)
+        with pytest.raises(AssertionError) as exc_info:
+            self._test_equal(b, c)
+        msg = str(exc_info.value)
+        # assert_('Mismatched elements: 2 / 2 (100%)\n'
+        #         'Max absolute difference among violations: 2.\n'
+        #         'Max relative difference among violations: inf'
+        #         in msg)
+
     def test_subclass_that_does_not_implement_npall(self):
         class MyArray(np.ndarray):
             def __array_function__(self, *args, **kwargs):
@@ -218,12 +246,27 @@ def test_array_vs_scalar_is_equal(self):
 
         self._test_equal(a, b)
 
-    def test_array_vs_scalar_not_equal(self):
+    def test_array_vs_array_not_equal(self):
         """Test comparing an array with a scalar when not all values equal."""
-        a = np.array([1., 2., 3.])
-        b = 1.
+        a = np.array([34986, 545676, 439655, 563766])
+        b = np.array([34986, 545676, 439655, 0])
 
-        self._test_not_equal(a, b)
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(a, b)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 4 (25%)\n'
+                'Max absolute difference among violations: 563766\n'
+                'Max relative difference among violations: inf'
+                in msg)
+        
+        a = np.array([34986, 545676, 439655.2, 563766])
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(a, b)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 2 / 4 (50%)\n'
+                'Max absolute difference among violations: 563766.\n'
+                'Max relative difference among violations: 4.54902139e-07'
+                in msg)
 
     def test_array_vs_scalar_strict(self):
         """Test comparing an array with a scalar with strict option."""
@@ -262,6 +305,8 @@ def test_build_err_msg_defaults(self):
              '2.00003, 3.00004])')
         assert_equal(a, b)
 
+    
+
     def test_build_err_msg_no_verbose(self):
         x = np.array([1.00001, 2.00002, 3.00003])
         y = np.array([1.00002, 2.00003, 3.00004])
@@ -402,14 +447,41 @@ def test_closeness(self):
         # so this check serves to preserve the wrongness.
 
         # test scalars
-        self._assert_func(1.499999, 0.0, decimal=0)
-        assert_raises(AssertionError,
-                          lambda: self._assert_func(1.5, 0.0, decimal=0))
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(1.5, 0.0, decimal=0)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+            'Max absolute difference among violations: 1.5\n'
+            'Max relative difference among violations: inf'
+            in msg)
 
         # test arrays
         self._assert_func([1.499999], [0.0], decimal=0)
-        assert_raises(AssertionError,
-                          lambda: self._assert_func([1.5], [0.0], decimal=0))
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func([1.5], [0.0], decimal=0)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+            'Max absolute difference among violations: 1.5\n'
+            'Max relative difference among violations: inf'
+            in msg)
+        
+        a = [1.4999999, 0.00003]
+        b = [1.49999991, 0]
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(a, b, decimal=7)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 2 (50%)\n'
+                'Max absolute difference among violations: 3.e-05\n'
+                'Max relative difference among violations: inf'
+                in msg)
+        
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(b, a, decimal=7)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 2 (50%)\n'
+            'Max absolute difference among violations: 3.e-05\n'
+            'Max relative difference among violations: 1.'
+            in msg)
 
     def test_simple(self):
         x = np.array([1234.2222])
@@ -417,8 +489,51 @@ def test_simple(self):
 
         self._assert_func(x, y, decimal=3)
         self._assert_func(x, y, decimal=4)
-        assert_raises(AssertionError,
-                lambda: self._assert_func(x, y, decimal=5))
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(x, y, decimal=5)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+            'Max absolute difference among violations: 1.e-04\n'
+            'Max relative difference among violations: 8.10226812e-08'
+            in msg)
+        
+    def test_array_vs_scalar(self):
+        a = [5498.42354, 849.54345, 0.00]
+        b = 5498.42354
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(a, b, decimal=9)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 2 / 3 (66.7%)\n'
+            'Max absolute difference among violations: 5498.42354\n'
+            'Max relative difference among violations: 1.'
+            in msg)
+
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(b, a, decimal=9)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 2 / 3 (66.7%)\n'
+                'Max absolute difference among violations: 5498.42354\n'
+                'Max relative difference among violations: 5.4722099'
+                in msg)
+        
+        a = [5498.42354, 0.00]
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(b, a, decimal=7)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 2 (50%)\n'
+            'Max absolute difference among violations: 5498.42354\n'
+            'Max relative difference among violations: inf'
+            in msg)
+        
+        b = 0
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(a, b, decimal=7)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 2 (50%)\n'
+            'Max absolute difference among violations: 5498.42354\n'
+            'Max relative difference among violations: inf'
+            in msg)
+
 
     def test_nan(self):
         anan = np.array([np.nan])
@@ -468,6 +583,35 @@ def test_subclass(self):
         self._test_equal(a, b)
         self._test_equal(b, a)
 
+    def test_subclass_2(self):
+        # While we cannot guarantee testing functions will always work for
+        # subclasses, the tests should ideally rely only on subclasses having
+        # comparison operators, not on them being able to store booleans
+        # (which, e.g., astropy Quantity cannot usefully do). See gh-8452.
+        class MyArray(np.ndarray):
+            def __eq__(self, other):
+                return super().__eq__(other).view(np.ndarray)
+
+            def __lt__(self, other):
+                return super().__lt__(other).view(np.ndarray)
+
+            def all(self, *args, **kwargs):
+                return all(self)
+
+        a = np.array([1., 2.]).view(MyArray)
+        self._assert_func(a, a)
+
+        z = np.array([True, True]).view(MyArray)
+        all(z)
+        b = np.array([1., 202]).view(MyArray)
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(a, b)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 2 (50%)\n'
+                'Max absolute difference among violations: 200.\n'
+                'Max relative difference among violations: 0.99009'
+                in msg)
+        
     def test_subclass_that_cannot_be_bool(self):
         # While we cannot guarantee testing functions will always work for
         # subclasses, the tests should ideally rely only on subclasses having
@@ -486,7 +630,6 @@ def all(self, *args, **kwargs):
         a = np.array([1., 2.]).view(MyArray)
         self._assert_func(a, a)
 
-
 class TestAlmostEqual(_GenericTest):
 
     def setup_method(self):
@@ -556,8 +699,10 @@ def test_error_message(self):
             self._assert_func(x, y, decimal=12)
         msgs = str(exc_info.value).split('\n')
         assert_equal(msgs[3], 'Mismatched elements: 3 / 3 (100%)')
-        assert_equal(msgs[4], 'Max absolute difference: 1.e-05')
-        assert_equal(msgs[5], 'Max relative difference: 3.33328889e-06')
+        assert_equal(msgs[4], 'Max absolute difference among violations: '
+                     '1.e-05')
+        assert_equal(msgs[5], 'Max relative difference among violations: '
+                     '3.33328889e-06')
         assert_equal(
             msgs[6],
             ' x: array([1.00000000001, 2.00000000002, 3.00003      ])')
@@ -572,8 +717,10 @@ def test_error_message(self):
             self._assert_func(x, y)
         msgs = str(exc_info.value).split('\n')
         assert_equal(msgs[3], 'Mismatched elements: 1 / 3 (33.3%)')
-        assert_equal(msgs[4], 'Max absolute difference: 1.e-05')
-        assert_equal(msgs[5], 'Max relative difference: 3.33328889e-06')
+        assert_equal(msgs[4], 'Max absolute difference among violations: '
+                     '1.e-05')
+        assert_equal(msgs[5], 'Max relative difference among violations: '
+                     '3.33328889e-06')
         assert_equal(msgs[6], ' x: array([1.     , 2.     , 3.00003])')
         assert_equal(msgs[7], ' y: array([1.     , 2.     , 3.00004])')
 
@@ -584,8 +731,8 @@ def test_error_message(self):
             self._assert_func(x, y)
         msgs = str(exc_info.value).split('\n')
         assert_equal(msgs[3], 'Mismatched elements: 1 / 2 (50%)')
-        assert_equal(msgs[4], 'Max absolute difference: 1.')
-        assert_equal(msgs[5], 'Max relative difference: 1.')
+        assert_equal(msgs[4], 'Max absolute difference among violations: 1.')
+        assert_equal(msgs[5], 'Max relative difference among violations: 1.')
         assert_equal(msgs[6], ' x: array([inf,  0.])')
         assert_equal(msgs[7], ' y: array([inf,  1.])')
 
@@ -596,8 +743,8 @@ def test_error_message(self):
             self._assert_func(x, y)
         msgs = str(exc_info.value).split('\n')
         assert_equal(msgs[3], 'Mismatched elements: 2 / 2 (100%)')
-        assert_equal(msgs[4], 'Max absolute difference: 2')
-        assert_equal(msgs[5], 'Max relative difference: inf')
+        assert_equal(msgs[4], 'Max absolute difference among violations: 2')
+        assert_equal(msgs[5], 'Max relative difference among violations: inf')
 
     def test_error_message_2(self):
         """Check the message is formatted correctly when either x or y is a scalar."""
@@ -607,8 +754,8 @@ def test_error_message_2(self):
             self._assert_func(x, y)
         msgs = str(exc_info.value).split('\n')
         assert_equal(msgs[3], 'Mismatched elements: 20 / 20 (100%)')
-        assert_equal(msgs[4], 'Max absolute difference: 1.')
-        assert_equal(msgs[5], 'Max relative difference: 1.')
+        assert_equal(msgs[4], 'Max absolute difference among violations: 1.')
+        assert_equal(msgs[5], 'Max relative difference among violations: 1.')
 
         y = 2
         x = np.ones(20)
@@ -616,8 +763,8 @@ def test_error_message_2(self):
             self._assert_func(x, y)
         msgs = str(exc_info.value).split('\n')
         assert_equal(msgs[3], 'Mismatched elements: 20 / 20 (100%)')
-        assert_equal(msgs[4], 'Max absolute difference: 1.')
-        assert_equal(msgs[5], 'Max relative difference: 0.5')
+        assert_equal(msgs[4], 'Max absolute difference among violations: 1.')
+        assert_equal(msgs[5], 'Max relative difference among violations: 0.5')
 
     def test_subclass_that_cannot_be_bool(self):
         # While we cannot guarantee testing functions will always work for
@@ -698,12 +845,28 @@ def test_simple_arrays(self):
         assert_raises(AssertionError, lambda: self._assert_func(x, y))
         assert_raises(AssertionError, lambda: self._assert_func(y, x))
 
+        a = np.array([1, 3, 6, 20])
+        b = np.array([2, 4, 6, 8])
+
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(a, b)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 2 / 4 (50%)\n'
+                'Max absolute difference among violations: 12\n'
+                'Max relative difference among violations: 1.5' in msg)
+
     def test_rank2(self):
         x = np.array([[1.1, 2.2], [3.3, 4.4]])
         y = np.array([[1.2, 2.3], [3.4, 4.5]])
 
         self._assert_func(x, y)
-        assert_raises(AssertionError, lambda: self._assert_func(y, x))
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(y, x)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 4 / 4 (100%)\n'
+            'Max absolute difference among violations: 0.1\n'
+            'Max relative difference among violations: 0.09090909'
+            in msg)
 
         y = np.array([[1.0, 2.3], [3.4, 4.5]])
 
@@ -718,8 +881,13 @@ def test_rank3(self):
         assert_raises(AssertionError, lambda: self._assert_func(y, x))
 
         y[0, 0, 0] = 0
-
-        assert_raises(AssertionError, lambda: self._assert_func(x, y))
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(x, y)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 8 (12.5%)\n'
+            'Max absolute difference among violations: 1.\n'
+            'Max relative difference among violations: inf' in msg)
+        
         assert_raises(AssertionError, lambda: self._assert_func(y, x))
 
     def test_simple_items(self):
@@ -727,7 +895,12 @@ def test_simple_items(self):
         y = 2.2
 
         self._assert_func(x, y)
-        assert_raises(AssertionError, lambda: self._assert_func(y, x))
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(y, x)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+            'Max absolute difference among violations: 1.1\n'
+            'Max relative difference among violations: 1.' in msg)
 
         y = np.array([2.2, 3.3])
 
@@ -738,6 +911,74 @@ def test_simple_items(self):
 
         assert_raises(AssertionError, lambda: self._assert_func(x, y))
 
+    def test_simple_items_and_array(self):
+        x = np.array([[621.345454, 390.5436, 43.54657, 626.4535],
+                      [54.54, 627.3399, 13., 405.5435],
+                      [543.545, 8.34, 91.543, 333.3]])
+        y = 627.34
+        self._assert_func(x, y)
+
+        y = 8.339999
+        self._assert_func(y, x)
+
+        x = np.array([[3.4536, 2390.5436, 435.54657, 324525.4535],
+                      [5449.54, 999090.54, 130303.54, 405.5435],
+                      [543.545, 8.34, 91.543, 999090.53999]])
+        y = 999090.54
+
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(x, y)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 12 (8.33%)\n'
+            'Max absolute difference among violations: 0.\n'
+            'Max relative difference among violations: 0.' in msg)
+
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(y, x)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 12 / 12 (100%)\n'
+            'Max absolute difference among violations: 999087.0864\n'
+            'Max relative difference among violations: 289288.5934676'
+            in msg)  
+
+    def test_zeroes(self):
+        x = np.array([546456., 0, 15.455])
+        y = np.array(87654.)
+
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(x, y)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 3 (33.3%)\n'
+            'Max absolute difference among violations: 458802.\n'
+            'Max relative difference among violations: 5.23423917'
+            in msg)
+        
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(y, x)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 2 / 3 (66.7%)\n'
+            'Max absolute difference among violations: 87654.\n'
+            'Max relative difference among violations: 5670.5626011'
+            in msg)
+        
+        y = 0
+
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(x, y)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 3 / 3 (100%)\n'
+            'Max absolute difference among violations: 546456.\n'
+            'Max relative difference among violations: inf'
+            in msg)
+        
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(y, x)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 3 (33.3%)\n'
+            'Max absolute difference among violations: 0.\n'
+            'Max relative difference among violations: inf'
+            in msg)
+    
     def test_nan_noncompare(self):
         anan = np.array(np.nan)
         aone = np.array(1)
@@ -849,6 +1090,27 @@ def test_simple(self):
 
         assert_allclose(x, y, atol=1)
         assert_raises(AssertionError, assert_allclose, x, y)
+        with pytest.raises(AssertionError) as exc_info:
+            assert_allclose(x, y)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+            'Max absolute difference among violations: 0.001\n'
+            'Max relative difference among violations: 999999.' in msg)
+        
+        z = 0
+        with pytest.raises(AssertionError) as exc_info:
+            assert_allclose(y, z)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+            'Max absolute difference among violations: 1.e-09\n'
+            'Max relative difference among violations: inf' in msg)
+        
+        with pytest.raises(AssertionError) as exc_info:
+            assert_allclose(z, y)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+            'Max absolute difference among violations: 1.e-09\n'
+            'Max relative difference among violations: 1.' in msg)
 
         a = np.array([x, y, x, y])
         b = np.array([x, y, x, x])
@@ -863,6 +1125,22 @@ def test_simple(self):
         assert_allclose(6, 10, rtol=0.5)
         assert_raises(AssertionError, assert_allclose, 10, 6, rtol=0.5)
 
+        b = np.array([x, y, x, x])
+        c = np.array([x, y, x, z])
+        with pytest.raises(AssertionError) as exc_info:
+            assert_allclose(b, c)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 4 (25%)\n'
+            'Max absolute difference among violations: 0.001\n'
+            'Max relative difference among violations: inf' in msg)
+        
+        with pytest.raises(AssertionError) as exc_info:
+            assert_allclose(c, b)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 4 (25%)\n'
+            'Max absolute difference among violations: 0.001\n'
+            'Max relative difference among violations: 1.' in msg)
+
     def test_min_int(self):
         a = np.array([np.iinfo(np.int_).min], dtype=np.int_)
         # Should not raise:
@@ -876,8 +1154,8 @@ def test_report_fail_percentage(self):
             assert_allclose(a, b)
         msg = str(exc_info.value)
         assert_('Mismatched elements: 1 / 4 (25%)\n'
-                'Max absolute difference: 1\n'
-                'Max relative difference: 0.5' in msg)
+                'Max absolute difference among violations: 1\n'
+                'Max relative difference among violations: 0.5' in msg)
 
     def test_equal_nan(self):
         a = np.array([np.nan])
@@ -908,7 +1186,7 @@ def test_report_max_relative_error(self):
         with pytest.raises(AssertionError) as exc_info:
             assert_allclose(a, b)
         msg = str(exc_info.value)
-        assert_('Max relative difference: 0.5' in msg)
+        assert_('Max relative difference among violations: 0.5' in msg)
 
     def test_timedelta(self):
         # see gh-18286
@@ -927,7 +1205,7 @@ def test_error_message_unsigned(self):
         with pytest.raises(AssertionError) as exc_info:
             assert_allclose(x, y, atol=3)
         msgs = str(exc_info.value).split('\n')
-        assert_equal(msgs[4], 'Max absolute difference: 4')
+        assert_equal(msgs[4], 'Max absolute difference among violations: 4')
 
 
 class TestArrayAlmostEqualNulp:


Code review comments:
diff_hunk: @@ -89,16 +89,27 @@ def foo(t):
     def test_0_ndim_array(self):
         x = np.array(473963742225900817127911193656584771)
         y = np.array(18535119325151578301457182298393896)
-        assert_raises(AssertionError, self._assert_func, x, y)
+        
+        with pytest.raises(AssertionError) as exc_info:
+            self._assert_func(x, y)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 1 (100%)\n'
+        in msg)
path: numpy/testing/tests/test_utils.py
text: @user1:
nit, here and below: you can pass a `match=some_regex` to `pytest.raises` to check for a matching message.

@author:
Thank you for the review, i have updated the error message checks to use match

diff_hunk: @@ -190,6 +201,23 @@ def __ne__(self, other):
         self._test_not_equal(a, b)
         self._test_not_equal(b, a)
 
+        with pytest.raises(AssertionError) as exc_info:
+            self._test_equal(a, b)
+        msg = str(exc_info.value)
+        assert_('Mismatched elements: 1 / 2 (50%)\n'
+                'Max absolute difference among violations: 1.\n'
+                'Max relative difference among violations: 0.5'
+                in msg)
+        
+        c = np.array([0., 2.9]).view(MyArray)
+        with pytest.raises(AssertionError) as exc_info:
+            self._test_equal(b, c)
+        msg = str(exc_info.value)
+        # assert_('Mismatched elements: 2 / 2 (100%)\n'
+        #         'Max absolute difference among violations: 2.\n'
+        #         'Max relative difference among violations: inf'
+        #         in msg)
path: numpy/testing/tests/test_utils.py
text: @user1:
why is this commented out?

@author:
I have uncommented this out, just to confirm the mismatched elements is 1 / 2 and max relative difference among violations should be inf right? (as per the results in the original numpy on the main branch)