Repository: hjwp/pytest-icdiff (language: python)

Issue:
ValueError raised when comparing numpy / pandas iterables
## Context

Given a small test that uses a numpy `array`, working in a clean Python 3.11 virtual environment:

```sh
pip install pytest numpy
cat > test.py
```

```py
import numpy as np

def test() -> None:
    stuff = np.array([1, 2, 3])

    assert all(stuff == 1)
```

`^d`

Test fails as expected.

```sh
pytest test.py
```

Error output from Pytest is clear (just `test`'s failure block shown):

```
_________________________________________________ test __________________________________________________

    def test() -> None:
        stuff = np.array([1, 2, 3])

>       assert all(stuff == 1)
E       assert False
E        +  where False = all(array([1, 2, 3]) == 1)

test.py:6: AssertionError
```

## Current behaviour

When `pytest-icdiff` is installed and tests run, then an error is shown when the test fails because a `ValueError` is raised.

```sh
pip install pytest-icdiff
pytest test.py
```

```
_________________________________________________ test __________________________________________________

    def test() -> None:
        stuff = np.array([1, 2, 3])

>       assert all(stuff == 1)

test.py:6:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv311/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:491: in _call_reprcompare
    custom = util._reprcompare(ops[i], each_obj[i], each_obj[i + 1])
venv311/lib/python3.11/site-packages/_pytest/assertion/__init__.py:141: in callbinrepr
    hook_result = ihook.pytest_assertrepr_compare(
venv311/lib/python3.11/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
venv311/lib/python3.11/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

config = <_pytest.config.Config object at 0x7fddd1908450>, op = '==', left = array([1, 2, 3]), right = 1

    def pytest_assertrepr_compare(config, op, left, right):
        if op != "==":
            return

        try:
>           if abs(left + right) < 19999:
E           ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

venv311/lib/python3.11/site-packages/pytest_icdiff.py:22: ValueError
```

This is happening because the `+`, `abs` and `>` operations in `abs(left + right) < 19999` do not result in a single value. Instead they are propagated across the array, yielding a new array:

```py
In [1]: import numpy as np

In [2]: left = np.array([1, 2, 3])

In [3]: right = 1

In [4]: left + right
Out[4]: array([2, 3, 4])

In [5]: abs(left + right) > 19999
Out[5]: array([False, False, False])
```

Numpy then warns the user if an Array of `bool` is treated as a `bool`:

```py
In [6]: bool(_)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[6], line 1
----> 1 bool(_)

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

## Expected behaviour

With `pytest-icdiff` installed, the original `assert False where False = all(array([1, 2, 3]) == 1)` test failure is shown - no `ValueError` is raised.



Discussion / hints:
this is a great bug report than you so much!
do you want to have a go at fixing it?  i'm thinking the fix would happen here:

https://github.com/hjwp/pytest-icdiff/blob/master/pytest_icdiff.py#L24

```python
    except ValueError:
    # should change to
    except (TypeError, ValueError):
    
    ## also this whole section probably needs a comment.  eg
    try:
       # specialcase small numbers, let normal pytest handle the diff.
        if abs(left + right) < 19999:

```

ofc it'd be lovely to have a test first! otoh, adding numpy as a dependency is a big ask.  i wonder if there's another way to reproduce a ValueError from doing that `left + right`? hmmm
> this is a great bug report than you so much!

Thanks 😊 

> do you want to have a go at fixing it?

Yep - just finding the time 🙏🏻 . I agree with what I think you've said - handling the `ValueError` seems to solve the problem. I then went down the road of trying to build a failing test - but also agree that adding `numpy` is too much as a test dependency.

(I then went searching for a mock-numpy library like moto is to boto, but couldn't find anything... just libraries for building mock data _with_ numpy / pandas 😞 )

My guess is I can build a fake numpy-like class and use it in a test... I'll give that a go. If successful, I'll get it into PR and if I fail, then I'll message back here with findings. 🤞🏻 

Fix patch:
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
index 4b276ab..f69e055 100644
--- a/.github/workflows/ci.yaml
+++ b/.github/workflows/ci.yaml
@@ -16,6 +16,7 @@ jobs:
       - name: Install dependencies
         run: |
           python setup.py develop
+          pip install numpy
 
       - name: Test
         shell: 'script -q -e -c "bash {0}"'


Test patch:
diff --git a/pytest_icdiff.py b/pytest_icdiff.py
index 16fbba5..db41c49 100644
--- a/pytest_icdiff.py
+++ b/pytest_icdiff.py
@@ -23,6 +23,10 @@ def pytest_assertrepr_compare(config, op, left, right):
             return
     except TypeError:
         pass
+    except ValueError:
+        # ValueErrors are raised when numpy / pandas errors are checked
+        # Bail out of generating a diff and use pytest default output
+        return
 
     half_cols = COLS / 2 - MARGINS
 
diff --git a/tests/test_pytest_icdiff.py b/tests/test_pytest_icdiff.py
index 597f030..faa9fcc 100644
--- a/tests/test_pytest_icdiff.py
+++ b/tests/test_pytest_icdiff.py
@@ -2,6 +2,7 @@
 from unittest import mock
 import pytest
 import re
+import sys
 from pprintpp import pformat
 
 YELLOW_ON = '\x1b[1;33m'
@@ -292,3 +293,25 @@ def test_one():
     output = testdir.runpytest('-vv', '--color=yes', '-r=no').stdout.str()
     assert len(output.splitlines()) < 50
     assert "---" in output  # context split marker
+
+
+@pytest.mark.skipif('numpy' not in sys.modules, reason="requires numpy library")
+def test_np_arrays_can_use_equals(testdir) -> None:
+    """
+    Numpy iterables will fall back to pytest default output
+    """
+    testdir.makepyfile("""
+    import numpy as np
+
+    def test():
+        result = np.array([1, 2, 3])
+
+        assert all(result == 2)
+    """)
+
+    result = testdir.runpytest()
+
+    output = result.stdout.str()
+    assert 'ValueError' not in output
+    assert 'AssertionError' in output
+    assert 'where False = all(equals failed' not in output, 'pytest-icdiff not used'