Repository: numpy/numpy

Issue:
Value of bitwise_and.identity does not make sense
The value of `bitwise_and.identity` is `1`. This makes `bitwise_and.reduce()` basically useless for numbers other than zeros and ones. The value can not be changed or a different one passed in to `reduce()`. The correct value is `-1` or `0xFFF...` or something similar that will not change the bits of the initial value of the thing being reduced. A related question on StackOverflow can be found here: http://stackoverflow.com/questions/21050875/numpy-bitwise-and-reduce-behaving-unexpectedly.



Discussion:
On first sight, changing it to -1 should probably work correctly for all types and continue to work for all cases where it used to make sense. So I think we should do it, do you want to create a PR?

Will do shortly.

I was looking at the code and while `-1` is technically correct on two's complement systems, it is incorrect on one's complement machines. I think that a more appropriate value would actually be `None`. The `PyArray_InitializeReduceResult` routine in `numpy/core/src/umath/reduction.c` will use the first element in the array, which is perfect for the reduction in this case.

Yes, I was not quite sure about non ones complement (actually, I have no clue if they exist in practice or not :)). The other issue is the `object` dtype. `-1` as an integer seems to work reasonably there though, though of course it could be wrong, but that is similar problem in other cases as well.

But you are right, when in doubt refuse the temptation to guess. `None` is likely safer. The difference is what happens with empty arrays, they cannot be handled without an identiy and an error will be raised. There are some interesting points, such as `np.subtract` has no idenity, and if it was 0 it would return a different value. But I guess subtract is pretty special.

`bitwise_and` is pretty sensitive, I dunno about special. It will not work for non-integer arrays anyway, so I figure that empty arrays can raise an exception all they want. It actually makes sense that way. I created https://github.com/numpy/numpy/pull/7061 with `ReorderableNone` as the identity.

Has there been any effort to add an optional identity parameter to `ufunc.reduce()`? The default could be to use the identity attribute of the `ufunc`, preserving backwards compatibility. That would make `subtract`, `bitwise_and` and probably others easier to work with in many situations.

However, just changing to no identiy can break peoples code, so I am a bit hesitant. It could be nice to make dtype specific identities, but I doubt that is currently easy. My guess is the only safe place to use it can just as well (and probably is already) use a logical_and.reduce, so likely we are fine.

@madphysicist, I have personally thought that it oculd make sense to have identity as a parameter there, but I don't think we ever discussed it seriously.

Setting this to `None` will definitely break code -- _lots_ of code uses `&` on boolean arrays, and that's `bitwise_and`. There's no comparable shorthand for `logical_and`.

I am 100% unworried about ones-complement machines. There are none that numpy cares about (it's not clear whether they even still exist as physical hardware, even in niche applications), and I'm 99% sure we already assume twos-complement all over the place.

Fair enough. However, won't setting -1 break just as much code since the current value is one? Does `&` on boolean arrays use the identity at any point? I though that was just there for `reduce()`, which is already broken.

@njsmith no, code uses the & operator, that has nothing to do with the identity since it is not a reduction operation. And very little code should do `np.bitwise_and.reduce(bool_arr)` and not `np.all(bool_ar)` or `np.logical_and.reduce(bool_ar)` or one of those alternatives. Though of course some code may do it.

What might need some thinking is whether or not it is nonsense for object arrays, but basically that just gets better with fixing it to -1. I would even prefer some destiction between types + an identity kwarg if someone really needs it, but....
So my guess is also that we gain by putting it to -1, while disabling might cause some distress. And the possible awkwardness of -1 should be almost non-existing.

@seberg I agree that -1 is certainly much better than the current value of 1. I am not sure that disabling it will cause any significant distress since the current defective value has not been noticed/reported until now. The number of people passing empty arrays to `bitwise_and.reduce()` must be miniscule. If you think that -1 is still optimal, I will rescind my pull request, but I am not sure where/how to set the correct value in that case.

This `numba` post https://github.com/numba/numba/issues/829#issuecomment-69209598 is part of what convinced me to set the value to `ReorderableNone` by the way.

@seberg: Oh, you're right of course :-)

I guess that possibly we could get away with removing the identity altogether then, but I am still a bit wary of unforeseen breakage, given that there actually is a correct value we could in principle switch to instead. (At least for non-object arrays, I mean; for object arrays I agree that no-identity is the right thing.) It's jsut unfortunate that fixing it properly is difficult given how ufuncs are currently put together :-/

This would need a post on the mailing list + a deprecation cycle I think.


Resolving patch:
diff --git a/numpy/core/code_generators/generate_umath.py b/numpy/core/code_generators/generate_umath.py
--- a/numpy/core/code_generators/generate_umath.py
+++ b/numpy/core/code_generators/generate_umath.py
@@ -13,6 +13,7 @@
 Zero = "PyUFunc_Zero"
 One = "PyUFunc_One"
 None_ = "PyUFunc_None"
+AllOnes = "PyUFunc_MinusOne"
 ReorderableNone = "PyUFunc_ReorderableNone"
 
 # Sentinel value to specify using the full type description in the
@@ -493,7 +494,7 @@ def english_upper(s):
           TD(flts, f="logaddexp2", astype={'e':'f'})
           ),
 'bitwise_and':
-    Ufunc(2, 1, One,
+    Ufunc(2, 1, AllOnes,
           docstrings.get('numpy.core.umath.bitwise_and'),
           None,
           TD(bints),
@@ -507,7 +508,7 @@ def english_upper(s):
           TD(O, f='PyNumber_Or'),
           ),
 'bitwise_xor':
-    Ufunc(2, 1, None,
+    Ufunc(2, 1, Zero,
           docstrings.get('numpy.core.umath.bitwise_xor'),
           None,
           TD(bints),


Test patch:
diff --git a/numpy/core/tests/test_umath.py b/numpy/core/tests/test_umath.py
--- a/numpy/core/tests/test_umath.py
+++ b/numpy/core/tests/test_umath.py
@@ -991,6 +991,91 @@ def test_truth_table_bitwise(self):
         assert_equal(np.bitwise_xor(arg1, arg2), out)
 
 
+class TestBitwiseUFuncs(TestCase):
+
+    bitwise_types = [np.dtype(c) for c in '?' + 'bBhHiIlLqQ' + 'O']
+
+    def test_values(self):
+        for dt in self.bitwise_types:
+            zeros = np.array([0], dtype=dt)
+            ones = np.array([-1], dtype=dt)
+            msg = "dt = '%s'" % dt.char
+
+            assert_equal(np.bitwise_not(zeros), ones, err_msg=msg)
+            assert_equal(np.bitwise_not(ones), zeros, err_msg=msg)
+
+            assert_equal(np.bitwise_or(zeros, zeros), zeros, err_msg=msg)
+            assert_equal(np.bitwise_or(zeros, ones), ones, err_msg=msg)
+            assert_equal(np.bitwise_or(ones, zeros), ones, err_msg=msg)
+            assert_equal(np.bitwise_or(ones, ones), ones, err_msg=msg)
+
+            assert_equal(np.bitwise_xor(zeros, zeros), zeros, err_msg=msg)
+            assert_equal(np.bitwise_xor(zeros, ones), ones, err_msg=msg)
+            assert_equal(np.bitwise_xor(ones, zeros), ones, err_msg=msg)
+            assert_equal(np.bitwise_xor(ones, ones), zeros, err_msg=msg)
+
+            assert_equal(np.bitwise_and(zeros, zeros), zeros, err_msg=msg)
+            assert_equal(np.bitwise_and(zeros, ones), zeros, err_msg=msg)
+            assert_equal(np.bitwise_and(ones, zeros), zeros, err_msg=msg)
+            assert_equal(np.bitwise_and(ones, ones), ones, err_msg=msg)
+
+    def test_types(self):
+        for dt in self.bitwise_types:
+            zeros = np.array([0], dtype=dt)
+            ones = np.array([-1], dtype=dt)
+            msg = "dt = '%s'" % dt.char
+
+            assert_(np.bitwise_not(zeros).dtype == dt, msg)
+            assert_(np.bitwise_or(zeros, zeros).dtype == dt, msg)
+            assert_(np.bitwise_xor(zeros, zeros).dtype == dt, msg)
+            assert_(np.bitwise_and(zeros, zeros).dtype == dt, msg)
+
+
+    def test_identity(self):
+        assert_(np.bitwise_or.identity == 0, 'bitwise_or')
+        assert_(np.bitwise_xor.identity == 0, 'bitwise_xor')
+        assert_(np.bitwise_and.identity == -1, 'bitwise_and')
+
+    def test_reduction(self):
+        binary_funcs = (np.bitwise_or, np.bitwise_xor, np.bitwise_and)
+
+        for dt in self.bitwise_types:
+            zeros = np.array([0], dtype=dt)
+            ones = np.array([-1], dtype=dt)
+            for f in binary_funcs:
+                msg = "dt: '%s', f: '%s'" % (dt, f)
+                assert_equal(f.reduce(zeros), zeros, err_msg=msg)
+                assert_equal(f.reduce(ones), ones, err_msg=msg)
+
+        # Test empty reduction, no object dtype
+        for dt in self.bitwise_types[:-1]:
+            # No object array types
+            empty = np.array([], dtype=dt)
+            for f in binary_funcs:
+                msg = "dt: '%s', f: '%s'" % (dt, f)
+                tgt = np.array(f.identity, dtype=dt)
+                res = f.reduce(empty)
+                assert_equal(res, tgt, err_msg=msg)
+                assert_(res.dtype == tgt.dtype, msg)
+
+        # Empty object arrays use the identity.  Note that the types may
+        # differ, the actual type used is determined by the assign_identity
+        # function and is not the same as the type returned by the identity
+        # method.
+        for f in binary_funcs:
+            msg = "dt: '%s'" % (f,)
+            empty = np.array([], dtype=object)
+            tgt = f.identity
+            res = f.reduce(empty)
+            assert_equal(res, tgt, err_msg=msg)
+
+        # Non-empty object arrays do not use the identity
+        for f in binary_funcs:
+            msg = "dt: '%s'" % (f,)
+            btype = np.array([True], dtype=object)
+            assert_(type(f.reduce(btype)) is bool, msg)
+
+
 class TestInt(TestCase):
     def test_logical_not(self):
         x = np.ones(10, dtype=np.int16)