Repository: numpy/numpy

Issue:
Shift operator cycles for int32 and int64 (Trac #1856)
_Original ticket http://projects.scipy.org/numpy/ticket/1856 on 2011-06-02 by trac user klasj, assigned to unknown._

The bitshift operations, both right and left cycles when they are shifted more bits than
the length of the datatype. This happens for both int32 and int64 but not for int16,
unsigned integers works as expected. In the example below version 1.6.1 is used but
this is also present in at least 1.5.0.
```
In [1]: import numpy as np

In [2]: np.right_shift(np.int32(8),32)

Out[2]: 8

In [3]: np.right_shift(np.int64(8),64)

Out[3]: 8

In [4]: np.right_shift(np.int16(8),16)

Out[4]: 0

In [5]: np.left_shift(np.int32(8),32)

Out[5]: 8

In [6]: np.left_shift(np.int64(8),64)

Out[6]: 8

In [7]: np.left_shift(np.int16(8),16)

Out[7]: 524288

In [8]: np.version.full_version

Out[8]: '1.6.1.dev-08953ea'
```


Discussion:
_@charris wrote on 2011-06-03_

Looks like a compiler thing. On my fedora amd64 system

```
#include <stdio.h>

int main(int argc, char **args)
{
    long x = 8;

    printf("%ld\n", x << 64);

    return 0;
}
```

gives

```
$charris@f13 ~$ ./a.out
8
```

I suspect the problem is that 64 is equal to the register width and so a cyclic shift is used; the problem doesn't occur for a shift of 63. There is probably a compiler flag that can be used to change this...

_@charris wrote on 2011-06-03_

See [http://tinyurl.com/3vqv7qz this]. Apparently the Intel hardware masks the shift bits. On 32 bit machines 5 bits are used, max shift 31, and it seems on 64 bit architectures 6 bits are used.

We should be able to fix this, but the c shift operators are not precisely defined anyway. They can be cyclic or not, arithmetic or logical, or at least that used to be the case. I don't think numpy has a spec for this.

_trac user gregsmith_tor wrote on 2012-06-20_

I would suggest rename to 'numpy shift operators behave as underlying C operator', since that's the issue here. 

The C shift operator _is_ well defined, at least, it has well-defined areas of doubt and uncertainty. The shift amount can range from 0, to one less than the width of the integer type being shifted. So, to get a consistent behaviour you need to check (in the numpy C implementation) the value of the shift and produce a zero (or -1 for >> of negative) if out of range. I don't think it's appropriate to put the burden of that on the numpy user; even worse, to potentially induce users to rely (perhaps accidentally) on the cyclic behaviour or any other result which happens to occur on a specific machine.

There's another issue with shifts and numpy scalars:

I ran into this issue when a simple scalar value (which had been read from a numpy array) did not produce the expected result - it was something like x << 62, where x was 7, and the result was -0x4000000000000000 instead of the expected 0x1C000000000000000. The reason was that 'x' was a numpy.int64 scalar value, which doesn't behave as a python int does. 

This is an issue I've run into before with numpy - since numpy started returning 'numpy scalar' objects from array indexing, instead of python types, we have a situation where these types act very much like the builtin types, and thus it's common that they are passed far into generic python code (code which is not aware of numpy, for instance), and these occasional cases where they don't act like the builtin types are problematic. Some of the problems have been due to python bugs, where the python core code was neglecting to attempt coercion of certain things to integers or longs.

I fixed my shift bug by explicitly converting to int certain values that I was reading from numpy arrays. 
I suggest that
(1) when shift operators are applied to numpy arrays, the code should check for out-of-range shifts and handle appropriately (i.e. the result should not depend on the compiler/machine); result may overflow on left shifts, but otherwise is the same as for python shifts.
(2) when shift operators are applied to numpy _scalar_ integers, that the result should be as for (1), but if a left shift would overflow, the return value should be a numpy int or long type with the proper value. So, operations on numpy scalars would be consistent with operations on python types.

P.S. @charris as I recall right shift of signed types in C is not guaranteed to sign extend; this is considered platform dependent, but I don't know of any machine where it doesn't, and it's probably reasonable to decline full numpy support of such machines.

Still a problem. I think we should fix this in the code with a size check on the shift amount.


Resolving patch:
diff --git a/numpy/core/setup.py b/numpy/core/setup.py
--- a/numpy/core/setup.py
+++ b/numpy/core/setup.py
@@ -463,6 +463,12 @@ def generate_config_h(ext, build_dir):
             rep = check_long_double_representation(config_cmd)
             moredefs.append(('HAVE_LDOUBLE_%s' % rep, 1))
 
+            if check_for_right_shift_internal_compiler_error(config_cmd):
+                moredefs.append('NPY_DO_NOT_OPTIMIZE_LONG_right_shift')
+                moredefs.append('NPY_DO_NOT_OPTIMIZE_ULONG_right_shift')
+                moredefs.append('NPY_DO_NOT_OPTIMIZE_LONGLONG_right_shift')
+                moredefs.append('NPY_DO_NOT_OPTIMIZE_ULONGLONG_right_shift')
+
             # Py3K check
             if sys.version_info[0] == 3:
                 moredefs.append(('NPY_PY3K', 1))
diff --git a/numpy/core/setup_common.py b/numpy/core/setup_common.py
--- a/numpy/core/setup_common.py
+++ b/numpy/core/setup_common.py
@@ -5,6 +5,7 @@
 import warnings
 import copy
 import binascii
+import textwrap
 
 from numpy.distutils.misc_util import mingw32
 
@@ -414,3 +415,41 @@ def long_double_representation(lines):
     else:
         # We never detected the after_sequence
         raise ValueError("Could not lock sequences (%s)" % saw)
+
+
+def check_for_right_shift_internal_compiler_error(cmd):
+    """
+    On our arm CI, this fails with an internal compilation error
+
+    The failure looks like the following, and can be reproduced on ARM64 GCC 5.4:
+
+        <source>: In function 'right_shift':
+        <source>:4:20: internal compiler error: in expand_shift_1, at expmed.c:2349
+               ip1[i] = ip1[i] >> in2;
+                      ^
+        Please submit a full bug report,
+        with preprocessed source if appropriate.
+        See <http://gcc.gnu.org/bugs.html> for instructions.
+        Compiler returned: 1
+
+    This function returns True if this compiler bug is present, and we need to
+    turn off optimization for the function
+    """
+    cmd._check_compiler()
+    has_optimize = cmd.try_compile(textwrap.dedent("""\
+        __attribute__((optimize("O3"))) void right_shift() {}
+        """), None, None)
+    if not has_optimize:
+        return False
+
+    no_err = cmd.try_compile(textwrap.dedent("""\
+        typedef long the_type;  /* fails also for unsigned and long long */
+        __attribute__((optimize("O3"))) void right_shift(the_type in2, the_type *ip1, int n) {
+            for (int i = 0; i < n; i++) {
+                if (in2 < (the_type)sizeof(the_type) * 8) {
+                    ip1[i] = ip1[i] >> in2;
+                }
+            }
+        }
+        """), None, None)
+    return not no_err


Test patch:
diff --git a/numpy/core/tests/test_scalarmath.py b/numpy/core/tests/test_scalarmath.py
--- a/numpy/core/tests/test_scalarmath.py
+++ b/numpy/core/tests/test_scalarmath.py
@@ -664,3 +664,31 @@ def test_builtin_abs(self):
 
     def test_numpy_abs(self):
         self._test_abs_func(np.abs)
+
+
+class TestBitShifts(object):
+
+    @pytest.mark.parametrize('type_code', np.typecodes['AllInteger'])
+    @pytest.mark.parametrize('op',
+        [operator.rshift, operator.lshift], ids=['>>', '<<'])
+    def test_shift_all_bits(self, type_code, op):
+        """ Shifts where the shift amount is the width of the type or wider """
+        # gh-2449
+        dt = np.dtype(type_code)
+        nbits = dt.itemsize * 8
+        for val in [5, -5]:
+            for shift in [nbits, nbits + 4]:
+                val_scl = dt.type(val)
+                shift_scl = dt.type(shift)
+                res_scl = op(val_scl, shift_scl)
+                if val_scl < 0 and op is operator.rshift:
+                    # sign bit is preserved
+                    assert_equal(res_scl, -1)
+                else:
+                    assert_equal(res_scl, 0)
+
+                # Result on scalars should be the same as on arrays
+                val_arr = np.array([val]*32, dtype=dt)
+                shift_arr = np.array([shift]*32, dtype=dt)
+                res_arr = op(val_arr, shift_arr)
+                assert_equal(res_arr, res_scl)