Repository: numpy/numpy

Issue:
Noting ediff1d casting changes in release notes, docs?
As a result of #11490 (ediff1d handles data type errors badly), ediff1d grew some input checks.

These checks are probably a good thing overall, but are also a functional change where code that works OK now gives a `TypeError` and that causes regressions ([pbcore](https://github.com/PacificBiosciences/pbcore) being the one that I'm looking at right now). 

Using a variant of ediff1d's own documentation to illustrate:

With numpy 1.12.1:

    In [1]: import numpy as np
    In [2]: x = np.array([1, 2, 4, 7, 0], dtype=np.int32)
    In [3]: np.ediff1d(x, to_begin=-99, to_end=np.array([88, 99]))
    Out[3]: array([-99,   1,   2,   3,  -7,  88,  99], dtype=int32)

And with numpy 1.16.0rc2:

    In [1]: import numpy as np
    In [2]: x = np.array([1, 2, 4, 7, 0], dtype=np.int32)
    In [3]: np.ediff1d(x, to_begin=-99, to_end=np.array([88, 99]))
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    <ipython-input-3-29beedef979f> in <module>()
    ----> 1 np.ediff1d(x, to_begin=-99, to_end=np.array([88, 99]))
    /usr/lib/python3/dist-packages/numpy/lib/arraysetops.py in ediff1d(ary, to_end, to_begin)
        120         to_end = np.asanyarray(to_end)
        121         if not np.can_cast(to_end, dtype_req):
    --> 122             raise TypeError("dtype of to_end must be compatible "
        123                             "with input ary")
        124 
    TypeError: dtype of to_end must be compatible with input ary

    In [4]: np.ediff1d(x, to_begin=-99, to_end=np.array([88, 99], dtype=np.int32))
    Out[4]: array([-99,   1,   2,   3,  -7,  88,  99], dtype=int32)

Quirkily, the way `can_cast` checks values for array scalars but not for arrays, also results in the following:

    In [5]: np.ediff1d(x, to_begin=[-99], to_end=[99])
    ...
    TypeError: dtype of to_begin must be compatible with input ary

    In [6]: np.ediff1d(x, to_begin=-99, to_end=99)
    Out[6]: array([-99,   1,   2,   3,  -7,  99], dtype=int32)

These last two are both entirely safe and valid code even if they are going through an int64 to int32 cast; the value -99 is preserved in that transformation.

At RC2 stage, I assume that this change is not going to be reverted even if it does break some valid existing code. It would be handy to see this in the release notes and in the `ediff1d` documentation so that those dealing with failing tests have a useful pointer. The key information to include is:

  * if single values are being passed to `to_begin` or `to_end` then pass them as values not lists or arrays
  * if the input array is a narrowed dtype, be explicit about the dtype of the prepended/appended data.

I'd be happy to give you a PR to do that if you want.

### Numpy/Python version information:

    1.12.1 3.5.3 (default, Sep 27 2018, 17:25:39) 
    [GCC 6.3.0 20170516]

vs

    1.16.0rc2 3.7.2 (default, Jan  3 2019, 02:55:40) 
    [GCC 8.2.0]

(all from Debian packages)


Discussion:
I think `1.16.0` may be targeted for ~the weekend, so yeah behavior changes perhaps unlikely for that specific release.

Clear / concise doc improvements are usually welcome of course.

I suppose there might be a case for a more refined check on the danger of overflow / lack of data preservation for the prepend / append behaviors. I'm not sure offhand if we can just change the can_cast stringency (there are some keyword options, but I don't see one that is documented to do exactly what we'd want) or if we'd have to use another helper function, or (hopefully not) have to do checks against something like `np.iinfo(np.int32).max` & so on.
Would this work? (modify the code to proactively use `dtype_req` in `asanyarray`)
```
        to_begin = np.asanyarray(to_begin, dtype=dtype_req).ravel()
        to_begin = np.asanyarray(to_begin)
        if not np.can_cast(to_begin, dtype_req):
            raise TypeError("dtype of to_begin must be compatible "
                            "with input ary")
```

Resolving patch:
diff --git a/numpy/lib/arraysetops.py b/numpy/lib/arraysetops.py
--- a/numpy/lib/arraysetops.py
+++ b/numpy/lib/arraysetops.py
@@ -94,8 +94,7 @@ def ediff1d(ary, to_end=None, to_begin=None):
     # force a 1d array
     ary = np.asanyarray(ary).ravel()
 
-    # we have unit tests enforcing
-    # propagation of the dtype of input
+    # enforce propagation of the dtype of input
     # ary to returned result
     dtype_req = ary.dtype
 
@@ -106,23 +105,22 @@ def ediff1d(ary, to_end=None, to_begin=None):
     if to_begin is None:
         l_begin = 0
     else:
-        to_begin = np.asanyarray(to_begin)
-        if not np.can_cast(to_begin, dtype_req):
-            raise TypeError("dtype of to_begin must be compatible "
-                            "with input ary")
-
-        to_begin = to_begin.ravel()
+        _to_begin = np.asanyarray(to_begin, dtype=dtype_req)
+        if not np.all(_to_begin == to_begin):
+            raise ValueError("cannot convert 'to_begin' to array with dtype "
+                            "'%r' as required for input ary" % dtype_req)
+        to_begin = _to_begin.ravel()
         l_begin = len(to_begin)
 
     if to_end is None:
         l_end = 0
     else:
-        to_end = np.asanyarray(to_end)
-        if not np.can_cast(to_end, dtype_req):
-            raise TypeError("dtype of to_end must be compatible "
-                            "with input ary")
-
-        to_end = to_end.ravel()
+        _to_end = np.asanyarray(to_end, dtype=dtype_req)
+        # check that casting has not overflowed
+        if not np.all(_to_end == to_end):
+            raise ValueError("cannot convert 'to_end' to array with dtype "
+                             "'%r' as required for input ary" % dtype_req)
+        to_end = _to_end.ravel()
         l_end = len(to_end)
 
     # do the calculation in place and copy to_begin and to_end


Test patch:
diff --git a/numpy/core/tests/test_regression.py b/numpy/core/tests/test_regression.py
--- a/numpy/core/tests/test_regression.py
+++ b/numpy/core/tests/test_regression.py
@@ -46,7 +46,7 @@ def test_pickle_transposed(self):
             assert_array_equal(a, b)
 
     def test_typeNA(self):
-        # Issue gh-515 
+        # Issue gh-515
         with suppress_warnings() as sup:
             sup.filter(np.VisibleDeprecationWarning)
             assert_equal(np.typeNA[np.int64], 'Int64')
@@ -2415,3 +2415,11 @@ def test_object_casting_errors(self):
         # gh-11993
         arr = np.array(['AAAAA', 18465886.0, 18465886.0], dtype=object)
         assert_raises(TypeError, arr.astype, 'c8')
+
+    def test_eff1d_casting(self):
+        # gh-12711
+        x = np.array([1, 2, 4, 7, 0], dtype=np.int16)
+        res = np.ediff1d(x, to_begin=-99, to_end=np.array([88, 99]))
+        assert_equal(res, [-99,   1,   2,   3,  -7,  88,  99])
+        assert_raises(ValueError, np.ediff1d, x, to_begin=(1<<20))
+        assert_raises(ValueError, np.ediff1d, x, to_end=(1<<20))
diff --git a/numpy/lib/tests/test_arraysetops.py b/numpy/lib/tests/test_arraysetops.py
--- a/numpy/lib/tests/test_arraysetops.py
+++ b/numpy/lib/tests/test_arraysetops.py
@@ -136,8 +136,8 @@ def test_ediff1d(self):
          np.nan),
         # should fail because attempting
         # to downcast to smaller int type:
-        (np.array([1, 2, 3], dtype=np.int32),
-         np.array([5, 7, 2], dtype=np.int64),
+        (np.array([1, 2, 3], dtype=np.int16),
+         np.array([5, 1<<20, 2], dtype=np.int32),
          None),
         # should fail because attempting to cast
         # two special floating point values
@@ -152,8 +152,8 @@ def test_ediff1d_forbidden_type_casts(self, ary, prepend, append):
         # specifically, raise an appropriate
         # Exception when attempting to append or
         # prepend with an incompatible type
-        msg = 'must be compatible'
-        with assert_raises_regex(TypeError, msg):
+        msg = 'cannot convert'
+        with assert_raises_regex(ValueError, msg):
             ediff1d(ary=ary,
                     to_end=append,
                     to_begin=prepend)