Repository: pandas-dev/pandas

Issue:
pd.Series.diff() on boolean values
#### Code Sample

```python
>> x = pd.Series([True,False,True])
>> x.diff()
0     NaN
1    True
2    True
dtype: object
>> x - x.shift()
0    NaN
1     -1
2      1
dtype: object
```
#### Problem description

It's counter-intuitive that the results of above are different.

The current implementation of pd.Series.diff uses algorithms.diff that subtracts 2 numpy arrays in the following way

```python
out_arr[res_indexer] = arr[res_indexer] - arr[lag_indexer]
```

As pointed [here](https://github.com/numpy/numpy/issues/9251) such behaviour is deprecated in favor to np.diff. But np.diff also treats booleans in binary operations in its own numpy way, that is different from native python (replace False with 0, replace True with 1).

```python
>> np.array([True, False]) - np.array([False, True])
/home/deoxys/miniconda3/lib/python3.6/site-packages/ipykernel_launcher.py:1: DeprecationWarning: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.
array([ True,  True], dtype=bool)
>> np.diff([False, True, False])
array([ True,  True], dtype=bool)
>> True - False
1
>> False - True
-1
```

#### Expected Output

I believe there is no correct way of subtracting booleans. But, it's definitely strange that operations like `x - x.shift()` and `x.diff()` provide different results.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------

commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.11.9-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.1
pytest: None
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
xarray: None
IPython: 6.0.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

</details>

pd.Series.diff() on boolean values
#### Code Sample

```python
>> x = pd.Series([True,False,True])
>> x.diff()
0     NaN
1    True
2    True
dtype: object
>> x - x.shift()
0    NaN
1     -1
2      1
dtype: object
```
#### Problem description

It's counter-intuitive that the results of above are different.

The current implementation of pd.Series.diff uses algorithms.diff that subtracts 2 numpy arrays in the following way

```python
out_arr[res_indexer] = arr[res_indexer] - arr[lag_indexer]
```

As pointed [here](https://github.com/numpy/numpy/issues/9251) such behaviour is deprecated in favor to np.diff. But np.diff also treats booleans in binary operations in its own numpy way, that is different from native python (replace False with 0, replace True with 1).

```python
>> np.array([True, False]) - np.array([False, True])
/home/deoxys/miniconda3/lib/python3.6/site-packages/ipykernel_launcher.py:1: DeprecationWarning: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.
array([ True,  True], dtype=bool)
>> np.diff([False, True, False])
array([ True,  True], dtype=bool)
>> True - False
1
>> False - True
-1
```

#### Expected Output

I believe there is no correct way of subtracting booleans. But, it's definitely strange that operations like `x - x.shift()` and `x.diff()` provide different results.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------

commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.11.9-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.1
pytest: None
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
xarray: None
IPython: 6.0.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
@tarashypka so easiest is to raise on all boolean .shift/.diff operations. Or is there a correct method here?
@jreback My suggestion that in this case it will be correct for 

```
>> x - x.shift()
```
to return
```
0    NaN
1    True
2    True
dtype: object
```
that will be consistent with the way numpy treats booleans or raise an error/deprecation warning as numpy does.

In order to get desired behaviour user may want to clarify his intentions to treat booleans as 0, 1 with
```
>> x.astype(int) - x.astype(int).shift()
```
or
```
>> x.astype(int).diff()
```
that will lead to
```
0    NaN
1   -1.0
2    1.0
dtype: float64
```

Also it may be helpful to raise a warning when user subtracts booleans.

Doing:

    pd.Series([False, True]).diff()

With `numpy.__version__: 1.14.1`, raises:

    Traceback (most recent call last):
      File "D:\Anaconda\lib\site-packages\pandas\core\series.py", line 1527, in diff
        result = algorithms.diff(_values_from_object(self), periods)
      File "D:\Anaconda\lib\site-packages\pandas\core\algorithms.py", line 1547, in diff
        out_arr[res_indexer] = arr[res_indexer] - arr[lag_indexer]
    TypeError: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

`pandas.__version__: 0.22.0`
@tarashypka so easiest is to raise on all boolean .shift/.diff operations. Or is there a correct method here?
@jreback My suggestion that in this case it will be correct for 

```
>> x - x.shift()
```
to return
```
0    NaN
1    True
2    True
dtype: object
```
that will be consistent with the way numpy treats booleans or raise an error/deprecation warning as numpy does.

In order to get desired behaviour user may want to clarify his intentions to treat booleans as 0, 1 with
```
>> x.astype(int) - x.astype(int).shift()
```
or
```
>> x.astype(int).diff()
```
that will lead to
```
0    NaN
1   -1.0
2    1.0
dtype: float64
```

Also it may be helpful to raise a warning when user subtracts booleans.

Doing:

    pd.Series([False, True]).diff()

With `numpy.__version__: 1.14.1`, raises:

    Traceback (most recent call last):
      File "D:\Anaconda\lib\site-packages\pandas\core\series.py", line 1527, in diff
        result = algorithms.diff(_values_from_object(self), periods)
      File "D:\Anaconda\lib\site-packages\pandas\core\algorithms.py", line 1547, in diff
        out_arr[res_indexer] = arr[res_indexer] - arr[lag_indexer]
    TypeError: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

`pandas.__version__: 0.22.0`

Resolving patch:
diff --git a/pandas/core/algorithms.py b/pandas/core/algorithms.py
--- a/pandas/core/algorithms.py
+++ b/pandas/core/algorithms.py
@@ -1915,6 +1915,7 @@ def diff(arr, n, axis=0):
     dtype = arr.dtype
 
     is_timedelta = False
+    is_bool = False
     if needs_i8_conversion(arr):
         dtype = np.float64
         arr = arr.view("i8")
@@ -1923,6 +1924,7 @@ def diff(arr, n, axis=0):
 
     elif is_bool_dtype(dtype):
         dtype = np.object_
+        is_bool = True
 
     elif is_integer_dtype(dtype):
         dtype = np.float64
@@ -1962,6 +1964,8 @@ def diff(arr, n, axis=0):
             result = res - lag
             result[mask] = na
             out_arr[res_indexer] = result
+        elif is_bool:
+            out_arr[res_indexer] = arr[res_indexer] ^ arr[lag_indexer]
         else:
             out_arr[res_indexer] = arr[res_indexer] - arr[lag_indexer]
 


Test patch:
diff --git a/pandas/tests/series/test_analytics.py b/pandas/tests/series/test_analytics.py
--- a/pandas/tests/series/test_analytics.py
+++ b/pandas/tests/series/test_analytics.py
@@ -20,6 +20,7 @@
 from pandas.api.types import is_scalar
 from pandas.core.index import MultiIndex
 from pandas.core.indexes.datetimes import Timestamp
+from pandas.core.indexes.timedeltas import TimedeltaIndex
 import pandas.util.testing as tm
 from pandas.util.testing import (
     assert_almost_equal,
@@ -228,7 +229,7 @@ def test_cummax_timedelta64(self):
         result = s.cummax(skipna=False)
         tm.assert_series_equal(expected, result)
 
-    def test_npdiff(self):
+    def test_np_diff(self):
         pytest.skip("skipping due to Series no longer being an ndarray")
 
         # no longer works as the return type of np.diff is now nd.array
@@ -237,6 +238,67 @@ def test_npdiff(self):
         r = np.diff(s)
         assert_series_equal(Series([nan, 0, 0, 0, nan]), r)
 
+    def test_int_diff(self):
+        # int dtype
+        a = 10000000000000000
+        b = a + 1
+        s = Series([a, b])
+
+        result = s.diff()
+        assert result[1] == 1
+
+    def test_tz_diff(self):
+        # Combined datetime diff, normal diff and boolean diff test
+        ts = tm.makeTimeSeries(name="ts")
+        ts.diff()
+
+        # neg n
+        result = ts.diff(-1)
+        expected = ts - ts.shift(-1)
+        assert_series_equal(result, expected)
+
+        # 0
+        result = ts.diff(0)
+        expected = ts - ts
+        assert_series_equal(result, expected)
+
+        # datetime diff (GH3100)
+        s = Series(date_range("20130102", periods=5))
+        result = s.diff()
+        expected = s - s.shift(1)
+        assert_series_equal(result, expected)
+
+        # timedelta diff
+        result = result - result.shift(1)  # previous result
+        expected = expected.diff()  # previously expected
+        assert_series_equal(result, expected)
+
+        # with tz
+        s = Series(
+            date_range("2000-01-01 09:00:00", periods=5, tz="US/Eastern"), name="foo"
+        )
+        result = s.diff()
+        expected = Series(TimedeltaIndex(["NaT"] + ["1 days"] * 4), name="foo")
+        assert_series_equal(result, expected)
+
+    @pytest.mark.parametrize(
+        "input,output,diff",
+        [([False, True, True, False, False], [nan, True, False, True, False], 1)],
+    )
+    def test_bool_diff(self, input, output, diff):
+        # boolean series (test for fixing #17294)
+        s = Series(input)
+        result = s.diff()
+        expected = Series(output)
+        assert_series_equal(result, expected)
+
+    def test_obj_diff(self):
+        # object series
+        s = Series([False, True, 5.0, nan, True, False])
+        result = s.diff()
+        expected = s - s.shift(1)
+        assert_series_equal(result, expected)
+
     def _check_accum_op(self, name, datetime_series_, check_dtype=True):
         func = getattr(np, name)
         tm.assert_numpy_array_equal(
diff --git a/pandas/tests/series/test_timeseries.py b/pandas/tests/series/test_timeseries.py
--- a/pandas/tests/series/test_timeseries.py
+++ b/pandas/tests/series/test_timeseries.py
@@ -355,48 +355,6 @@ def test_asfreq_datetimeindex_empty_series(self):
         )
         tm.assert_index_equal(expected.index, result.index)
 
-    def test_diff(self):
-        # Just run the function
-        self.ts.diff()
-
-        # int dtype
-        a = 10000000000000000
-        b = a + 1
-        s = Series([a, b])
-
-        rs = s.diff()
-        assert rs[1] == 1
-
-        # neg n
-        rs = self.ts.diff(-1)
-        xp = self.ts - self.ts.shift(-1)
-        assert_series_equal(rs, xp)
-
-        # 0
-        rs = self.ts.diff(0)
-        xp = self.ts - self.ts
-        assert_series_equal(rs, xp)
-
-        # datetime diff (GH3100)
-        s = Series(date_range("20130102", periods=5))
-        rs = s - s.shift(1)
-        xp = s.diff()
-        assert_series_equal(rs, xp)
-
-        # timedelta diff
-        nrs = rs - rs.shift(1)
-        nxp = xp.diff()
-        assert_series_equal(nrs, nxp)
-
-        # with tz
-        s = Series(
-            date_range("2000-01-01 09:00:00", periods=5, tz="US/Eastern"), name="foo"
-        )
-        result = s.diff()
-        assert_series_equal(
-            result, Series(TimedeltaIndex(["NaT"] + ["1 days"] * 4), name="foo")
-        )
-
     def test_pct_change(self):
         rs = self.ts.pct_change(fill_method=None)
         assert_series_equal(rs, self.ts / self.ts.shift(1) - 1)