Repository: pandas-dev/pandas

Issue:
ExtensionArray.shift incorrect for periods larger than length of array
```python
In [9]: from pandas.tests.extension.decimal import to_decimal

In [10]: to_decimal([1, 2]).shift(4)
Out[10]:
DecimalArray(array([Decimal('NaN'), Decimal('NaN'), Decimal('NaN'), Decimal('NaN')],
      dtype=object))
```

The expected output is

```python
>>> to_decimal(['NaN', 'NaN'])
DecimalArray(array([Decimal('NaN'), Decimal('NaN')], dtype=object))
```

similar to categorical

```python
In [12]: pd.Categorical([1, 2]).shift(4)
Out[12]:
[NaN, NaN]
Categories (2, int64): [1, 2]
```

This will need a new base test in `tests/extension/base/methods.py` and a fix in https://github.com/pandas-dev/pandas/blob/a7b187ab42c48342478ed8664145f21578f50622/pandas/core/arrays/base.py#L430


Discussion:
Hi,

I would like to try and tackle this issue to bite my teeth into the pandas codebase, however I'm not entirely sure about the expected behaviour of the shift method.

From what I understand, when shift is called, the first _period_ elements (up to the array's len) become NaN, and if period is smaller than the len of the array the original _len-period_ elements become the last _len-period_ elements, respecting the condition that _period_+_len-period_ == len of the original array.

Is this correct?
@Devilmoon thanks. I think your summary is correct.

The implementation is correct when `abs(period) < len(self)`. The only bug is when `periods` is larger (in absolute terms).

The issue is on https://github.com/pandas-dev/pandas/blob/a7b187ab42c48342478ed8664145f21578f50622/pandas/core/arrays/base.py#L454


That should be

```python
        empty = self._from_sequence(
            [self.dtype.na_value] * min(abs(periods), len(self)),
            dtype=self.dtype
        )
 ```
@TomAugspurger I'm trying to grok the correct way to put in a test for this.

I want to understand why a new base test is required under `tests/extension/base/methods.py` - wouldn't `test_container_shift` also cover testing the DecimalArray (generated by `to_decimal`) by casting it to a `Series`? Couldn't we just add the relevant test cases under `@pytest.mark.parametrize` and be done? Or am I thinking about this wrong?
We write base tests for things that should be true of all extension array implementations http://pandas-docs.github.io/pandas-docs-travis/extending.html#testing-extension-arrays

I think something like

```python
def test_shift_large_periods(self, data, na_value):
    data = data[:5]
    result = data.shift(10)
    expected = type(data)._construct_from_sequence([na_value] * 10)
    self.assert_extension_array_equal(result, expected)
```

should work, but I haven't tested it. Maybe also test for `periods=-10`.

Understood. Different question, why then should it be in that particular file which contains tests under the `BaseMethodsTest` class that are meant for Series and DataFrame methods?
It could also go under the interface tests.

On Mon, Nov 26, 2018 at 11:42 AM Varad Gunjal <notifications@github.com>
wrote:

> Understood. Different question, why then should it be in that particular
> file which contains tests under the BaseMethodsTest class that are meant
> for Series and DataFrame methods?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/23911#issuecomment-441730319>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHItGGYotleCMFBKt-a9pEH6lYpgSWks5uzCgcgaJpZM4YyCb_>
> .
>

I followed a slightly different approach for the tests in the PR. Please let me know if that works.

Resolving patch:
diff --git a/pandas/core/arrays/base.py b/pandas/core/arrays/base.py
--- a/pandas/core/arrays/base.py
+++ b/pandas/core/arrays/base.py
@@ -418,13 +418,13 @@ def fillna(self, value=None, method=None, limit=None):
         return new_values
 
     def dropna(self):
-        """ Return ExtensionArray without NA values
+        """
+        Return ExtensionArray without NA values
 
         Returns
         -------
         valid : ExtensionArray
         """
-
         return self[~self.isna()]
 
     def shift(self, periods=1):
@@ -446,13 +446,25 @@ def shift(self, periods=1):
         Returns
         -------
         shifted : ExtensionArray
+
+        Notes
+        -----
+        If ``self`` is empty or ``periods`` is 0, a copy of ``self`` is
+        returned.
+
+        If ``periods > len(self)``, then an array of size
+        len(self) is returned, with all values filled with
+        ``self.dtype.na_value``.
         """
         # Note: this implementation assumes that `self.dtype.na_value` can be
         # stored in an instance of your ExtensionArray with `self.dtype`.
-        if periods == 0:
+        if not len(self) or periods == 0:
             return self.copy()
-        empty = self._from_sequence([self.dtype.na_value] * abs(periods),
-                                    dtype=self.dtype)
+
+        empty = self._from_sequence(
+            [self.dtype.na_value] * min(abs(periods), len(self)),
+            dtype=self.dtype
+        )
         if periods > 0:
             a = empty
             b = self[:-periods]
@@ -462,7 +474,8 @@ def shift(self, periods=1):
         return self._concat_same_type([a, b])
 
     def unique(self):
-        """Compute the ExtensionArray of unique values.
+        """
+        Compute the ExtensionArray of unique values.
 
         Returns
         -------
diff --git a/pandas/core/arrays/sparse.py b/pandas/core/arrays/sparse.py
--- a/pandas/core/arrays/sparse.py
+++ b/pandas/core/arrays/sparse.py
@@ -882,7 +882,7 @@ def fillna(self, value=None, method=None, limit=None):
 
     def shift(self, periods=1):
 
-        if periods == 0:
+        if not len(self) or periods == 0:
             return self.copy()
 
         subtype = np.result_type(np.nan, self.dtype.subtype)
@@ -893,8 +893,11 @@ def shift(self, periods=1):
         else:
             arr = self
 
-        empty = self._from_sequence([self.dtype.na_value] * abs(periods),
-                                    dtype=arr.dtype)
+        empty = self._from_sequence(
+            [self.dtype.na_value] * min(abs(periods), len(self)),
+            dtype=arr.dtype
+        )
+
         if periods > 0:
             a = empty
             b = arr[:-periods]


Test patch:
diff --git a/pandas/tests/extension/base/methods.py b/pandas/tests/extension/base/methods.py
--- a/pandas/tests/extension/base/methods.py
+++ b/pandas/tests/extension/base/methods.py
@@ -189,6 +189,30 @@ def test_container_shift(self, data, frame, periods, indices):
 
         compare(result, expected)
 
+    @pytest.mark.parametrize('periods, indices', [
+        [-4, [-1, -1]],
+        [-1, [1, -1]],
+        [0, [0, 1]],
+        [1, [-1, 0]],
+        [4, [-1, -1]]
+    ])
+    def test_shift_non_empty_array(self, data, periods, indices):
+        # https://github.com/pandas-dev/pandas/issues/23911
+        subset = data[:2]
+        result = subset.shift(periods)
+        expected = subset.take(indices, allow_fill=True)
+        self.assert_extension_array_equal(result, expected)
+
+    @pytest.mark.parametrize('periods', [
+        -4, -1, 0, 1, 4
+    ])
+    def test_shift_empty_array(self, data, periods):
+        # https://github.com/pandas-dev/pandas/issues/23911
+        empty = data[:0]
+        result = empty.shift(periods)
+        expected = empty
+        self.assert_extension_array_equal(result, expected)
+
     @pytest.mark.parametrize("as_frame", [True, False])
     def test_hash_pandas_object_works(self, data, as_frame):
         # https://github.com/pandas-dev/pandas/issues/23066