DatetimeIndex 0 length shift (GH9903)
State: closed

closes #9903 

Fixes a bug where calling shift on an empty DatetimeIndex would result in the returned value being an Index rather than a DatetimeIndex.


Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
pls add a release note

@jreback just calling `shallow_copy` with the same list comprehension doesn't quite work because a list of Timestamps doesn't get its dtype inferred correctly and `DatetimeIndex.simple_new` ends up trying to convert to int64s, at which point everything goes to hell.  Explicitly constructing a numpy array and forwarding our dtype seems to do the right thing though.

Also, release note added.

minor comment - pls ping when pushed and green

lgtm otherwise 

```
In [2]: x = pd.DatetimeIndex([])

In [4]: x._shallow_copy([],dtype=x.dtype)
Out[4]: 
<class 'pandas.tseries.index.DatetimeIndex'>
Length: 0, Freq: None, Timezone: None
```

you can just do this, e.g. pass in the dtype explicity. Trying to convert thru numpy arrays is quite tricky actually. You have to always have it in an i8 format (and not an array of say Timestamps).

@jreback calling `shallow_copy` in the `Index` constructor with a list causes failures in tests that hit the same code path with a **non-empty** `DatetimeIndex`.  The sequence of events is roughly as follows:

Code calls `DatetimeIndex.shift`, which hits the following code path:

```
        if freq is not None and freq != self.freq:
            if isinstance(freq, compat.string_types):
                freq = to_offset(freq)
-->         result = Index.shift(self, n, freq)
```

With your suggestion, in `Index.shift`, we do:

```
return self._shallow_copy(
    [val + offset for val in self],
    dtype=self.dtype,
)
```

This eventually causes us to hit a call to `DatetimeIndexOpsMixin._simple_new`, which looks like:

```
def _simple_new(cls, values, name=None, freq=None, tz=None, **kwargs):
    if not getattr(values,'dtype',None):
        values = np.array(values,copy=False)
    if values.dtype != _NS_DTYPE:
        values = com._ensure_int64(values).view(_NS_DTYPE)
```

Here, `values` is a list of Timestamps, which means that it doesn't have a `dtype` attribute, so we end up constructing an array of dtype `object` containing all our timestamps, which causes us to barf in `_ensure_int64`.

My original solution to this problem was to explicitly construct a numpy array in `Index.shift`, but that failed travis on the Python 2.6 build only.

One possible solution for all of this would be to stop deferring to the base `Index.shift` and make the call to `_shallow_copy` directly in `DatetimeIndex.shift`.  This would make it easier to ensure that dtypes get set/inferred correctly, but I'm not knowledgeable enough about what the various class responsibilities are supposed to be to know if that's a reasonable solution.

the issue is that `_shallow_copy` expects things to be already fully formed. 
- So options are to make `.shift` in `pandas/tseries/index`
- use `.asi8` on the array (but this 'loses' the freq info, though shifting is conceptually very simple)

let's do option 1

> So options are to make .shift in pandas/tseries/index

There is already an implementation of `shift` in `DatetimeOpsMixin`, from which `DatetimeIndex` inherits.  Are you saying that we should add an additional specialization in `DatetimeIndex` itself?

ahh that is not being called as it should be (and does the right thing)

may need to switch the order of the inheritance as I think the mro is hitting Index.shift first (wrongly )

`DatetimeOpsMixin` explicitly calls `Index.shift` in the pathological codepaths [here](https://github.com/pydata/pandas/blob/master/pandas/tseries/base.py#L449).  I wonder if that superclass call is no longer appropriate?

ahh ok yes makes more sense to just fix it and not call the simpler superclass 

xref #8038 as well.

can you rebase?

can you update? 

@ssanderson can you rebase / update

closing, but if you'd like to update, pls reopen