ENH: create Timedelta scalar / TimedeltaIndex
State: closed

closes #3009
closes #8187
closes #8209
closes #4533 
closes #8190
closes #7869 
closes #8214 

This PR creates a new `timedelta` sub-class, the scalar `Timedelta`, very similar to how `Timestamp` sub-classes `datetime`. Further it creates a `TimedeltaIndex` modelled on `DatetimeIndex` that holds the underlying data (boxing is only on iteration).
- ToDo
  - Document API changes:
    - [x] `to_timedelta` return `TimedeltaIndex` rather than a `Series`
    - [x] `to_timedelta` with scalar returns `Timedelta` (rather than `np.timedelta64`)
    - [x] Boxing for `list(Series(...))` is now `Timedelta`
    - [x] possible repr changes
    - [x] `timedelta_range`
    - [x] usage / examples
    - [x] API
  - Timedelta
    - [x] conversions `to_timedelta/to_pytimedelta`
  - TimedeltaIndex
    - [x] conversions `to_pytimedelta`
  - [x] what else needs boxing in Series?
  - [x] fix formatters (Timedelta64Formatter; remove `repr_timedelta64` (no longer necessary)
  - [x] create `timedelta_range` & fix freq / inferred_freq
  - [x] fix remaining copied methods from DatetimeIndex
  - more tests (of TimedeltaIndex / scalar)
    - [x] scalar comparisons
    - [x] index comparisons
    - [x] getitem / partial string access
    - [x] sql expecting `np.timedelta64` ? (currently skipping these 2 tests!!!!), see: https://github.com/jreback/pandas/commit/2743ef16f7dfe55ba63663fa65c5671031af94f1
    - [x] validate series / td ops
  - move methods of `TimedeltaIndex/DatetimeIndex`
    - [x] move `DatetimeOpsMixin` to `tseries/base.py`?
    - [x] `shift/take/order`
    - [x] `slice_indexer`
    - [x] `append/union/intersection`
    - [x] comparbless?
    - [x] `.dt` accessor are ok now
    - [x] move `.td` to `.tdi`


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design
name: Internals
description: Related to non-user accessible pandas implementation
name: Timedelta
description: Timedelta data type

Comments:
Hi Jeff,

I was attached on the last thread and wanted to add some comments.  First, let me say that this is a really nice addition and seeing the output helps clarify some of the points you mentioned to me in an email.

One question I have is can the TimeDeltaIndex be constructed directly from a DatetimeIndex?  Something like:

```
@classmethod
def from_datetimeindex(cls, datetimeindex):
    ....
```

For experimental research, data is often timestamped, but users would like to have access to the interval representation as well as the original timestamps.  Maintaining both representations is what motivated me to create a hybrid Index that I spoke of.  I'd imagine that even with this new index, converting between DatetimeIndex and TimeDelta would mainly be out of the scope of these classes per-se, and left to any application that needs to implement such behavior.  Or am I mistaken?

I also wanted to share a notebook with you that shows the `TimeIndex` hybrid that I wrote, just in case seeing it clarifies some of the questions I had asked previously.  As you see, the `convert()` method transfers between various representations of the data.  Now that you've built the TimeDelta index, I think I will move all of my intervals into this index, and maybe add some methods like `TimeDelta.as_seconds()` which would return an array of total seconds etc... or is the behavior already implemented?  Part of the motivation for this is for plotting; if the user is working in a system of total seconds, then when they plot, they'd probably want seconds on the axis not timestamps.  Again, this is custom behavior that I've implemented in a hybrid index, but want to refactor to utilize the new index in 0.15.  

Here's the notebook if you have time to glance at it, maybe it will clarify what I'm trying to say.  The TimeIndex section is the important one.  Also, the SpecIndex does indeed subclass Float64 as you suggested:

http://nbviewer.ipython.org/github/hugadams/pyuvvis/blob/master/examples/Notebooks/units.ipynb
Thanks

@hugadams I think the way you spell `from_datetimeindex` is `datetimeindex - datetimeindex[0]` (or insert your preferred reference time).

I agree that conversion to floating point is useful, and maybe we want something like a `to_seconds()` method, but again you can already do very succinctly with arithmetic: `timedeltaindex / pd.to_timedelta('1s')`.

who knew this would work!

```
In [4]: pd.timedelta_range('1 days, 00:00:02',periods=5,freq='2h')
Out[4]: 
<class 'pandas.tseries.td.TimedeltaIndex'>
[1 days, 00:00:02, ..., 1 days, 08:00:02]
Length: 5, Freq: <2 * Hours>

In [5]: pd.timedelta_range('1 days, 00:00:02',periods=5,freq='2h').tolist()
Out[5]: 
[Timedelta('1 days, 00:00:02'),
 Timedelta('1 days, 02:00:02'),
 Timedelta('1 days, 04:00:02'),
 Timedelta('1 days, 06:00:02'),
 Timedelta('1 days, 08:00:02')]
```

Thanks Stephan.  That's a good point.  I didn't realize you could pass those types of string arguments into to_timedelta(), which I believe Jeff was trying to explain to me and my dense self.

@jorisvandenbossche 

https://travis-ci.org/jreback/pandas/jobs/34546538

I am having some sql failures when passing `Timedelta` (which is a sub-class of `timedelta` which is what was passed before _i think_).

Is their some sort of conversion done on `Timestamp` (which is the same thing I would need to do for `Timedelta`)

?

No, now `timedelta64` columns were actually int64's, and they were written to the database as such. Now it get's a `Timedelta` object instead of an int, and it chokes. 

But SQLAlchemy has a `Interval` type (http://docs.sqlalchemy.org/en/rel_0_9/core/types.html#sqlalchemy.types.Interval) that can handle `datetime.timedelta`. So I think there are two options: 
1. keep it as is (store timedelta as int) and therefore convert Timedelta columns to timedelta64/int64 on input
2. or use the `Interval` type and make sqlalchemy aware of to treat `Timedelta` the same as `datetime.timedelta` (this will store it for postgresql in a native timedelta format, see http://www.postgresql.org/docs/9.1/static/datatype-datetime.html, for others like MySQL it will store it as datetime from 1970)

See here for some discussion on it https://github.com/pydata/pandas/issues/6921 and here for the subsequent PR #7076

For `Timestamp`, there should be some conversion (on some drivers it fails), but this is not yet done. See #7103 for that

a small not-code related comment: I think `td.py` is not a very clear file name, why not just `timedelta.py`?

I have this method of `.components` on an individual `Timedelta`.
I could expose it on a `TimedeltaIndex` (and actually just return a frame)
though its prob not super duper performant (its in cython on the indivual timedeltas).

Normally people just use the Timedeltas as the bottom (e.g. express as say number of seconds).
But the breakdown is useful no? (its actually how I do the repr)

```
In [33]: s = Series(timedelta_range('1 day 23:59:59',periods=4,freq='s'))

In [34]: s
Out[34]: 
0   1 days, 23:59:59
1   2 days, 00:00:00
2   2 days, 00:00:01
3   2 days, 00:00:02
dtype: timedelta64[ns]

In [35]: s.dt.components()
Out[35]: 
   sign  days  hours  minutes  seconds  milli  micro  nano
0     1     1     23       59       59      0      0     0
1     1     2      0        0        0      0      0     0
2     1     2      0        0        1      0      0     0
3     1     2      0        0        2      0      0     0

In [36]: s.dt.minutes
Out[36]: 
0    2880
1    2880
2    2880
3    2880
dtype: int64
```

I think the components breakdown is quite useful personally.

@jorisvandenbossche @cpcloud @shoyer 

ok ready for comments. I think this is pretty much ready. Prob a couple of doc issues (though I _think_ I got most of them).

generally looks OK to me but there is a lot of code here :).

I will try to review tomorrow

I have implemented an API change for the combination of `DatetimeIndex` and `TimedeltaIndex` when for +/-. These make it have numeric operations (rather than union and diff which are meaningless). https://github.com/jreback/pandas/commit/7b6f30c652a0df6ed06b73fc32591e7109e4f8cd

So the following are then possible. (these are similar to how the DatetimeIndex/TimedeltaIndex now interact with Timestamp/Timedelta scalars, e.g. its numeric).

```
In [1]: tdi = pd.TimedeltaIndex(['1 days',pd.NaT,'2 days'])

In [2]: tdi.tolist()
Out[2]: [Timedelta('1 days, 00:00:00'), NaT, Timedelta('2 days, 00:00:00')]

In [3]: dti = date_range('20130101',periods=3)

In [4]: dti.tolist()
Out[4]: 
[Timestamp('2013-01-01 00:00:00', offset='D'),
 Timestamp('2013-01-02 00:00:00', offset='D'),
 Timestamp('2013-01-03 00:00:00', offset='D')]

In [5]: (dti + tdi).tolist()
Out[5]: [Timestamp('2013-01-02 00:00:00'), NaT, Timestamp('2013-01-05 00:00:00')]

In [6]: (dti - tdi).tolist()
Out[6]: [Timestamp('2012-12-31 00:00:00'), NaT, Timestamp('2013-01-01 00:00:00')]

In [7]: (tdi + tdi).tolist()
Out[7]: [Timedelta('2 days, 00:00:00'), NaT, Timedelta('4 days, 00:00:00')]

In [8]: (tdi - tdi).tolist()
Out[8]: [Timedelta('0 days, 00:00:00'), NaT, Timedelta('0 days, 00:00:00')]
```

@jreback as much as I would love `dti - tdi` and `tdi + tdi` to work, I feel there are some serious consistency concerns here. `-` is overloaded to mean two totally operations for indexes (subtract or set difference) depending on context.

Until now, I thought the rule was "do set operations if both arguments are indexes; otherwise do arithmetic," which is already overly complex. With this change, it will depend on the types of the indexes. This is a bad precedent unless it is coupled with a broader shift/deprecation of the existing behavior (`-`/`+` for set difference/union) for all index types.

@shoyer prob is already have a special case going with `Int64Index(...) + 2` which is IMHO worse.

@jreback Added a bunch of comments.

The main thing I am not fully sure about is the behaviour of the component attributes:

```
In [181]: pytd = dt.timedelta(days=1, seconds=15)
In [182]: td = pd.Timedelta(days=1, seconds=15)

In [183]: pytd.seconds
Out[183]: 15

In [184]: pytd.total_seconds()
Out[184]: 86415.0

In [185]: td.seconds
Out[185]: 86415

In [186]: td.total_seconds()
Out[186]: 86415.0
```

This is rather different behaviour, for something is said to behave similarly. (But I also think the pandas behaviour is more useful, as you have the components property to get that) 

Also, the repr `,` issue should still be fixes:

```
In [191]: pd.TimedeltaIndex(['1 days, 01:00:00', '1 days, 02:00:00'])
Out[191]: 
<class 'pandas.tseries.tdi.TimedeltaIndex'>
[1 days, 01:00:00, 1 days, 02:00:00]
Length: 2, Freq: None
```

Idea: let the `components` property return something like a named tuple?

So that you could do 

```
In [196]: td.components
Out[196]: (1.0, 2L, 5L, 0L, 0L, 0L, 0L, 0L)

In [197]: td.components.hours
5
```

like you can do with the TimedeltaIndex as this gives named columns:

```
In [198]: tdi = pd.TimedeltaIndex([td])

In [199]: tdi.components().hours
Out[199]: 
0    5
Name: hours, dtype: int64
```

@jorisvandenbossche good comments. I will respond (mostly agreeing).

I mistakenly that that `timedelta(days=1,seconds=1).seconds` returned 86401, rather than `.total_seconds`.

so this IS a change then (but I agree) its more 'natural' to get the full frequency conversion. so

```
td = Timedelta(1 days 00:00:01)

td.seconds -> 86401
td.days -> 1.001? (or 1 days, e.g. rounded?)
td.components.seconds = 1
td.componentd.days = 1
```

FYI, I have `components()` as a method in `TimedateIndex` as maybe need ags in the future (but I think will change back). And mayking a named tuple is easy (it will actually be a class I think as its created in cython, but same idea).

so this is an 'inconsistency' now:

```
td / np.timedelta(1, 'D') -> 1.00012
td.astype('timedelta64[D]') -> 1
```

so should td.days should return the float?

The new repr

```
In [1]: pd.timedelta_range('1 days 00:00:01',periods=3,freq='s')
Out[1]: 
<class 'pandas.tseries.tdi.TimedeltaIndex'>
['1 days, 00:00:01', ..., '1 days, 00:00:03']
Length: 3, Freq: <Second>
```

![time deltas pandas 0 14 1 dev-0ba604c documentation](https://cloud.githubusercontent.com/assets/953992/4224306/fd6e5dc6-3923-11e4-8bc4-8696274d58a9.png)

though the timeseries.rst was getting a bit long

@jorisvandenbossche let me know of any updates (and thoughts on `.days` giving the rounded result back)

this will now close #8190
(covers quite a bit of abbreviations); might be a tad slow though.

```
In [5]:    Timedelta('1 days 00:00:00')
Out[5]: Timedelta('1 days, 00:00:00')

In [6]:    Timedelta('1 days 2 hours')
Out[6]: Timedelta('1 days, 02:00:00')

In [7]:    Timedelta('-1 days 2 min 3us')
Out[7]: Timedelta('-1 days, 00:02:00.000003')
```

Nice docs!

For the components/attributes of Timedelta(Index), I have been thinking a bit more about it, and maybe it _could_ also be logical to return the components as the direct attributes (so like datetime.timedelta, and not the total values).

For example, if `td.hours` returns the component, and not the total, you can eg index with that, like "I want to get all rows at hour 0":

```
In [255]: tdi = pd.timedelta_range('00:00:00', freq='H', periods=48)

In [256]: tdi
Out[256]: 
<class 'pandas.tseries.tdi.TimedeltaIndex'>
[0 days, 00:00:00, ..., 1 days, 23:00:00]
Length: 48, Freq: <Hour>
```

And then be able to do `tdi[tdi.hours == 0]`. It seems more natural to need this, than the total hours in such a case. 

Further, it is maybe more 'logical' for an attribute, as the Timedelta consists of these components (also with a Timestamp, you don't get the total hours (which is not really defined, but OK), but only that part of the representation that is hours). The total hours is more a conversion, and would be more adequately expressed like something as `as_hours()` (not proposing this name).

Note however that this does not solve the incompatibility with datetime.timedelta (as they have fewer components, and add up the hours and minutes in the seconds).

The question is, if the attributes would return the components, how would the total ones be accessed easily. But you can already do something like this with `td.astype('timedelta[h]')` or `td / np.timedelta64(1, 'h')`

Just some thoughts, not really sure what the best thing to do is here.

ok if I change back the components (e.g. `Timedelta('1 days 2 hours').hours` -> 2)

maybe instead have a `.totals`? -> (e.g. to return the frequency converted for all?)

e.g. `1 days`, `26 hours` (maybe as floats)?

`totals` is also a possibility (although `components` seemed more natural to do it in that way (the tuple or dataframe with columns) as they add up to the original Timedelta)

Some other food for thought:
- `Timestamp.as_numeric(unit='h')` (or resolution keyword) (idea comes from R difftime)
- mxDateTime uses the difference between the singular and plural version -> hour / hours, second / seconds, ... for component/totals

What do others think about this? @cpcloud @hayd @TomAugspurger @shoyer 
So basically the question is: what should the `days/hours/minutes/seconds/...` attributes on Timedelta(Index) return? The components (all components add up to the Timedelta) or the totals (eg total hours is timedelta as integer expressed in hours, rounded or not):
- `Timedelta('1 days 2 hours').hours` -> 2 or 26 ?
- and if above returns 2, what is a nice API to retrieve the total 26?

@jreback Two other things: 
- `astype('timedelta64[h]')` does not work on TimedeltaIndex as it does on a timedelta series (to do conversions as is explained in the docs). Should that?
- the `round` method of Timestamp would maybe also be interesting for TimedeltaIndex (and TimedeltaProperties)? -> although this is maybe a broader issue: support for datetime/timedelta in Series.round

@jorisvandenbossche I am going to make (almost done)

`.days` for both `Timedelta/TimedeltaIndex` return the days field. Makes most sense.

so no 'totals' for now. Easy enough by simply doing `tdi / np.timedelta64(1,'d')` or `tdi.astype('timedelta64[D]')`

maybe make an `as_freq(freq)` to return a Float/Int result?

![time deltas pandas 0 14 1 dev-0ba604c documentation](https://cloud.githubusercontent.com/assets/953992/4227862/8a9c00de-394e-11e4-811f-e551b6513b94.png)