Repository: pandas-dev/pandas

Issue:
Partial indexing only valid for ordered time series
This was mentioned on [StackOverflow](http://stackoverflow.com/questions/13737992/indexing-timeseries-by-date-string), I thought I ought to post it here. I'm not sure whether or not this is a bug:

You can select by a date string in an ordered tseries, but not in an ordered one:

```
import pandas as pd
from numpy.random import randn
from random import shuffle
rng = pd.date_range(start='2011-01-01', end='2011-12-31')
rng2 = list(rng)
shuffle(rng2)
```

```
ts = pd.Series(randn(len(rng)), index=rng)
ts2 = pd.Series(randn(len(rng)), index=rng2)
```

```
ts.index
<class 'pandas.tseries.index.DatetimeIndex'>
[2011-01-01 00:00:00, ..., 2011-12-31 00:00:00]
Length: 365, Freq: D, Timezone: None

ts['2011-01-01']
# -1.1454418070543406
```

```
ts2.index
<class 'pandas.tseries.index.DatetimeIndex'>
[2011-04-16 00:00:00, ..., 2011-03-10 00:00:00]
Length: 365, Freq: None, Timezone: None

ts2['2011-01-01'] # same for ts2.ix['2011-01-01']
```

```
TimeSeriesError                           Traceback (most recent call last)
<ipython-input-112-051e81424a0d> in <module>()
----> 1 ts2['2011-01-01']

/Library/Python/2.7/site-packages/pandas-0.9.0-py2.7-macosx-10.7-intel.egg/pandas/core/series.pyc in __getitem__(self, key)
    462     def __getitem__(self, key):
    463         try:
--> 464             return self.index.get_value(self, key)
    465         except InvalidIndexError:
    466             pass

/Library/Python/2.7/site-packages/pandas-0.9.0-py2.7-macosx-10.7-intel.egg/pandas/tseries/index.pyc in get_value(self, series, key)
   1015 
   1016             try:
-> 1017                 loc = self._get_string_slice(key)
   1018                 return series[loc]
   1019             except (TypeError, ValueError, KeyError):

/Library/Python/2.7/site-packages/pandas-0.9.0-py2.7-macosx-10.7-intel.egg/pandas/tseries/index.pyc in _get_string_slice(self, key)
   1062         asdt, parsed, reso = parse_time_string(key, freq)
   1063         key = asdt
-> 1064         loc = self._partial_date_slice(reso, parsed)
   1065         return loc
   1066 

/Library/Python/2.7/site-packages/pandas-0.9.0-py2.7-macosx-10.7-intel.egg/pandas/tseries/index.pyc in _partial_date_slice(self, reso, parsed)
    977     def _partial_date_slice(self, reso, parsed):
    978         if not self.is_monotonic:
--> 979             raise TimeSeriesError('Partial indexing only valid for ordered time'
    980                                   ' series')
    981 

TimeSeriesError: Partial indexing only valid for ordered time series
```



Discussion:
Now I think about it: it's not clear how to pick a Timestamp from a date unless they are ordered! Which explains this behaviour...

In the ordered case the first result which satisfies the partial date is selected.

In the unordered case returning sometimes returning a series would be confusing, and picking a "closest" wouldn't really make sense.

I respectfully disagree with aspects of this behavior as it stands.  For example: whenever a user specifies a range of dates (in one of the several ways possible to do this), they can expect (quite reasonably) to receive a list of entries from an ordered time series.  What's wrong with implementing this same behavior on an unordered series?  Right now, for an unordered series, you get an error if you try to specify a date range.  This can be fixed without impacting the question of which entry should be "first" .

This may not fly with you guys, but I'd even take it one step further.  If a user specifies a single date as an index, they should receive _all_ entries with the matching date.  It's better behavior, because its consistent: if you ask for a month, you get the whole month; a year, you get the whole year; what's so special about a day that the behavior should suddenly change?

@jbrdly that's a convincing argument (at least, you've convinced me).

In fact, this also seems inconsistent when you compare to the behaviour of Series:

```
In [1]: s = Series(np.arange(5), index=[1, 1, 1, 2, 3])

In [2]: s
Out[2]: 
1    0
1    1
1    2
2    3
3    4

In [3]: s.ix[1] # same as s[1]
Out[3]: 
1    0
1    1
1    2

In [4]: s.ix[2]
Out[4]: 3
```

I guess the error message should read "not yet supported". Would be nice to fix this in the future


Resolving patch:
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -1042,9 +1042,6 @@ def intersection(self, other):
             return self._view_like(left_chunk)
 
     def _partial_date_slice(self, reso, parsed):
-        if not self.is_monotonic:
-            raise TimeSeriesError('Partial indexing only valid for ordered '
-                                  'time series.')
 
         if reso == 'year':
             t1 = Timestamp(datetime(parsed.year, 1, 1), tz=self.tz)
@@ -1079,11 +1076,19 @@ def _partial_date_slice(self, reso, parsed):
                                      tz=self.tz).value - 1)
         else:
             raise KeyError
+        
 
         stamps = self.asi8
-        left = stamps.searchsorted(t1.value, side='left')
-        right = stamps.searchsorted(t2.value, side='right')
-        return slice(left, right)
+
+        if self.is_monotonic:
+
+            # a monotonic (sorted) series can be sliced
+            left = stamps.searchsorted(t1.value, side='left')
+            right = stamps.searchsorted(t2.value, side='right')
+            return slice(left, right)
+
+        # try to find a the dates
+        return ((stamps>=t1.value) & (stamps<=t2.value)).nonzero()[0]
 
     def _possibly_promote(self, other):
         if other.inferred_type == 'date':


Test patch:
diff --git a/pandas/tseries/tests/test_timeseries.py b/pandas/tseries/tests/test_timeseries.py
--- a/pandas/tseries/tests/test_timeseries.py
+++ b/pandas/tseries/tests/test_timeseries.py
@@ -18,6 +18,7 @@
 import pandas.core.datetools as datetools
 import pandas.tseries.offsets as offsets
 import pandas.tseries.frequencies as fmod
+from pandas.tseries.index import TimeSeriesError
 import pandas as pd
 
 from pandas.util.testing import assert_series_equal, assert_almost_equal
@@ -168,6 +169,32 @@ def test_indexing_over_size_cutoff(self):
         finally:
             _index._SIZE_CUTOFF = old_cutoff
 
+    def test_indexing_unordered(self):
+
+        # GH 2437
+        from pandas import concat
+        rng = date_range(start='2011-01-01', end='2011-01-15')
+        ts  = Series(randn(len(rng)), index=rng)
+        ts2 = concat([ts[0:4],ts[-4:],ts[4:-4]])
+
+        for t in ts.index:
+            s = str(t)
+            expected = ts[t]
+            result = ts2[t]
+            self.assertTrue(expected == result)
+
+        result = ts2['2011'].sort_index()
+        expected = ts['2011']
+        assert_series_equal(result,expected)
+
+        # diff freq
+        rng = date_range(datetime(2005, 1, 1), periods=20, freq='M')
+        ts = Series(np.arange(len(rng)), index=rng)
+        ts = ts.take(np.random.permutation(20))
+
+        result = ts['2005']
+        for t in result.index:
+            self.assertTrue(t.year == 2005)
 
 def assert_range_equal(left, right):
     assert(left.equals(right))
@@ -2017,13 +2044,6 @@ def test_partial_slice_minutely(self):
         self.assert_(s['2005-1-1 23:59:00'] == s.ix[0])
         self.assertRaises(Exception, s.__getitem__, '2004-12-31 00:00:00')
 
-    def test_partial_not_monotonic(self):
-        rng = date_range(datetime(2005, 1, 1), periods=20, freq='M')
-        ts = Series(np.arange(len(rng)), index=rng)
-        ts = ts.take(np.random.permutation(20))
-
-        self.assertRaises(Exception, ts.__getitem__, '2005')
-
     def test_date_range_normalize(self):
         snap = datetime.today()
         n = 50