Repository: pandas-dev/pandas

Issue:
BUG: DateOffset weekday around DST produces unexpected results
I want next Monday:

```
def next_week_start(t):
    return t + pd.DateOffset(weekday=0)
```

But for DST (fall back is Sunday 2013-11-03):

```
 >> next_week_start(pd.Timestamp("2013-10-30", tz="US/Pacific"))
 Timestamp('2013-11-03 23:00:00-0800', tz='US/Pacific')
```

which is Sunday at 11pm, not Monday.

BUG: DateOffset weekday around DST produces unexpected results
I want next Monday:

```
def next_week_start(t):
    return t + pd.DateOffset(weekday=0)
```

But for DST (fall back is Sunday 2013-11-03):

```
 >> next_week_start(pd.Timestamp("2013-10-30", tz="US/Pacific"))
 Timestamp('2013-11-03 23:00:00-0800', tz='US/Pacific')
```

which is Sunday at 11pm, not Monday.



Discussion:
cc @rockg

can you take a look?

Testing:

```
def next_monday(t):
    return t + pd.DateOffset(weekday=0)

tz = "US/Pacific"
def test_week_start():    
    # a Tuesday - should return next week
    t = next_monday(pd.Timestamp("2013-01-01", tz=tz))
    assert(t.hour == 0 and t.dayofweek == 0 and t.day == 7)

    # a Monday - should return same day
    t = next_monday(pd.Timestamp("2013-01-07", tz=tz))
    assert(t.hour == 0 and t.dayofweek == 0 and t.day == 7)

     # fall back is Sun. Nov 03
    t = next_monday(pd.Timestamp("2013-10-30", tz=tz))
    assert(t.hour == 0 and t.dayofweek == 0)

    # spring fwd is Sun. Mar 10
    t = next_monday(pd.Timestamp("2013-03-08", tz=tz))
    assert(t.hour == 0  and t.dayofweek == 0)
```

see the last part of this section: http://pandas.pydata.org/pandas-docs/dev/timeseries.html#timeseries-timezone

there is a new kw, `infer_dst` that you can set in localize to 'fix' this problem; might need to be enabled here somehow

Also, there is a bug in dateutil (the default for DateOffset) which does not take into account tz when going from a dst to non-dst time. I don't know if we want to fix it there or account for it in DateOffset (the easiest fix is to remove the tzinfo, apply the offset, and then reapply the tzinfo). 

@adamgreenhall @jreback This is what I'm talking about.  The issue is that the tzinfo remains the same offset despite the dst offset actually changing.  So tzinfo needs to be removed, the offset applied, and then the result needs to be localized using the initial tzinfo.  This should probably be implemented in DateOffset for everything.

```
>>> ts = pd.Timestamp('2013-03-08', tz='US/Pacific')
>>> ts
Timestamp('2013-03-08 00:00:00-0800', tz='US/Pacific')
>>> tzinfo = ts.tzinfo
>>> ts = ts.replace(tzinfo=None)
>>> ts + pd.DateOffset(weekday=0)
Timestamp('2013-03-11 00:00:00', tz=None)
>>> ts = ts + pd.DateOffset(weekday=0)
>>> newts = tzinfo.localize(ts)
>>> pd.Timestamp(newts)
Timestamp('2013-03-11 00:00:00-0700', tz='US/Pacific')
```

@rockg is this true ONLY for a dst transition? (eg. should it always be done)?

@jreback Ideally only for offsets that span dst transitions, but I don't know how that will be known ahead of time (or at least in an efficient manner as you would basically need to do the operation twice). Localizing and unlocalizing are optimized for DatetimeIndex so I think doing it always makes sense in that case, but there will be some new plumbing.  This will only fix his last two test cases.

ok....will put you down for this! if you want to try for 0.13 ok....but would need by next week

Okay. I probably won't get to it seriously until early next week so probably not for the release, but shortly after.

gr8

cc @rockg

can you take a look?

Testing:

```
def next_monday(t):
    return t + pd.DateOffset(weekday=0)

tz = "US/Pacific"
def test_week_start():    
    # a Tuesday - should return next week
    t = next_monday(pd.Timestamp("2013-01-01", tz=tz))
    assert(t.hour == 0 and t.dayofweek == 0 and t.day == 7)

    # a Monday - should return same day
    t = next_monday(pd.Timestamp("2013-01-07", tz=tz))
    assert(t.hour == 0 and t.dayofweek == 0 and t.day == 7)

     # fall back is Sun. Nov 03
    t = next_monday(pd.Timestamp("2013-10-30", tz=tz))
    assert(t.hour == 0 and t.dayofweek == 0)

    # spring fwd is Sun. Mar 10
    t = next_monday(pd.Timestamp("2013-03-08", tz=tz))
    assert(t.hour == 0  and t.dayofweek == 0)
```

see the last part of this section: http://pandas.pydata.org/pandas-docs/dev/timeseries.html#timeseries-timezone

there is a new kw, `infer_dst` that you can set in localize to 'fix' this problem; might need to be enabled here somehow

Also, there is a bug in dateutil (the default for DateOffset) which does not take into account tz when going from a dst to non-dst time. I don't know if we want to fix it there or account for it in DateOffset (the easiest fix is to remove the tzinfo, apply the offset, and then reapply the tzinfo). 

@adamgreenhall @jreback This is what I'm talking about.  The issue is that the tzinfo remains the same offset despite the dst offset actually changing.  So tzinfo needs to be removed, the offset applied, and then the result needs to be localized using the initial tzinfo.  This should probably be implemented in DateOffset for everything.

```
>>> ts = pd.Timestamp('2013-03-08', tz='US/Pacific')
>>> ts
Timestamp('2013-03-08 00:00:00-0800', tz='US/Pacific')
>>> tzinfo = ts.tzinfo
>>> ts = ts.replace(tzinfo=None)
>>> ts + pd.DateOffset(weekday=0)
Timestamp('2013-03-11 00:00:00', tz=None)
>>> ts = ts + pd.DateOffset(weekday=0)
>>> newts = tzinfo.localize(ts)
>>> pd.Timestamp(newts)
Timestamp('2013-03-11 00:00:00-0700', tz='US/Pacific')
```

@rockg is this true ONLY for a dst transition? (eg. should it always be done)?

@jreback Ideally only for offsets that span dst transitions, but I don't know how that will be known ahead of time (or at least in an efficient manner as you would basically need to do the operation twice). Localizing and unlocalizing are optimized for DatetimeIndex so I think doing it always makes sense in that case, but there will be some new plumbing.  This will only fix his last two test cases.

ok....will put you down for this! if you want to try for 0.13 ok....but would need by next week

Okay. I probably won't get to it seriously until early next week so probably not for the release, but shortly after.

gr8


Resolving patch:
diff --git a/pandas/tseries/offsets.py b/pandas/tseries/offsets.py
--- a/pandas/tseries/offsets.py
+++ b/pandas/tseries/offsets.py
@@ -152,6 +152,12 @@ def __add__(date):
     """
     _cacheable = False
     _normalize_cache = True
+    _kwds_use_relativedelta = (
+        'years', 'months', 'weeks', 'days',
+        'year', 'month', 'week', 'day', 'weekday',
+        'hour', 'minute', 'second', 'microsecond'
+        )
+    _use_relativedelta = False
 
     # default for prior pickles
     normalize = False
@@ -160,21 +166,52 @@ def __init__(self, n=1, normalize=False, **kwds):
         self.n = int(n)
         self.normalize = normalize
         self.kwds = kwds
-        if len(kwds) > 0:
-            self._offset = relativedelta(**kwds)
+        self._offset, self._use_relativedelta = self._determine_offset()
+
+    def _determine_offset(self):
+        # timedelta is used for sub-daily plural offsets and all singular offsets
+        # relativedelta is used for plural offsets of daily length or more
+        # nanosecond(s) are handled by apply_wraps
+        kwds_no_nanos = dict(
+            (k, v) for k, v in self.kwds.items()
+            if k not in ('nanosecond', 'nanoseconds')
+            )
+        use_relativedelta = False
+
+        if len(kwds_no_nanos) > 0:
+            if any(k in self._kwds_use_relativedelta for k in kwds_no_nanos):
+                use_relativedelta = True
+                offset = relativedelta(**kwds_no_nanos)
+            else:
+                # sub-daily offset - use timedelta (tz-aware)
+                offset = timedelta(**kwds_no_nanos)
         else:
-            self._offset = timedelta(1)
+            offset = timedelta(1)
+        return offset, use_relativedelta
 
     @apply_wraps
     def apply(self, other):
+        if self._use_relativedelta:
+            other = as_datetime(other)
+
         if len(self.kwds) > 0:
+            tzinfo = getattr(other, 'tzinfo', None)
+            if tzinfo is not None and self._use_relativedelta:
+                # perform calculation in UTC
+                other = other.replace(tzinfo=None)
+
             if self.n > 0:
                 for i in range(self.n):
                     other = other + self._offset
             else:
                 for i in range(-self.n):
                     other = other - self._offset
-            return other
+
+            if tzinfo is not None and self._use_relativedelta:
+                # bring tz back from UTC calculation
+                other = tslib._localize_pydatetime(other, tzinfo)
+
+            return as_timestamp(other)
         else:
             return other + timedelta(self.n)
 


Test patch:
diff --git a/pandas/tseries/tests/test_offsets.py b/pandas/tseries/tests/test_offsets.py
--- a/pandas/tseries/tests/test_offsets.py
+++ b/pandas/tseries/tests/test_offsets.py
@@ -3104,6 +3104,134 @@ def test_str_for_named_is_name(self):
             self.assertEqual(str(offset), name)
 
 
+def get_utc_offset_hours(ts):
+    # take a Timestamp and compute total hours of utc offset
+    o = ts.utcoffset()
+    return (o.days * 24 * 3600 + o.seconds) / 3600.0
+
+
+class TestDST(tm.TestCase):
+    """
+    test DateOffset additions over Daylight Savings Time
+    """
+    # one microsecond before the DST transition
+    ts_pre_fallback = "2013-11-03 01:59:59.999999"
+    ts_pre_springfwd = "2013-03-10 01:59:59.999999"
+
+    # test both basic names and dateutil timezones
+    timezone_utc_offsets = {
+        'US/Eastern': dict(
+            utc_offset_daylight=-4,
+            utc_offset_standard=-5,
+            ),
+        'dateutil/US/Pacific': dict(
+            utc_offset_daylight=-7,
+            utc_offset_standard=-8,
+            )
+        }
+    valid_date_offsets_singular = [
+        'weekday', 'day', 'hour', 'minute', 'second', 'microsecond'
+        ]
+    valid_date_offsets_plural = [
+        'weeks', 'days',
+        'hours', 'minutes', 'seconds',
+        'milliseconds', 'microseconds'
+        ]
+
+    def _test_all_offsets(self, n, **kwds):
+        valid_offsets = self.valid_date_offsets_plural if n > 1 \
+            else self.valid_date_offsets_singular
+
+        for name in valid_offsets:
+            self._test_offset(offset_name=name, offset_n=n, **kwds)
+
+    def _test_offset(self, offset_name, offset_n, tstart, expected_utc_offset):
+        offset = DateOffset(**{offset_name: offset_n})
+        t = tstart + offset
+        if expected_utc_offset is not None:
+            self.assertTrue(get_utc_offset_hours(t) == expected_utc_offset)
+
+        if offset_name == 'weeks':
+            # dates should match
+            self.assertTrue(
+                t.date() ==
+                timedelta(days=7 * offset.kwds['weeks']) + tstart.date()
+                )
+            # expect the same day of week, hour of day, minute, second, ...
+            self.assertTrue(
+                t.dayofweek == tstart.dayofweek and
+                t.hour == tstart.hour and
+                t.minute == tstart.minute and
+                t.second == tstart.second
+                )
+        elif offset_name == 'days':
+            # dates should match
+            self.assertTrue(timedelta(offset.kwds['days']) + tstart.date() == t.date())
+            # expect the same hour of day, minute, second, ...
+            self.assertTrue(
+                t.hour == tstart.hour and
+                t.minute == tstart.minute and
+                t.second == tstart.second
+                )
+        elif offset_name in self.valid_date_offsets_singular:
+            # expect the signular offset value to match between tstart and t
+            datepart_offset = getattr(t, offset_name if offset_name != 'weekday' else 'dayofweek')
+            self.assertTrue(datepart_offset == offset.kwds[offset_name])
+        else:
+            # the offset should be the same as if it was done in UTC
+            self.assertTrue(
+                t == (tstart.tz_convert('UTC') + offset).tz_convert('US/Pacific')
+                )
+
+    def _make_timestamp(self, string, hrs_offset, tz):
+        offset_string = '{hrs:02d}00'.format(hrs=hrs_offset) if hrs_offset >= 0 else \
+            '-{hrs:02d}00'.format(hrs=-1 * hrs_offset)
+        return Timestamp(string + offset_string).tz_convert(tz)
+
+    def test_fallback_plural(self):
+        """test moving from daylight savings to standard time"""
+        for tz, utc_offsets in self.timezone_utc_offsets.items():
+            hrs_pre = utc_offsets['utc_offset_daylight']
+            hrs_post = utc_offsets['utc_offset_standard']
+            self._test_all_offsets(
+                n=3,
+                tstart=self._make_timestamp(self.ts_pre_fallback, hrs_pre, tz),
+                expected_utc_offset=hrs_post
+                )
+
+    def test_springforward_plural(self):
+        """test moving from standard to daylight savings"""
+        for tz, utc_offsets in self.timezone_utc_offsets.items():
+            hrs_pre = utc_offsets['utc_offset_standard']
+            hrs_post = utc_offsets['utc_offset_daylight']
+            self._test_all_offsets(
+                n=3,
+                tstart=self._make_timestamp(self.ts_pre_springfwd, hrs_pre, tz),
+                expected_utc_offset=hrs_post
+                )
+
+    def test_fallback_singular(self):
+        # in the case of signular offsets, we dont neccesarily know which utc offset
+        # the new Timestamp will wind up in (the tz for 1 month may be different from 1 second)
+        # so we don't specify an expected_utc_offset
+        for tz, utc_offsets in self.timezone_utc_offsets.items():
+            hrs_pre = utc_offsets['utc_offset_standard']
+            self._test_all_offsets(
+                n=1,
+                tstart=self._make_timestamp(self.ts_pre_fallback, hrs_pre, tz),
+                expected_utc_offset=None
+                )
+
+    def test_springforward_singular(self):
+        for tz, utc_offsets in self.timezone_utc_offsets.items():
+            hrs_pre = utc_offsets['utc_offset_standard']
+            self._test_all_offsets(
+                n=1,
+                tstart=self._make_timestamp(self.ts_pre_springfwd, hrs_pre, tz),
+                expected_utc_offset=None
+                )
+
+
 if __name__ == '__main__':
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],
                    exit=False)