Repository: pandas-dev/pandas

Issue:
tz_localize should support is_dst input array
When storing datetimes with timezone information in mysql I split out the is_dst flag into a separate column.  Then when reconstructing the Timestamps I am either forced to iterate through each row and call pytz.timezone.localize on every Timestamp which is very slow or do some magic with localizing what I can and then manually dealing with the fall transition time (note that infer_dst won't work because there could be many rows that have transitions in them).  I would much rather create the DatetimeIndex from the column of dates and then call tz_localize with the is_dst column.  This would then appropriately set the offset.

```
di = DatetimeIndex(frame['DateColumn'])
di = di.tz_localize(TimeZone, is_dst_flat=frame['IsDstColumn'])
```

Thoughts?



Discussion:
we already support `infer_dst`? how does that interact? 

`infer_dst` only works when there are only two dates for the fall dst transition time.  If there are more than two, then it cannot infer simply based on order.  In that case, something else needs to be done to explicitly tell what times are dst times and what times are not.

ok, makes sense. would making `infer_dst='force'` make sense? or you simply want another keyword? (`is_dst=None`). I suppose its an error if `is_dst` is not None and `infer_dst=True` ?

I would like an is_dst keyword and this also mirrors pytz's localize which takes is_dst as a parameter.  I agree that it is an error if is_dst is not None and infer_dst=True (they are mutually exclusive).  I'll take a stab add adding it to localize.  Should be straightforward if I recall.

ok, sounds good

I suspect that `is_dst` is too entrenched for me to have a chance of painting this bikeshed my color, but I am against `is_dst` as a name because not all time zone discontinuities are DST boundaries.  I think in this case, it is probably conceptually clearer to split off DST information as a `timedelta64` offset rather than an `is_dst` flag, which among other things makes converting to UTC trivial.

That said, since the problem of converting local time and time zone to UTC (essentially inverting the time zone, if we view it as a map from UTC to local time) is simply one of selecting the correct UTC time from a list of zero, one or two times, and since in the case of an ambiguous time at a DST boundary the DST time must always come first, I propose that if we use `is_dst` as a parameter, it should mean "select the _first_ (as opposed to the  _last_) UTC time that maps to this local time, regardless of DST", and `is_dst=True` should be the default.

I also propose that we not put any effort into supporting the case where there are three or more UTC times corresponding to a given local time, because this case requires pathological government action and I don't know of any instance where it has happened.

Your second paragraph is not exactly correct and why this is needed.  If you only had two ambiguous times, then the `infer_dst` argument is all that is needed and works like you say.  However, when there are more times it becomes a problem.  For example, when reading database records where there are multiple items that have ambiguous times associated with them, rather than discretizing among those items and inferring the right dst/non-dst time from for each item it will be much faster and easier to already use a pre-computed column when the data was input in my particular case.  Here's a simple example of data I'm looking at.

Item,Time,IS_DST
1,2014-11-02 00:00,1
1,2014-11-02 01:00,1
1,2014-11-02 01:00,0
2,2014-11-02 00:00,1
2,2014-11-02 01:00,1
2,2014-11-02 01:00,0

What I'm proposing is that (if you are in `America/Chicago`) you will have this:

```
Item,Time,Offset
1,2014-11-02 00:00,-05:00:00
1,2014-11-02 01:00,-05:00:00
1,2014-11-02 01:00,-06:00:00
2,2014-11-02 00:00,-05:00:00
2,2014-11-02 01:00,-05:00:00
2,2014-11-02 01:00,-06:00:00
```

Which is just as good.

Yes, I know that's your first point...I was more addressing your subsequent points.

I agree that it's just as good and contains the same information.  My data is structured with bools and hence I want to make that work (it was only a couple lines of code to make this work).  I don't know offhand how to make the offsets in the underlying tz_localize code.  I don't see any harm in making both work if possible.

To @ischwabacher's point, I'm not against adding `is_dst=` to `tz_localize()`. But I also agree that a better solution to @rockg's problem is to store timestamps in UTC and timezone separately in two columns in a database table.

I always store timestamp in UTC in database to avoid ambiguity, unless the database has explicit support for `timestamp with time zone` like postgresql.

While storing data in UTC is unambiguous, it makes it very difficult when people are working with data in the database.  With my work, having the data stored in the actual time zone has saved many errors with users not having to manually do the dst offset in their heads...it's so much easier and not worth the hassle for only 2 out of 8760 hours.


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -23,7 +23,7 @@
                                 _maybe_box_datetimelike, ABCSeries,
                                 SettingWithCopyError, SettingWithCopyWarning)
 import pandas.core.nanops as nanops
-from pandas.util.decorators import Appender, Substitution
+from pandas.util.decorators import Appender, Substitution, deprecate_kwarg
 from pandas.core import config
 
 # goal is to be able to define the docs close to function, while still being
@@ -3558,8 +3558,11 @@ def _tz_convert(ax, tz):
         result = self._constructor(self._data, copy=copy)
         result.set_axis(axis,ax)
         return result.__finalize__(self)
-
-    def tz_localize(self, tz, axis=0, level=None, copy=True, infer_dst=False):
+    
+    @deprecate_kwarg(old_arg_name='infer_dst', new_arg_name='ambiguous',
+                     mapping={True: 'infer', False: 'raise'})
+    def tz_localize(self, tz, axis=0, level=None, copy=True, 
+                    ambiguous='raise'):
         """
         Localize tz-naive TimeSeries to target time zone
 
@@ -3572,16 +3575,22 @@ def tz_localize(self, tz, axis=0, level=None, copy=True, infer_dst=False):
             must be None
         copy : boolean, default True
             Also make a copy of the underlying data
-        infer_dst : boolean, default False
-            Attempt to infer fall dst-transition times based on order
-
+        ambiguous : 'infer', bool-ndarray, 'NaT', default 'raise'
+            - 'infer' will attempt to infer fall dst-transition hours based on order
+            - bool-ndarray where True signifies a DST time, False designates
+              a non-DST time (note that this flag is only applicable for ambiguous times)
+            - 'NaT' will return NaT where there are ambiguous times
+            - 'raise' will raise an AmbiguousTimeError if there are ambiguous times
+        infer_dst : boolean, default False (DEPRECATED)
+            Attempt to infer fall dst-transition hours based on order
+            
         Returns
         -------
         """
         axis = self._get_axis_number(axis)
         ax = self._get_axis(axis)
 
-        def _tz_localize(ax, tz, infer_dst):
+        def _tz_localize(ax, tz, ambiguous):
             if not hasattr(ax, 'tz_localize'):
                 if len(ax) > 0:
                     ax_name = self._get_axis_name(axis)
@@ -3590,19 +3599,19 @@ def _tz_localize(ax, tz, infer_dst):
                 else:
                     ax = DatetimeIndex([],tz=tz)
             else:
-                ax = ax.tz_localize(tz, infer_dst=infer_dst)
+                ax = ax.tz_localize(tz, ambiguous=ambiguous)
             return ax
 
         # if a level is given it must be a MultiIndex level or
         # equivalent to the axis name
         if isinstance(ax, MultiIndex):
             level = ax._get_level_number(level)
-            new_level = _tz_localize(ax.levels[level], tz, infer_dst)
+            new_level = _tz_localize(ax.levels[level], tz, ambiguous)
             ax = ax.set_levels(new_level, level=level)
         else:
             if level not in (None, 0, ax.name):
                 raise ValueError("The level {0} is not valid".format(level))
-            ax =  _tz_localize(ax, tz, infer_dst)
+            ax =  _tz_localize(ax, tz, ambiguous)
 
         result = self._constructor(self._data, copy=copy)
         result.set_axis(axis,ax)
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -6,6 +6,8 @@
 
 import numpy as np
 
+import warnings
+
 from pandas.core.common import (_NS_DTYPE, _INT64_DTYPE,
                                 _values_from_object, _maybe_box,
                                 ABCSeries)
@@ -18,7 +20,7 @@
 from pandas.core.base import DatetimeIndexOpsMixin
 from pandas.tseries.offsets import DateOffset, generate_range, Tick, CDay
 from pandas.tseries.tools import parse_time_string, normalize_date
-from pandas.util.decorators import cache_readonly
+from pandas.util.decorators import cache_readonly, deprecate_kwarg
 import pandas.core.common as com
 import pandas.tseries.offsets as offsets
 import pandas.tseries.tools as tools
@@ -145,6 +147,15 @@ class DatetimeIndex(DatetimeIndexOpsMixin, Int64Index):
     closed : string or None, default None
         Make the interval closed with respect to the given frequency to
         the 'left', 'right', or both sides (None)
+    tz : pytz.timezone or dateutil.tz.tzfile
+    ambiguous : 'infer', bool-ndarray, 'NaT', default 'raise'
+        - 'infer' will attempt to infer fall dst-transition hours based on order
+        - bool-ndarray where True signifies a DST time, False signifies
+          a non-DST time (note that this flag is only applicable for ambiguous times)
+        - 'NaT' will return NaT where there are ambiguous times
+        - 'raise' will raise an AmbiguousTimeError if there are ambiguous times
+    infer_dst : boolean, default False (DEPRECATED)
+        Attempt to infer fall dst-transition hours based on order
     name : object
         Name to be stored in the index
     """
@@ -180,15 +191,17 @@ class DatetimeIndex(DatetimeIndexOpsMixin, Int64Index):
                          'is_quarter_start','is_quarter_end','is_year_start','is_year_end']
     _is_numeric_dtype = False
 
+
+    @deprecate_kwarg(old_arg_name='infer_dst', new_arg_name='ambiguous',
+                     mapping={True: 'infer', False: 'raise'})
     def __new__(cls, data=None,
                 freq=None, start=None, end=None, periods=None,
                 copy=False, name=None, tz=None,
                 verify_integrity=True, normalize=False,
-                closed=None, **kwargs):
+                closed=None, ambiguous='raise', **kwargs):
 
         dayfirst = kwargs.pop('dayfirst', None)
         yearfirst = kwargs.pop('yearfirst', None)
-        infer_dst = kwargs.pop('infer_dst', False)
 
         freq_infer = False
         if not isinstance(freq, DateOffset):
@@ -214,7 +227,7 @@ def __new__(cls, data=None,
         if data is None:
             return cls._generate(start, end, periods, name, freq,
                                  tz=tz, normalize=normalize, closed=closed,
-                                 infer_dst=infer_dst)
+                                 ambiguous=ambiguous)
 
         if not isinstance(data, (np.ndarray, Index, ABCSeries)):
             if np.isscalar(data):
@@ -240,7 +253,7 @@ def __new__(cls, data=None,
                         data.name = name
 
                     if tz is not None:
-                        return data.tz_localize(tz, infer_dst=infer_dst)
+                        return data.tz_localize(tz, ambiguous=ambiguous)
 
                     return data
 
@@ -309,7 +322,7 @@ def __new__(cls, data=None,
                     # Convert tz-naive to UTC
                     ints = subarr.view('i8')
                     subarr = tslib.tz_localize_to_utc(ints, tz,
-                                                      infer_dst=infer_dst)
+                                                      ambiguous=ambiguous)
 
                 subarr = subarr.view(_NS_DTYPE)
 
@@ -333,7 +346,7 @@ def __new__(cls, data=None,
 
     @classmethod
     def _generate(cls, start, end, periods, name, offset,
-                  tz=None, normalize=False, infer_dst=False, closed=None):
+                  tz=None, normalize=False, ambiguous='raise', closed=None):
         if com._count_not_none(start, end, periods) != 2:
             raise ValueError('Must specify two of start, end, or periods')
 
@@ -447,7 +460,7 @@ def _generate(cls, start, end, periods, name, offset,
 
             if tz is not None and getattr(index, 'tz', None) is None:
                 index = tslib.tz_localize_to_utc(com._ensure_int64(index), tz,
-                                                 infer_dst=infer_dst)
+                                                 ambiguous=ambiguous)
                 index = index.view(_NS_DTYPE)
 
         index = cls._simple_new(index, name=name, freq=offset, tz=tz)
@@ -1645,7 +1658,9 @@ def tz_convert(self, tz):
         # No conversion since timestamps are all UTC to begin with
         return self._shallow_copy(tz=tz)
 
-    def tz_localize(self, tz, infer_dst=False):
+    @deprecate_kwarg(old_arg_name='infer_dst', new_arg_name='ambiguous',
+                 mapping={True: 'infer', False: 'raise'})
+    def tz_localize(self, tz, ambiguous='raise'):
         """
         Localize tz-naive DatetimeIndex to given time zone (using pytz/dateutil),
         or remove timezone from tz-aware DatetimeIndex
@@ -1656,7 +1671,13 @@ def tz_localize(self, tz, infer_dst=False):
             Time zone for time. Corresponding timestamps would be converted to
             time zone of the TimeSeries.
             None will remove timezone holding local time.
-        infer_dst : boolean, default False
+        ambiguous : 'infer', bool-ndarray, 'NaT', default 'raise'
+            - 'infer' will attempt to infer fall dst-transition hours based on order
+            - bool-ndarray where True signifies a DST time, False signifies
+              a non-DST time (note that this flag is only applicable for ambiguous times)
+            - 'NaT' will return NaT where there are ambiguous times
+            - 'raise' will raise an AmbiguousTimeError if there are ambiguous times
+        infer_dst : boolean, default False (DEPRECATED)
             Attempt to infer fall dst-transition hours based on order
 
         Returns
@@ -1671,7 +1692,9 @@ def tz_localize(self, tz, infer_dst=False):
         else:
             tz = tslib.maybe_get_tz(tz)
             # Convert to UTC
-            new_dates = tslib.tz_localize_to_utc(self.asi8, tz, infer_dst=infer_dst)
+            
+            new_dates = tslib.tz_localize_to_utc(self.asi8, tz, 
+                                                 ambiguous=ambiguous)
         new_dates = new_dates.view(_NS_DTYPE)
         return self._shallow_copy(new_dates, tz=tz)
 


Test patch:
diff --git a/pandas/tseries/tests/test_timezones.py b/pandas/tseries/tests/test_timezones.py
--- a/pandas/tseries/tests/test_timezones.py
+++ b/pandas/tseries/tests/test_timezones.py
@@ -423,31 +423,98 @@ def test_with_tz_ambiguous_times(self):
         dr = date_range(datetime(2011, 3, 13), periods=48,
                         freq=datetools.Minute(30), tz=pytz.utc)
 
-    def test_infer_dst(self):
+    def test_ambiguous_infer(self):
         # November 6, 2011, fall back, repeat 2 AM hour
         # With no repeated hours, we cannot infer the transition
         tz = self.tz('US/Eastern')
         dr = date_range(datetime(2011, 11, 6, 0), periods=5,
                         freq=datetools.Hour())
-        self.assertRaises(pytz.AmbiguousTimeError, dr.tz_localize,
-                          tz, infer_dst=True)
+        self.assertRaises(pytz.AmbiguousTimeError, dr.tz_localize, tz)
 
         # With repeated hours, we can infer the transition
         dr = date_range(datetime(2011, 11, 6, 0), periods=5,
                         freq=datetools.Hour(), tz=tz)
-        di = DatetimeIndex(['11/06/2011 00:00', '11/06/2011 01:00',
-                            '11/06/2011 01:00', '11/06/2011 02:00',
-                            '11/06/2011 03:00'])
-        localized = di.tz_localize(tz, infer_dst=True)
+        times = ['11/06/2011 00:00', '11/06/2011 01:00',
+                 '11/06/2011 01:00', '11/06/2011 02:00',
+                 '11/06/2011 03:00']
+        di = DatetimeIndex(times)
+        localized = di.tz_localize(tz, ambiguous='infer')
         self.assert_numpy_array_equal(dr, localized)
-
+        localized_old = di.tz_localize(tz, infer_dst=True)
+        self.assert_numpy_array_equal(dr, localized_old)
+        self.assert_numpy_array_equal(dr, DatetimeIndex(times, tz=tz, ambiguous='infer'))
+        
         # When there is no dst transition, nothing special happens
         dr = date_range(datetime(2011, 6, 1, 0), periods=10,
                         freq=datetools.Hour())
         localized = dr.tz_localize(tz)
-        localized_infer = dr.tz_localize(tz, infer_dst=True)
+        localized_infer = dr.tz_localize(tz, ambiguous='infer')
         self.assert_numpy_array_equal(localized, localized_infer)
+        localized_infer_old = dr.tz_localize(tz, infer_dst=True)
+        self.assert_numpy_array_equal(localized, localized_infer_old)
+
+    def test_ambiguous_flags(self):
+        # November 6, 2011, fall back, repeat 2 AM hour
+        tz = self.tz('US/Eastern')
+
+        # Pass in flags to determine right dst transition
+        dr = date_range(datetime(2011, 11, 6, 0), periods=5,
+                        freq=datetools.Hour(), tz=tz)
+        times = ['11/06/2011 00:00', '11/06/2011 01:00',
+                 '11/06/2011 01:00', '11/06/2011 02:00',
+                 '11/06/2011 03:00']
+        
+        # Test tz_localize
+        di = DatetimeIndex(times)
+        is_dst = [1, 1, 0, 0, 0]
+        localized = di.tz_localize(tz, ambiguous=is_dst)
+        self.assert_numpy_array_equal(dr, localized)
+        self.assert_numpy_array_equal(dr, DatetimeIndex(times, tz=tz, ambiguous=is_dst))
+        
+        localized = di.tz_localize(tz, ambiguous=np.array(is_dst))
+        self.assert_numpy_array_equal(dr, localized)
+        
+        localized = di.tz_localize(tz, ambiguous=np.array(is_dst).astype('bool'))
+        self.assert_numpy_array_equal(dr, localized)
+        
+        # Test constructor
+        localized = DatetimeIndex(times, tz=tz, ambiguous=is_dst)
+        self.assert_numpy_array_equal(dr, localized)
+             
+        # Test duplicate times where infer_dst fails
+        times += times
+        di = DatetimeIndex(times)
+        
+        # When the sizes are incompatible, make sure error is raised
+        self.assertRaises(Exception, di.tz_localize, tz, ambiguous=is_dst)
+        
+        # When sizes are compatible and there are repeats ('infer' won't work)
+        is_dst = np.hstack((is_dst, is_dst))
+        localized = di.tz_localize(tz, ambiguous=is_dst)
+        dr = dr.append(dr)
+        self.assert_numpy_array_equal(dr, localized)
 
+        # When there is no dst transition, nothing special happens
+        dr = date_range(datetime(2011, 6, 1, 0), periods=10,
+                        freq=datetools.Hour())
+        is_dst = np.array([1] * 10)
+        localized = dr.tz_localize(tz)
+        localized_is_dst = dr.tz_localize(tz, ambiguous=is_dst)
+        self.assert_numpy_array_equal(localized, localized_is_dst)
+        
+    def test_ambiguous_nat(self):
+        tz = self.tz('US/Eastern')
+        times = ['11/06/2011 00:00', '11/06/2011 01:00',
+                 '11/06/2011 01:00', '11/06/2011 02:00',
+                 '11/06/2011 03:00']
+        di = DatetimeIndex(times)
+        localized = di.tz_localize(tz, ambiguous='NaT')
+        
+        times = ['11/06/2011 00:00', np.NaN,
+                 np.NaN, '11/06/2011 02:00',
+                 '11/06/2011 03:00']
+        di_test = DatetimeIndex(times, tz='US/Eastern')
+        self.assert_numpy_array_equal(di_test, localized)
 
     # test utility methods
     def test_infer_tz(self):
diff --git a/pandas/tseries/tests/test_tslib.py b/pandas/tseries/tests/test_tslib.py
--- a/pandas/tseries/tests/test_tslib.py
+++ b/pandas/tseries/tests/test_tslib.py
@@ -232,6 +232,17 @@ def test_tz(self):
         conv = local.tz_convert('US/Eastern')
         self.assertEqual(conv.nanosecond, 5)
         self.assertEqual(conv.hour, 19)
+        
+    def test_tz_localize_ambiguous(self):
+        
+        ts = Timestamp('2014-11-02 01:00')
+        ts_dst = ts.tz_localize('US/Eastern', ambiguous=True)
+        ts_no_dst = ts.tz_localize('US/Eastern', ambiguous=False)
+        
+        rng = date_range('2014-11-02', periods=3, freq='H', tz='US/Eastern')
+        self.assertEqual(rng[1], ts_dst)
+        self.assertEqual(rng[2], ts_no_dst)
+        self.assertRaises(ValueError, ts.tz_localize, 'US/Eastern', ambiguous='infer')
 
         # GH 8025
         with tm.assertRaisesRegexp(TypeError, 'Cannot localize tz-aware Timestamp, use '