Repository: pandas-dev/pandas

Issue:
ENH: Allow to reset tz from DatetimeIndex holding time representation
There looks no easy way to remove tz from tz-aware `DatetimeIndex` holding the timestamp represented. What required is inverse operation of `tz_localize`.

```
idx = pd.date_range('2014-01-01 09:00', '2014-01-01 20:00', freq='H', tz='Asia/Tokyo')
idx
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2014-01-01 09:00:00+09:00, ..., 2014-01-01 20:00:00+09:00]
# Length: 12, Freq: H, Timezone: Asia/Tokyo

# What I want to do is:
idx.tz_reset()
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2014-01-01 09:00:00, ..., 2014-01-01 20:00:00]
# Length: 12, Freq: H, Timezone: None

# This raises TypeError
idx.tz_localize(None)
# TypeError: Already tz-aware, use tz_convert to convert.

# This change the timestamp to UTC
idx.tz_convert(None)
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2014-01-01 00:00:00, ..., 2014-01-01 11:00:00]
# Length: 12, Freq: H, Timezone: None
```

In my case, there are some globally distributed data sources which holds local timezones. And want to merge them and analyze based on local subjective times (business hours, etc).



Discussion:
+1.  What if we support idx.tzinfo=None and that operates appropriately on the underlying data?

I agree with @rockg  (maybe `idx.tz = None` though).

I don't think this very common, so full function I don't think is necessary (though I guess can't hurt).

+1 : I suspect this is more common than imagined. :)

@nehalecky hahh!

ok, so method and setter? 

how do you do this now?

`DatetimeIndex(idx_with_tz.tz_convert('UTC'),tz=None)` ?

Initial idea may be generalized by adding `force` option to `tz_localize`.

```
idx = pd.date_range('2014-01-01 09:00', '2014-01-01 20:00', freq='H', tz='Asia/Tokyo')
idx
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2014-01-01 09:00:00+09:00, ..., 2014-01-01 20:00:00+09:00]
# Length: 12, Freq: H, Timezone: Asia/Tokyo

# TypeError by default
idx.tz_localize(None)
# TypeError: Already tz-aware, use tz_convert to convert.

idx.tz_localize(None, force=True)
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2014-01-01 09:00:00, ..., 2014-01-01 20:00:00]
# Length: 12, Freq: H, Timezone: None

# can be used to set different timezones with same representation
idx.tz_localize('US/Eastern', force=True)
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2014-01-01 09:00:00-05:00, ..., 2014-01-01 20:00:00-05:00]
# Length: 12, Freq: H, Timezone: US/Eastern
```

what about just allowing `tz_localize(None)` as a valid argument and/or (`tz_convert(None)`)

though this _may_ be more confusing

I think None is a great idea and I don't think it's too confusing.

should this work only on one if tz_convert/tz_localize?

side issue do we support setting directly of a tz? (which would convert/localize as appropriate)

I was going to say just tz_localize before but it probably does make more sense with tz_convert (converting a tz to a non-tz).   I think supporting both is okay.  I don't think we support directly setting tz except through tz_localize.  

ok, so:
- need  to figure out what cases the currently raised error is actually useful (e.g. wesm must have put it their for some reason - maybe just to remind the user)?
- raise `NotImplementedError` on directly setting `tz` if we are not going to allow it (I actually think its ok, maybe just a call to `tz_convert(tz)`

@jreback, thanks, and yeah, tz naive `DatetimeIndex`es allows for comparing time series that follow typical time-of-day driven processes that humans (and other phenomena such as weather) like to follow. I work with a lot of this type of data, so, it would be great to see pandas allow flexibility when performing comparisons across multi-tz datasets. :)

At first, I brute-forced the stripping of the underlying timestamp of tz info via `.apply`, then I got smart and saw that simply recreating (like you've suggested) was more performant. I've actually found this to be the case when wanting to modify other pandas objects as well.

+1 on the `.tz_convert(None)`, I actually think that is what I tried to call the first time I was attempting to do this.

Hmm, both `tz_localize` and `tz_convert` "converts" something, the actual behavior should be considered.
- `tz_localize` "converts" timezone holding time representation (holding subjective time and change absolute time).
- `tz_convert` "converts" timezone holding UTC time (holding absolute time and change subjective time).

The required operation is "holding subjective time and reset timezone", thus I feel `tz_localize(None)` is more natural. If we use `tz_convert(None)` for this operation, `tz_convert` has inconsistency which sometimes convert subjective time, otherwise convert absolute time.

@rockg ?

my2c:

I think we should do just `tz_localize`. Its an explicit user initiated operation, saying, hey I have an index/timestamp with a tz and I want to remove the tz. And its the inverse operation of `tz_localize(tz)` which takes a naive object and gives it a timezone

I'm fine with just `tz_localize`


Resolving patch:
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -1618,7 +1618,14 @@ def _view_like(self, ndarray):
 
     def tz_convert(self, tz):
         """
-        Convert DatetimeIndex from one time zone to another (using pytz/dateutil)
+        Convert tz-aware DatetimeIndex from one time zone to another (using pytz/dateutil)
+
+        Parameters
+        ----------
+        tz : string, pytz.timezone, dateutil.tz.tzfile or None
+            Time zone for time. Corresponding timestamps would be converted to
+            time zone of the TimeSeries.
+            None will remove timezone holding UTC time.
 
         Returns
         -------
@@ -1636,13 +1643,15 @@ def tz_convert(self, tz):
 
     def tz_localize(self, tz, infer_dst=False):
         """
-        Localize tz-naive DatetimeIndex to given time zone (using pytz/dateutil)
+        Localize tz-naive DatetimeIndex to given time zone (using pytz/dateutil),
+        or remove timezone from tz-aware DatetimeIndex
 
         Parameters
         ----------
-        tz : string or pytz.timezone or dateutil.tz.tzfile
+        tz : string, pytz.timezone, dateutil.tz.tzfile or None
             Time zone for time. Corresponding timestamps would be converted to
-            time zone of the TimeSeries
+            time zone of the TimeSeries.
+            None will remove timezone holding local time.
         infer_dst : boolean, default False
             Attempt to infer fall dst-transition hours based on order
 
@@ -1651,13 +1660,15 @@ def tz_localize(self, tz, infer_dst=False):
         localized : DatetimeIndex
         """
         if self.tz is not None:
-            raise TypeError("Already tz-aware, use tz_convert to convert.")
-        tz = tslib.maybe_get_tz(tz)
-
-        # Convert to UTC
-        new_dates = tslib.tz_localize_to_utc(self.asi8, tz, infer_dst=infer_dst)
+            if tz is None:
+                new_dates = tslib.tz_convert(self.asi8, 'UTC', self.tz)
+            else:
+                raise TypeError("Already tz-aware, use tz_convert to convert.")
+        else:
+            tz = tslib.maybe_get_tz(tz)
+            # Convert to UTC
+            new_dates = tslib.tz_localize_to_utc(self.asi8, tz, infer_dst=infer_dst)
         new_dates = new_dates.view(_NS_DTYPE)
-
         return self._simple_new(new_dates, self.name, self.offset, tz)
 
     def indexer_at_time(self, time, asof=False):


Test patch:
diff --git a/pandas/tseries/tests/test_timezones.py b/pandas/tseries/tests/test_timezones.py
--- a/pandas/tseries/tests/test_timezones.py
+++ b/pandas/tseries/tests/test_timezones.py
@@ -863,6 +863,7 @@ def test_cache_keys_are_distinct_for_pytz_vs_dateutil(self):
 
 class TestTimeZones(tm.TestCase):
     _multiprocess_can_split_ = True
+    timezones = ['UTC', 'Asia/Tokyo', 'US/Eastern', 'dateutil/US/Pacific']
 
     def setUp(self):
         tm._skip_if_no_pytz()
@@ -882,6 +883,24 @@ def test_tz_localize_naive(self):
 
         self.assertTrue(conv.equals(exp))
 
+    def test_tz_localize_roundtrip(self):
+        for tz in self.timezones:
+            idx1 = date_range(start='2014-01-01', end='2014-12-31', freq='M')
+            idx2 = date_range(start='2014-01-01', end='2014-12-31', freq='D')
+            idx3 = date_range(start='2014-01-01', end='2014-03-01', freq='H')
+            idx4 = date_range(start='2014-08-01', end='2014-10-31', freq='T')
+            for idx in [idx1, idx2, idx3, idx4]:
+                localized = idx.tz_localize(tz)
+                expected = date_range(start=idx[0], end=idx[-1], freq=idx.freq, tz=tz)
+                tm.assert_index_equal(localized, expected)
+
+                with tm.assertRaises(TypeError):
+                    localized.tz_localize(tz)
+
+                reset = localized.tz_localize(None)
+                tm.assert_index_equal(reset, idx)
+                self.assertTrue(reset.tzinfo is None)
+
     def test_series_frame_tz_localize(self):
 
         rng = date_range('1/1/2011', periods=100, freq='H')
@@ -930,6 +949,29 @@ def test_series_frame_tz_convert(self):
         ts = Series(1, index=rng)
         tm.assertRaisesRegexp(TypeError, "Cannot convert tz-naive", ts.tz_convert, 'US/Eastern')
 
+    def test_tz_convert_roundtrip(self):
+        for tz in self.timezones:
+            idx1 = date_range(start='2014-01-01', end='2014-12-31', freq='M', tz='UTC')
+            exp1 = date_range(start='2014-01-01', end='2014-12-31', freq='M')
+
+            idx2 = date_range(start='2014-01-01', end='2014-12-31', freq='D', tz='UTC')
+            exp2 = date_range(start='2014-01-01', end='2014-12-31', freq='D')
+
+            idx3 = date_range(start='2014-01-01', end='2014-03-01', freq='H', tz='UTC')
+            exp3 = date_range(start='2014-01-01', end='2014-03-01', freq='H')
+
+            idx4 = date_range(start='2014-08-01', end='2014-10-31', freq='T', tz='UTC')
+            exp4 = date_range(start='2014-08-01', end='2014-10-31', freq='T')
+
+
+            for idx, expected in [(idx1, exp1), (idx2, exp2), (idx3, exp3), (idx4, exp4)]:
+                converted = idx.tz_convert(tz)
+                reset = converted.tz_convert(None)
+                tm.assert_index_equal(reset, expected)
+                self.assertTrue(reset.tzinfo is None)
+                tm.assert_index_equal(reset, converted.tz_convert('UTC').tz_localize(None))
+
+
     def test_join_utc_convert(self):
         rng = date_range('1/1/2011', periods=100, freq='H', tz='utc')
 
diff --git a/pandas/tseries/tests/test_tslib.py b/pandas/tseries/tests/test_tslib.py
--- a/pandas/tseries/tests/test_tslib.py
+++ b/pandas/tseries/tests/test_tslib.py
@@ -97,6 +97,33 @@ def test_tz(self):
         self.assertEqual(conv.nanosecond, 5)
         self.assertEqual(conv.hour, 19)
 
+    def test_tz_localize_roundtrip(self):
+        for tz in ['UTC', 'Asia/Tokyo', 'US/Eastern', 'dateutil/US/Pacific']:
+            for t in ['2014-02-01 09:00', '2014-07-08 09:00', '2014-11-01 17:00',
+                      '2014-11-05 00:00']:
+                ts = Timestamp(t)
+                localized = ts.tz_localize(tz)
+                self.assertEqual(localized, Timestamp(t, tz=tz))
+
+                with tm.assertRaises(Exception):
+                    localized.tz_localize(tz)
+
+                reset = localized.tz_localize(None)
+                self.assertEqual(reset, ts)
+                self.assertTrue(reset.tzinfo is None)
+
+    def test_tz_convert_roundtrip(self):
+        for tz in ['UTC', 'Asia/Tokyo', 'US/Eastern', 'dateutil/US/Pacific']:
+            for t in ['2014-02-01 09:00', '2014-07-08 09:00', '2014-11-01 17:00',
+                      '2014-11-05 00:00']:
+                ts = Timestamp(t, tz='UTC')
+                converted = ts.tz_convert(tz)
+
+                reset = converted.tz_convert(None)
+                self.assertEqual(reset, Timestamp(t))
+                self.assertTrue(reset.tzinfo is None)
+                self.assertEqual(reset, converted.tz_convert('UTC').tz_localize(None))
+
     def test_barely_oob_dts(self):
         one_us = np.timedelta64(1).astype('timedelta64[us]')