Repository: pandas-dev/pandas

Issue:
BUG: coercion of mixed dt-aware data in Series constructor
I ran into an issue where the behavior of `.apply()` changed from 0.16 to 0.17, causing different results on tz aware data. Extracting the hour of day from datetimes is different for `Series(x).apply(func)` vs `func(x)`. Below is a minimal example of the issue in 0.17, it seems the behavior is the same in 0.18 but different (though still not equal) on master, also shown below.
#### On 0.17.1 and 0.18.0:

``` python
>>> import pandas as pd
>>> def hour_of_day(dt):
...     return dt.hour
...
>>> dt = pd.to_datetime(1462068217, unit='s')
>>> dt_localized = dt.tz_localize('UTC').tz_convert('US/Pacific')
>>> dt_list = [dt, dt_localized]
>>> apply_series = pd.Series(dt_list).apply(hour_of_day)
>>> map_series = pd.Series(map(hour_of_day, dt_list))
>>> print dt_list
[Timestamp('2016-05-01 02:03:37'), Timestamp('2016-04-30 19:03:37-0700', tz='US/Pacific')]
>>> print apply_series
0    2
1    2
dtype: int64
>>> print map_series
0     2
1    19
dtype: int64
>>> print apply_series - map_series
0     0
1   -17
dtype: int64
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Darwin
OS-release: 15.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.17.1
nose: 1.3.7
pip: 8.1.1
setuptools: 20.7.0
Cython: 0.24
numpy: 1.11.0
scipy: 0.17.0
statsmodels: None
IPython: 4.2.0
sphinx: 1.4.1
patsy: 0.4.1
dateutil: 2.5.2
pytz: 2016.3
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.5.2
matplotlib: 1.5.1
openpyxl: 2.3.2
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.8.5
lxml: 3.6.0
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.12
pymysql: 0.6.7.None
psycopg2: None
Jinja2: None
```
#### On 0.16.2 (what I expected):

``` python
# Same setup as above
>>> print apply_series
0     2
1    19
dtype: int64
>>> print map_series
0     2
1    19
dtype: int64
>>> print apply_series - map_series
0    0
1    0
dtype: int64
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Darwin
OS-release: 15.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.16.2
nose: None
Cython: None
numpy: 1.10.4
scipy: None
statsmodels: None
IPython: 4.2.0
sphinx: None
patsy: None
dateutil: 2.5.2
pytz: 2016.3
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
```
#### On master:

``` python
# Same setup as above
>>> print apply_series
0    19
1    19
dtype: int32
>>> print map_series
0     2
1    19
dtype: int64
>>> print apply_series - map_series
0    17
1     0
dtype: int64
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: 05e734ab171be0fda838c6b12839c38fa588da2c
python: 2.7.11.final.0
python-bits: 64
OS: Darwin
OS-release: 15.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.0+203.g05e734a
nose: None
pip: 8.1.1
setuptools: 20.7.0
Cython: None
numpy: 1.10.4
scipy: None
statsmodels: None
xarray: None
IPython: 4.2.0
sphinx: None
patsy: None
dateutil: 2.5.2
pytz: 2016.3
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: None
pandas_datareader: None
```
#### Expected Output

I would expect the output to be [2, 19], as in 0.16, and matching map(f, data). 



Discussion:
So this is correct if you construct the Series correctly.

```
In [17]: dt_list
Out[17]: 
[Timestamp('2016-05-01 02:03:37'),
 Timestamp('2016-04-30 19:03:37-0700', tz='US/Pacific')]

In [18]: Series(dt_list,dtype=object)
Out[18]: 
0          2016-05-01 02:03:37
1    2016-04-30 19:03:37-07:00
dtype: object

In [19]: Series(dt_list,dtype=object).apply(lambda x: x.hour)
Out[19]: 
0     2
1    19
dtype: int64
```

I suppose _this_ is a bug, we should not be coercing a non-tz-aware AND a dt-aware.

```
In [20]: Series(dt_list)
Out[20]: 
0   2016-04-30 19:03:37-07:00
1   2016-04-30 19:03:37-07:00
dtype: datetime64[ns, US/Pacific]
```

However as a user you have to be aware that putting tz-aware and tz-naive data together is basically meaningless.

cc @sinhrks 

Good to keep in mind, sorry my example wasn't the best. I don't think the breakage in my client code was due to a mixing of tz-aware and tz-native data though, of course it is possible I wasn't using the API correctly. Here's some more testing:

In 0.18.0 this is the case:

``` python
>>> pd.Series(dt_localized).apply(hour_of_day)
0    2
dtype: int64
# Adding dtype results in expected behavior.
>>> pd.Series(dt_localized, dtype=object).apply(hour_of_day)
0    19
dtype: int64
```

However in 0.16 and in master it does give the result I expect:

``` python
>>> pd.Series(dt_localized).apply(hour_of_day)
0    19
dtype: int32
```

@frankcleary what you are doing is quite inefficient, don't use apply expect as a last resort. Series holds a single dtype, using `object` is a further inefficiency.

```
In [1]: s =  Series(pd.date_range('20130101',periods=3,tz='US/Pacific', freq='2H'))

In [2]: s
Out[2]: 
0   2013-01-01 00:00:00-08:00
1   2013-01-01 02:00:00-08:00
2   2013-01-01 04:00:00-08:00
dtype: datetime64[ns, US/Pacific]

In [3]: s.dt.hour
Out[3]: 
0    0
1    2
2    4
dtype: int64
```


Resolving patch:
diff --git a/pandas/conftest.py b/pandas/conftest.py
--- a/pandas/conftest.py
+++ b/pandas/conftest.py
@@ -320,3 +320,20 @@ def mock():
         return importlib.import_module("unittest.mock")
     else:
         return pytest.importorskip("mock")
+
+
+@pytest.fixture(params=['__eq__', '__ne__', '__le__',
+                        '__lt__', '__ge__', '__gt__'])
+def all_compare_operators(request):
+    """
+    Fixture for dunder names for common compare operations
+
+    * >=
+    * >
+    * ==
+    * !=
+    * <
+    * <=
+    """
+
+    return request.param


Test patch:
diff --git a/pandas/tests/frame/test_indexing.py b/pandas/tests/frame/test_indexing.py
--- a/pandas/tests/frame/test_indexing.py
+++ b/pandas/tests/frame/test_indexing.py
@@ -2248,6 +2248,16 @@ def test_setitem_datetimelike_with_inference(self):
                           index=list('ABCDEFGH'))
         assert_series_equal(result, expected)
 
+    @pytest.mark.parametrize('idxer', ['var', ['var']])
+    def test_setitem_datetimeindex_tz(self, idxer, tz_naive_fixture):
+        # GH 11365
+        tz = tz_naive_fixture
+        idx = date_range(start='2015-07-12', periods=3, freq='H', tz=tz)
+        expected = DataFrame(1.2, index=idx, columns=['var'])
+        result = DataFrame(index=idx, columns=['var'])
+        result.loc[:, idxer] = expected
+        tm.assert_frame_equal(result, expected)
+
     def test_at_time_between_time_datetimeindex(self):
         index = date_range("2012-01-01", "2012-01-05", freq='30min')
         df = DataFrame(randn(len(index), 5), index=index)
diff --git a/pandas/tests/indexes/datetimes/test_arithmetic.py b/pandas/tests/indexes/datetimes/test_arithmetic.py
--- a/pandas/tests/indexes/datetimes/test_arithmetic.py
+++ b/pandas/tests/indexes/datetimes/test_arithmetic.py
@@ -276,6 +276,10 @@ def test_comparison_tzawareness_compat(self, op):
         with pytest.raises(TypeError):
             op(dz, ts)
 
+        # GH 12601: Check comparison against Timestamps and DatetimeIndex
+        with pytest.raises(TypeError):
+            op(ts, dz)
+
     @pytest.mark.parametrize('op', [operator.eq, operator.ne,
                                     operator.gt, operator.ge,
                                     operator.lt, operator.le])
diff --git a/pandas/tests/indexes/datetimes/test_date_range.py b/pandas/tests/indexes/datetimes/test_date_range.py
--- a/pandas/tests/indexes/datetimes/test_date_range.py
+++ b/pandas/tests/indexes/datetimes/test_date_range.py
@@ -292,6 +292,15 @@ def test_construct_over_dst(self):
                             freq='H', tz='US/Pacific')
         tm.assert_index_equal(result, expected)
 
+    def test_construct_with_different_start_end_string_format(self):
+        # GH 12064
+        result = date_range('2013-01-01 00:00:00+09:00',
+                            '2013/01/01 02:00:00+09:00', freq='H')
+        expected = DatetimeIndex([Timestamp('2013-01-01 00:00:00+09:00'),
+                                  Timestamp('2013-01-01 01:00:00+09:00'),
+                                  Timestamp('2013-01-01 02:00:00+09:00')])
+        tm.assert_index_equal(result, expected)
+
 
 class TestGenRangeGeneration(object):
 
diff --git a/pandas/tests/indexing/test_datetime.py b/pandas/tests/indexing/test_datetime.py
--- a/pandas/tests/indexing/test_datetime.py
+++ b/pandas/tests/indexing/test_datetime.py
@@ -252,3 +252,17 @@ def test_series_partial_set_period(self):
                                         check_stacklevel=False):
             result = ser.loc[keys]
         tm.assert_series_equal(result, exp)
+
+    def test_nanosecond_getitem_setitem_with_tz(self):
+        # GH 11679
+        data = ['2016-06-28 08:30:00.123456789']
+        index = pd.DatetimeIndex(data, dtype='datetime64[ns, America/Chicago]')
+        df = DataFrame({'a': [10]}, index=index)
+        result = df.loc[df.index[0]]
+        expected = Series(10, index=['a'], name=df.index[0])
+        tm.assert_series_equal(result, expected)
+
+        result = df.copy()
+        result.loc[df.index[0], 'a'] = -1
+        expected = DataFrame(-1, index=index, columns=['a'])
+        tm.assert_frame_equal(result, expected)
diff --git a/pandas/tests/scalar/timestamp/test_timestamp.py b/pandas/tests/scalar/timestamp/test_timestamp.py
--- a/pandas/tests/scalar/timestamp/test_timestamp.py
+++ b/pandas/tests/scalar/timestamp/test_timestamp.py
@@ -542,6 +542,14 @@ def test_construct_timestamp_near_dst(self, offset):
         result = Timestamp(expected, tz='Europe/Helsinki')
         assert result == expected
 
+    @pytest.mark.parametrize('arg', [
+        '2013/01/01 00:00:00+09:00', '2013-01-01 00:00:00+09:00'])
+    def test_construct_with_different_string_format(self, arg):
+        # GH 12064
+        result = Timestamp(arg)
+        expected = Timestamp(datetime(2013, 1, 1), tz=pytz.FixedOffset(540))
+        assert result == expected
+
 
 class TestTimestamp(object):
 
diff --git a/pandas/tests/series/test_constructors.py b/pandas/tests/series/test_constructors.py
--- a/pandas/tests/series/test_constructors.py
+++ b/pandas/tests/series/test_constructors.py
@@ -1185,3 +1185,11 @@ def test_constructor_range_dtype(self, dtype):
         expected = Series([0, 1, 2, 3, 4], dtype=dtype or 'int64')
         result = Series(range(5), dtype=dtype)
         tm.assert_series_equal(result, expected)
+
+    def test_constructor_tz_mixed_data(self):
+        # GH 13051
+        dt_list = [Timestamp('2016-05-01 02:03:37'),
+                   Timestamp('2016-04-30 19:03:37-0700', tz='US/Pacific')]
+        result = Series(dt_list)
+        expected = Series(dt_list, dtype=object)
+        tm.assert_series_equal(result, expected)
diff --git a/pandas/tests/series/test_replace.py b/pandas/tests/series/test_replace.py
--- a/pandas/tests/series/test_replace.py
+++ b/pandas/tests/series/test_replace.py
@@ -108,6 +108,13 @@ def test_replace_gh5319(self):
                              pd.Timestamp('20120101'))
         tm.assert_series_equal(result, expected)
 
+        # GH 11792: Test with replacing NaT in a list with tz data
+        ts = pd.Timestamp('2015/01/01', tz='UTC')
+        s = pd.Series([pd.NaT, pd.Timestamp('2015/01/01', tz='UTC')])
+        result = s.replace([np.nan, pd.NaT], pd.Timestamp.min)
+        expected = pd.Series([pd.Timestamp.min, ts], dtype=object)
+        tm.assert_series_equal(expected, result)
+
     def test_replace_with_single_list(self):
         ser = pd.Series([0, 1, 2, 3, 4])
         result = ser.replace([1, 2, 3])
diff --git a/pandas/tests/series/test_timezones.py b/pandas/tests/series/test_timezones.py
--- a/pandas/tests/series/test_timezones.py
+++ b/pandas/tests/series/test_timezones.py
@@ -300,3 +300,11 @@ def test_getitem_pydatetime_tz(self, tzstr):
         dt = datetime(2012, 12, 24, 17, 0)
         time_datetime = tslib._localize_pydatetime(dt, tz)
         assert ts[time_pandas] == ts[time_datetime]
+
+    def test_series_truncate_datetimeindex_tz(self):
+        # GH 9243
+        idx = date_range('4/1/2005', '4/30/2005', freq='D', tz='US/Pacific')
+        s = Series(range(len(idx)), index=idx)
+        result = s.truncate(datetime(2005, 4, 2), datetime(2005, 4, 4))
+        expected = Series([1, 2, 3], index=idx[1:4])
+        tm.assert_series_equal(result, expected)