Repository: pandas-dev/pandas
Pull request: BUG: Casting tz-aware DatetimeIndex to object-dtype ndarray/Index
Also fixes bug with `DateOffset == "infer"` incorrectly raising instead of returning `False`.

Also fixes bug(?) with `pd.Index(dtindex, dtype=object)` returning an index of datetimes instead of Timestamps, potentially losing nanoseconds.

- [x] closes #23491
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
BUG: DatetimeIndex cast to object dtype raises/wrong for tzaware

```
dti = pd.date_range('2016-01-01', periods=3, tz='US/Central')

>>> pd.Index(dti, dtype=object)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 294, in __new__
    dtype=dtype, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/datetimes.py", line 453, in __new__
    raise ValueError("cannot localize from non-UTC data")
ValueError: cannot localize from non-UTC data

>>> np.array(dti, dtype=object)
array([1451628000000000000L, 1451714400000000000L, 1451800800000000000L],
      dtype=object)
```

I expected these to match `pd.Index(list(dti), dtype=object)` and `np.array(list(dti))`, respectively.

Hints from issue discussion:

Commit message: add GH references

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.0.txt b/doc/source/whatsnew/v0.24.0.txt
index 6ace245a4bae1..3586a617a0f35 100644
--- a/doc/source/whatsnew/v0.24.0.txt
+++ b/doc/source/whatsnew/v0.24.0.txt
@@ -1162,6 +1162,7 @@ Offsets
 - Bug in :class:`FY5253` where date offsets could incorrectly raise an ``AssertionError`` in arithmetic operatons (:issue:`14774`)
 - Bug in :class:`DateOffset` where keyword arguments ``week`` and ``milliseconds`` were accepted and ignored.  Passing these will now raise ``ValueError`` (:issue:`19398`)
 - Bug in adding :class:`DateOffset` with :class:`DataFrame` or :class:`PeriodIndex` incorrectly raising ``TypeError`` (:issue:`23215`)
+- Bug in comparing :class:`DateOffset` objects with non-DateOffset objects, particularly strings, raising ``ValueError`` instead of returning ``False`` for equality checks and ``True`` for not-equal checks (:issue:`23524`)
 
 Numeric
 ^^^^^^^
@@ -1356,3 +1357,5 @@ Other
 - :meth:`~pandas.io.formats.style.Styler.bar` now also supports tablewise application (in addition to rowwise and columnwise) with ``axis=None`` and setting clipping range with ``vmin`` and ``vmax`` (:issue:`21548` and :issue:`21526`). ``NaN`` values are also handled properly.
 - Logical operations ``&, |, ^`` between :class:`Series` and :class:`Index` will no longer raise ``ValueError`` (:issue:`22092`)
 - Bug in :meth:`DataFrame.combine_first` in which column types were unexpectedly converted to float (:issue:`20699`)
+- Bug in :class:`Index` where passing a timezone-aware :class:`DatetimeIndex` and `dtype=object` would incorrectly raise a ``ValueError`` (:issue:`23524`)
+- Bug in :class:`DatetimeIndex` where calling `np.array(dtindex, dtype=object)` would incorrectly return an array of ``long`` objects (:issue:`23524`)
diff --git a/pandas/_libs/tslibs/offsets.pyx b/pandas/_libs/tslibs/offsets.pyx
index 78e1269aa5363..d739a60380805 100644
--- a/pandas/_libs/tslibs/offsets.pyx
+++ b/pandas/_libs/tslibs/offsets.pyx
@@ -306,8 +306,11 @@ class _BaseOffset(object):
 
     def __eq__(self, other):
         if is_string_object(other):
-            other = to_offset(other)
-
+            try:
+                other = to_offset(other)
+            except ValueError:
+                # e.g. "infer"
+                return False
         try:
             return self._params == other._params
         except AttributeError:
diff --git a/pandas/core/arrays/datetimes.py b/pandas/core/arrays/datetimes.py
index 0258e1e6e5973..00ae2ccbdc0d2 100644
--- a/pandas/core/arrays/datetimes.py
+++ b/pandas/core/arrays/datetimes.py
@@ -374,6 +374,16 @@ def _resolution(self):
     # ----------------------------------------------------------------
     # Array-like Methods
 
+    def __array__(self, dtype=None):
+        if is_object_dtype(dtype):
+            return np.array(list(self), dtype=object)
+        elif dtype == 'i8':
+            return self.asi8
+
+        # TODO: warn that dtype is not used?
+        #  warn that conversion may be lossy?
+        return self._data.view(np.ndarray)  # follow Index.__array__
+
     def __iter__(self):
         """
         Return an iterator over the boxed values
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
index ae64179b36485..73f699c89dc75 100644
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -301,11 +301,15 @@ def __new__(cls, data=None, dtype=None, copy=False, name=None,
               (dtype is not None and is_datetime64_any_dtype(dtype)) or
                 'tz' in kwargs):
             from pandas import DatetimeIndex
-            result = DatetimeIndex(data, copy=copy, name=name,
-                                   dtype=dtype, **kwargs)
+
             if dtype is not None and is_dtype_equal(_o_dtype, dtype):
-                return Index(result.to_pydatetime(), dtype=_o_dtype)
+                # GH#23524 if `data` is already a TZaware DatetimeIndex,
+                #  then passing dtype=object to the constructor would raise
+                result = DatetimeIndex(data, copy=copy, name=name, **kwargs)
+                return Index(list(result), dtype=_o_dtype)
             else:
+                result = DatetimeIndex(data, copy=copy, name=name,
+                                       dtype=dtype, **kwargs)
                 return result
 
         elif (is_timedelta64_dtype(data) or
diff --git a/pandas/tests/arrays/test_datetimelike.py b/pandas/tests/arrays/test_datetimelike.py
index 3fd03a351de7c..377531c1c748d 100644
--- a/pandas/tests/arrays/test_datetimelike.py
+++ b/pandas/tests/arrays/test_datetimelike.py
@@ -57,6 +57,31 @@ def timedelta_index(request):
 
 class TestDatetimeArray(object):
 
+    def test_array_object_dtype(self, tz_naive_fixture):
+        # GH#23524
+        tz = tz_naive_fixture
+        dti = pd.date_range('2016-01-01', periods=3, tz=tz)
+        arr = DatetimeArrayMixin(dti)
+
+        expected = np.array(list(dti))
+
+        result = np.array(arr, dtype=object)
+        tm.assert_numpy_array_equal(result, expected)
+
+        # also test the DatetimeIndex method while we're at it
+        result = np.array(dti, dtype=object)
+        tm.assert_numpy_array_equal(result, expected)
+
+    def test_array(self, tz_naive_fixture):
+        # GH#23524
+        tz = tz_naive_fixture
+        dti = pd.date_range('2016-01-01', periods=3, tz=tz)
+        arr = DatetimeArrayMixin(dti)
+
+        expected = dti.asi8.view('M8[ns]')
+        result = np.array(arr)
+        tm.assert_numpy_array_equal(result, expected)
+
     def test_from_dti(self, tz_naive_fixture):
         tz = tz_naive_fixture
         dti = pd.date_range('2016-01-01', periods=3, tz=tz)
diff --git a/pandas/tests/indexes/test_base.py b/pandas/tests/indexes/test_base.py
index daebc6e95aac4..726db0cdee4fe 100644
--- a/pandas/tests/indexes/test_base.py
+++ b/pandas/tests/indexes/test_base.py
@@ -132,7 +132,7 @@ def test_construction_list_tuples_nan(self, na_value, vtype):
     @pytest.mark.parametrize("cast_as_obj", [True, False])
     @pytest.mark.parametrize("index", [
         pd.date_range('2015-01-01 10:00', freq='D', periods=3,
-                      tz='US/Eastern'),  # DTI with tz
+                      tz='US/Eastern', name='Green Eggs & Ham'),  # DTI with tz
         pd.date_range('2015-01-01 10:00', freq='D', periods=3),  # DTI no tz
         pd.timedelta_range('1 days', freq='D', periods=3),  # td
         pd.period_range('2015-01-01', freq='D', periods=3)  # period
@@ -145,8 +145,16 @@ def test_constructor_from_index_dtlike(self, cast_as_obj, index):
 
         tm.assert_index_equal(result, index)
 
-        if isinstance(index, pd.DatetimeIndex) and hasattr(index, 'tz'):
+        if isinstance(index, pd.DatetimeIndex):
             assert result.tz == index.tz
+            if cast_as_obj:
+                # GH#23524 check that Index(dti, dtype=object) does not
+                #  incorrectly raise ValueError, and that nanoseconds are not
+                #  dropped
+                index += pd.Timedelta(nanoseconds=50)
+                result = pd.Index(index, dtype=object)
+                assert result.dtype == np.object_
+                assert list(result) == list(index)
 
     @pytest.mark.parametrize("index,has_tz", [
         (pd.date_range('2015-01-01 10:00', freq='D', periods=3,
diff --git a/pandas/tests/tseries/offsets/test_offsets.py b/pandas/tests/tseries/offsets/test_offsets.py
index cbd3e0903b713..d68dd65c9841b 100644
--- a/pandas/tests/tseries/offsets/test_offsets.py
+++ b/pandas/tests/tseries/offsets/test_offsets.py
@@ -187,6 +187,19 @@ def testMult2(self):
         assert self.d + (-5 * self._offset(-10)) == self.d + self._offset(50)
         assert self.d + (-3 * self._offset(-2)) == self.d + self._offset(6)
 
+    def test_compare_str(self):
+        # GH#23524
+        # comparing to strings that cannot be cast to DateOffsets should
+        #  not raise for __eq__ or __ne__
+        if self._offset is None:
+            return
+        off = self._get_offset(self._offset)
+
+        assert not off == "infer"
+        assert off != "foo"
+        # Note: inequalities are only implemented for Tick subclasses;
+        #  tests for this are in test_ticks
+
 
 class TestCommon(Base):
     # exected value created by Base._get_offset
diff --git a/pandas/tests/tseries/offsets/test_ticks.py b/pandas/tests/tseries/offsets/test_ticks.py
index 369c0971f1e9a..128010fe6d32c 100644
--- a/pandas/tests/tseries/offsets/test_ticks.py
+++ b/pandas/tests/tseries/offsets/test_ticks.py
@@ -267,3 +267,25 @@ def test_compare_ticks(cls):
     assert cls(4) > three
     assert cls(3) == cls(3)
     assert cls(3) != cls(4)
+
+
+@pytest.mark.parametrize('cls', tick_classes)
+def test_compare_ticks_to_strs(cls):
+    # GH#23524
+    off = cls(19)
+
+    # These tests should work with any strings, but we particularly are
+    #  interested in "infer" as that comparison is convenient to make in
+    #  Datetime/Timedelta Array/Index constructors
+    assert not off == "infer"
+    assert not "foo" == off
+
+    for left, right in [("infer", off), (off, "infer")]:
+        with pytest.raises(TypeError):
+            left < right
+        with pytest.raises(TypeError):
+            left <= right
+        with pytest.raises(TypeError):
+            left > right
+        with pytest.raises(TypeError):
+            left >= right
diff --git a/pandas/tseries/offsets.py b/pandas/tseries/offsets.py
index 6fb562e301ac2..133b03dee375c 100644
--- a/pandas/tseries/offsets.py
+++ b/pandas/tseries/offsets.py
@@ -2199,9 +2199,18 @@ def apply_index(self, i):
 
 
 def _tick_comp(op):
+    assert op not in [operator.eq, operator.ne]
+
     def f(self, other):
-        return op(self.delta, other.delta)
+        try:
+            return op(self.delta, other.delta)
+        except AttributeError:
+            # comparing with a non-Tick object
+            raise TypeError("Invalid comparison between {cls} and {typ}"
+                            .format(cls=type(self).__name__,
+                                    typ=type(other).__name__))
 
+    f.__name__ = '__{opname}__'.format(opname=op.__name__)
     return f
 
 
@@ -2220,8 +2229,6 @@ def __init__(self, n=1, normalize=False):
     __ge__ = _tick_comp(operator.ge)
     __lt__ = _tick_comp(operator.lt)
     __le__ = _tick_comp(operator.le)
-    __eq__ = _tick_comp(operator.eq)
-    __ne__ = _tick_comp(operator.ne)
 
     def __add__(self, other):
         if isinstance(other, Tick):
@@ -2242,8 +2249,11 @@ def __add__(self, other):
     def __eq__(self, other):
         if isinstance(other, compat.string_types):
             from pandas.tseries.frequencies import to_offset
-
-            other = to_offset(other)
+            try:
+                other = to_offset(other)
+            except ValueError:
+                # e.g. "infer"
+                return False
 
         if isinstance(other, Tick):
             return self.delta == other.delta
@@ -2258,8 +2268,11 @@ def __hash__(self):
     def __ne__(self, other):
         if isinstance(other, compat.string_types):
             from pandas.tseries.frequencies import to_offset
-
-            other = to_offset(other)
+            try:
+                other = to_offset(other)
+            except ValueError:
+                # e.g. "infer"
+                return True
 
         if isinstance(other, Tick):
             return self.delta != other.delta


Code review comments:
diff_hunk: @@ -301,11 +301,15 @@ def __new__(cls, data=None, dtype=None, copy=False, name=None,
               (dtype is not None and is_datetime64_any_dtype(dtype)) or
                 'tz' in kwargs):
             from pandas import DatetimeIndex
-            result = DatetimeIndex(data, copy=copy, name=name,
-                                   dtype=dtype, **kwargs)
+
             if dtype is not None and is_dtype_equal(_o_dtype, dtype):
-                return Index(result.to_pydatetime(), dtype=_o_dtype)
+                # GH#23524 if `data` is already a TZaware DatetimeIndex,
path: pandas/core/indexes/base.py
text: @user1:
this seems more like fixing the sympton rather than problem. either that or your comment is not very instructive here.



@author:
Definitely a problem with the comment.  Passing `dtype=object` to DatetimeIndex _should_ raise.  Anything other than None, datetime64[ns], or DatetimeTZDtype is nonsense in that context.

@user1:
right, so why are you fixing this rather than raising?

@author:
This is in `Index.__new__`, not `DatetimeIndex.__new__`.  The bug that this is addressing is:

```
dti = pd.date_range('2016-01-01', periods=3, tz='US/Pacific')  # arbitrary tz-aware
pd.Index(dti, dtype=object)   # <-- currently raises incorrectly
```