Repository: pandas-dev/pandas

Issue:
DataFrame.asof() : Timezone Awareness / Naivety comparison TypeError (incorrect)
```
import pandas as pd

# Create some random Timestamps
timestamp1 = pd.Timestamp('2018-01-01 21:00:05.001+00:00')
timestamp2 = pd.Timestamp('2018-01-01 22:35:10.550+00:00')

# Get an internal timestamp, so asof should give us the lesser value
timestamp_internal = timestamp2 + ((timestamp2 - timestamp1) / 2)

# Create a DataFrame
df = pd.DataFrame(data=[1,2], index=[timestamp1, timestamp2])

# display it
df
                                  0
2018-01-01 21:00:05.001000+00:00  1
2018-01-01 22:35:10.550000+00:00  2

# Now call asof() and show the issue
df.asof(timestamp_internal)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Anaconda3\envs\py36\lib\site-packages\pandas\core\generic.py", line 6144, in asof
    locs = self.index.asof_locs(where, ~(nulls.values))
  File "C:\Anaconda3\envs\py36\lib\site-packages\pandas\core\indexes\base.py", line 2489, in asof_locs
    result[(locs == 0) & (where < self.values[first])] = -1
  File "C:\Anaconda3\envs\py36\lib\site-packages\pandas\core\indexes\datetimes.py", line 136, in wrapper
    self._assert_tzawareness_compat(other)
  File "C:\Anaconda3\envs\py36\lib\site-packages\pandas\core\indexes\datetimes.py", line 672, in _assert_tzawareness_compat
    raise TypeError('Cannot compare tz-naive and tz-aware '
TypeError: Cannot compare tz-naive and tz-aware datetime-like objects

# Demonstrate other errant behavior - Try using a tz-naive Timestamp to lookup the frame
df.asof(timestamp_innie.tz_localize(None))
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Anaconda3\envs\py36\lib\site-packages\pandas\core\generic.py", line 6108, in asof
    if where < start:
  File "pandas\_libs\tslibs\timestamps.pyx", line 164, in pandas._libs.tslibs.timestamps._Timestamp.__richcmp__
  File "pandas\_libs\tslibs\timestamps.pyx", line 224, in pandas._libs.tslibs.timestamps._Timestamp._assert_tzawareness_compat
TypeError: Cannot compare tz-naive and tz-aware timestamps
```
#### Problem description
Somehow the DataFrame index is losing the 'awareness' of the timezone of the original Timestamps. 
This has only been noticed recently, following a recent upgrade to the newest version of pandas, however I cannot say for sure whether or not it has been a very recent change to pandas which has caused this.

I am confident (_though not **100%** certain_) that the expected usage worked previously (_nor am in a position to test right now_).

**EDIT:** Have tested this with pandas=0.22.0 and the expected output is now working fine.

Hopefully the bug is reproducible.

#### Expected Output
0    1
Name: 2018-01-01 23:22:43.324500+0000, dtype: int64

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]

pd.show_versions()
Matplotlib support failed
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 62 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.23.0
pytest: None
pip: 10.0.1
setuptools: 39.1.0
Cython: 0.28.2
numpy: 1.13.3
scipy: 1.1.0
pyarrow: 0.7.0
xarray: 0.10.4
IPython: 6.4.0
sphinx: 1.7.4
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: None
tables: 3.4.3
numexpr: 2.6.5
feather: 0.4.0
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: None
xlsxwriter: 1.0.4
lxml: 4.2.1
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: 1.2.7
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
Thanks for the report. It is indeed reproducible.

The bug is in https://github.com/pandas-dev/pandas/blob/6f1f9759ba8319736c2d51b6d05b071998f1add6/pandas/core/indexes/base.py#L2489

For a tz-aware DatetimeIndex `self.values` is a tz-naive ndarray.

The simplest fix is to maybe make that comparison be `where.values < self.values[first]`
@TomAugspurger I have checked your solution and it actually solve the problem. If you need some help I can make the correction and implement the test.
That would be great. Let us know if you have any issues getting started.

On Tue, May 29, 2018 at 7:41 AM, Marlene Silva Marchena <
notifications@github.com> wrote:

> @TomAugspurger <https://github.com/TomAugspurger> I have checked your
> solution and it actually solve the problem. If you need some help I can
> make the correction and implement the test.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/21194#issuecomment-392801679>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIgxZGpxUKyMP2LT5NlThM67HAoqdks5t3V4OgaJpZM4UMpnE>
> .
>

Any progress on this? 0.23 still doesn't work for me.
Looks like https://github.com/pandas-dev/pandas/pull/21284 is stalled. Feel
free to pick it up if you're interested.

On Fri, Aug 3, 2018 at 3:42 PM Daniel Wang <notifications@github.com> wrote:

> Any progress on this? 0.23 still doesn't work for me.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/21194#issuecomment-410370897>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIjN9y2iRijkGuRainCNll5ChXiuhks5uNLW9gaJpZM4UMpnE>
> .
>


Resolving patch:
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -2463,7 +2463,7 @@ def asof_locs(self, where, mask):
         result = np.arange(len(self))[mask].take(locs)
 
         first = mask.argmax()
-        result[(locs == 0) & (where < self.values[first])] = -1
+        result[(locs == 0) & (where.values < self.values[first])] = -1
 
         return result
 


Test patch:
diff --git a/pandas/tests/frame/test_asof.py b/pandas/tests/frame/test_asof.py
--- a/pandas/tests/frame/test_asof.py
+++ b/pandas/tests/frame/test_asof.py
@@ -1,6 +1,7 @@
 # coding=utf-8
 
 import numpy as np
+import pytest
 from pandas import (DataFrame, date_range, Timestamp, Series,
                     to_datetime)
 
@@ -106,3 +107,21 @@ def test_all_nans(self):
         result = DataFrame(np.nan, index=[1, 2], columns=['A', 'B']).asof(3)
         expected = Series(np.nan, index=['A', 'B'], name=3)
         tm.assert_series_equal(result, expected)
+
+    @pytest.mark.parametrize(
+        "stamp,expected",
+        [(Timestamp('2018-01-01 23:22:43.325+00:00'),
+          Series(2.0, name=Timestamp('2018-01-01 23:22:43.325+00:00'))),
+         (Timestamp('2018-01-01 22:33:20.682+01:00'),
+          Series(1.0, name=Timestamp('2018-01-01 22:33:20.682+01:00'))),
+         ]
+    )
+    def test_time_zone_aware_index(self, stamp, expected):
+        # GH21194
+        # Testing awareness of DataFrame index considering different
+        # UTC and timezone
+        df = DataFrame(data=[1, 2],
+                       index=[Timestamp('2018-01-01 21:00:05.001+00:00'),
+                              Timestamp('2018-01-01 22:35:10.550+00:00')])
+        result = df.asof(stamp)
+        tm.assert_series_equal(result, expected)