Repository: pandas-dev/pandas

Issue:
ERR: validate partial string indexing with tz-aware end-points
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import dateutil.parser

df = pd.DataFrame(index=pd.date_range('2016-01-01T00:00', '2016-12-31T23:59', freq='T'))

df[
    "2016-01-01T00:00-02:00"
    :
    "2016-01-01T02:03"
]  # returns 124 entries

df[
    dateutil.parser.parse("2016-01-01T00:00-02:00")
    :
    dateutil.parser.parse("2016-01-01T02:03")
]  # returns 4 entries

df[
    pd.Timestamp("2016-01-01T00:00-02:00")
    :
    pd.Timestamp("2016-01-01T02:03")
]  # returns 4 entries
```
#### Problem description

The current behavior is that the time zone information is ignored without any warning when providing strings.

#### Expected Output

Either use the time zone information and only return the four desired entries OR warn the user about the fact that the time zone information is not used

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 32
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.20.1
pytest: None
pip: 9.0.1
setuptools: 28.8.0
Cython: None
numpy: 1.12.1
scipy: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
Your end-point is naive (FYI dateutil.parser is not used anywhere in pandas except as a very last resort as it is slow).

```
In [8]: pd.Timestamp("2016-01-01T00:00-02:00")
Out[8]: Timestamp('2016-01-01 00:00:00-0200', tz='pytz.FixedOffset(-120)')

In [9]: pd.Timestamp("2016-01-01T02:03")
Out[9]: Timestamp('2016-01-01 02:03:00')

In [12]: dateutil.parser.parse("2016-01-01T02:03")
Out[12]: datetime.datetime(2016, 1, 1, 2, 3)

In [13]: dateutil.parser.parse("2016-01-01T00:00-02:00")
Out[13]: datetime.datetime(2016, 1, 1, 0, 0, tzinfo=tzoffset(None, -7200))
```

Full working example
```
In [4]: idx = pd.date_range('2016-01-01T00:00', '2016-12-31T23:59', freq='T')

In [5]: df = pd.DataFrame({'A': range(len(idx))}, index=idx)

In [6]: df
Out[6]: 
                          A
2016-01-01 00:00:00       0
2016-01-01 00:01:00       1
2016-01-01 00:02:00       2
2016-01-01 00:03:00       3
2016-01-01 00:04:00       4
2016-01-01 00:05:00       5
...                     ...
2016-12-31 23:54:00  527034
2016-12-31 23:55:00  527035
2016-12-31 23:56:00  527036
2016-12-31 23:57:00  527037
2016-12-31 23:58:00  527038
2016-12-31 23:59:00  527039

[527040 rows x 1 columns]

In [7]: df["2016-01-01T00:00-02:00":"2016-01-02T00:00-02:00"]
Out[7]: 
                        A
2016-01-01 00:00:00     0
2016-01-01 00:01:00     1
2016-01-01 00:02:00     2
2016-01-01 00:03:00     3
2016-01-01 00:04:00     4
2016-01-01 00:05:00     5
...                   ...
2016-01-01 23:55:00  1435
2016-01-01 23:56:00  1436
2016-01-01 23:57:00  1437
2016-01-01 23:58:00  1438
2016-01-01 23:59:00  1439
2016-01-02 00:00:00  1440

[1441 rows x 1 columns]
```

@1kastner if you are proposing that we raise if the partial-string indexing with a tz must match on both end-points (if given), I would agree.

If you would like to submit a PR would be great. Just step thru to see where the conversions happens.
Thank you very much for this explanation! I will try to find a good spot to insert the proposed exception.
@1kastner want to do a PR for this?
After 17th Oct. (probably a few days earlier) I will have some free time and I'd be interested to solve this. If you prefer to have it solved by somebody else before this date, that'd be no deal for me.
@1kastner totally fine.
So the easiest solution seems now to invoke pd.Timestamp("timestamp-string") on the respective endpoint. I do not agree that the timezones should match. Because in some countries they have daylight saving time, so it is completely normal for [several countries](https://en.wikipedia.org/wiki/Daylight_saving_time) to change their timezone twice a year.

Because of that it is so critical (when not working with UTC) that the timezone information is not ignored.
Well, in `datetimes.py` the parsed offset is totally ignored in the method `_parsed_string_to_bounds`, for my case lines 1292-1296. Instead, the assumed timezone defaults to the timezone of the DatetimeIndex. As said before, because of daylight saving time (also called "summer time" in German), this does not necessarily make sense. You should be free to choose whatever timezone you need for your project, changing with different contexts. I can understand that if no timezone is provided, it defaults to the timezone of the DatetimeIndex timezone instead of UTC. But if a timezone is provided, neglecting it here is not nice.
Please unsubscribe me

On Thu, Oct 19, 2017 at 7:16 AM, 1kastner <notifications@github.com> wrote:

> Well, in datetimes.py the parsed offset is totally ignored in the method
> _parsed_string_to_bounds, for my case lines 1292-1296. Instead, the
> assumed timezone defaults to the timezone of the DatetimeIndex. As said
> before, because of daylight saving time (also called "summer time" in
> German), this does not necessarily make sense. You should be free to choose
> whatever timezone you need for your project, changing with different
> contexts. I can understand that if no timezone is provided, it defaults to
> the timezone of the DatetimeIndex timezone instead of UTC. But if a
> timezone is provided, neglecting it here is not nice.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/16785#issuecomment-337877144>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAJfa05xyf0SkgOM_s_EQhD6A4waACXVks5sty9vgaJpZM4OHoT7>
> .
>



-- 
Greg Gurevich
greggurevich@gmail.com
cell: 1-917-504-7105

@bkmrkr Can't you unsubscribe yourself? 
@jreback Now I struggle with Visual Studio... It might take some some time to properly check everything. But the main idea is to introduce the following change:

```
    def _parsed_string_to_bounds(self, reso, parsed):
        # ...
        if parsed.tzinfo is None:
            target_tz = self.tz
        else:
            target_tz = parsed.tzinfo

        if reso == 'year':
            return (Timestamp(datetime(parsed.year, 1, 1), tz=target_tz),
                    Timestamp(datetime(parsed.year, 12, 31, 23,
                                       59, 59, 999999), tz=target_tz))
    #...
```

Of course throughout all resolutions `self.tz` needs to be replaced with `target_tz`.
At #24076 it seems like something has changed.
I will soon re-check whether this issue still exists or whether it can be closed.
I locally built the version 0.24.0.dev0+1448.gb9284a2.dirty and the problem continued to exist.

Resolving patch:
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -6,9 +6,10 @@
 import numpy as np
 
 from pandas._libs import (
-    Timedelta, algos as libalgos, index as libindex, join as libjoin, lib,
-    tslibs)
+    algos as libalgos, index as libindex, join as libjoin, lib)
 from pandas._libs.lib import is_datetime_array
+from pandas._libs.tslibs import OutOfBoundsDatetime, Timedelta, Timestamp
+from pandas._libs.tslibs.timezones import tz_compare
 import pandas.compat as compat
 from pandas.compat import range, set_function_name, u
 from pandas.compat.numpy import function as nv
@@ -447,7 +448,7 @@ def __new__(cls, data=None, dtype=None, copy=False, name=None,
                             try:
                                 return DatetimeIndex(subarr, copy=copy,
                                                      name=name, **kwargs)
-                            except tslibs.OutOfBoundsDatetime:
+                            except OutOfBoundsDatetime:
                                 pass
 
                     elif inferred.startswith('timedelta'):
@@ -4867,6 +4868,20 @@ def slice_locs(self, start=None, end=None, step=None, kind=None):
             # If it's a reverse slice, temporarily swap bounds.
             start, end = end, start
 
+        # GH 16785: If start and end happen to be date strings with UTC offsets
+        # attempt to parse and check that the offsets are the same
+        if (isinstance(start, (compat.string_types, datetime))
+                and isinstance(end, (compat.string_types, datetime))):
+            try:
+                ts_start = Timestamp(start)
+                ts_end = Timestamp(end)
+            except (ValueError, TypeError):
+                pass
+            else:
+                if not tz_compare(ts_start.tzinfo, ts_end.tzinfo):
+                    raise ValueError("Both dates must have the "
+                                     "same UTC offset")
+
         start_slice = None
         if start is not None:
             start_slice = self.get_slice_bound(start, 'left', kind)
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -32,9 +32,8 @@
 from pandas.core.ops import get_op_result_name
 import pandas.core.tools.datetimes as tools
 
-from pandas.tseries import offsets
 from pandas.tseries.frequencies import Resolution, to_offset
-from pandas.tseries.offsets import CDay, prefix_mapping
+from pandas.tseries.offsets import CDay, Nano, prefix_mapping
 
 
 def _new_DatetimeIndex(cls, d):
@@ -826,54 +825,57 @@ def _parsed_string_to_bounds(self, reso, parsed):
         lower, upper: pd.Timestamp
 
         """
+        valid_resos = {'year', 'month', 'quarter', 'day', 'hour', 'minute',
+                       'second', 'minute', 'second', 'microsecond'}
+        if reso not in valid_resos:
+            raise KeyError
         if reso == 'year':
-            return (Timestamp(datetime(parsed.year, 1, 1), tz=self.tz),
-                    Timestamp(datetime(parsed.year, 12, 31, 23,
-                                       59, 59, 999999), tz=self.tz))
+            start = Timestamp(parsed.year, 1, 1)
+            end = Timestamp(parsed.year, 12, 31, 23, 59, 59, 999999)
         elif reso == 'month':
             d = ccalendar.get_days_in_month(parsed.year, parsed.month)
-            return (Timestamp(datetime(parsed.year, parsed.month, 1),
-                              tz=self.tz),
-                    Timestamp(datetime(parsed.year, parsed.month, d, 23,
-                                       59, 59, 999999), tz=self.tz))
+            start = Timestamp(parsed.year, parsed.month, 1)
+            end = Timestamp(parsed.year, parsed.month, d, 23, 59, 59, 999999)
         elif reso == 'quarter':
             qe = (((parsed.month - 1) + 2) % 12) + 1  # two months ahead
             d = ccalendar.get_days_in_month(parsed.year, qe)  # at end of month
-            return (Timestamp(datetime(parsed.year, parsed.month, 1),
-                              tz=self.tz),
-                    Timestamp(datetime(parsed.year, qe, d, 23, 59,
-                                       59, 999999), tz=self.tz))
+            start = Timestamp(parsed.year, parsed.month, 1)
+            end = Timestamp(parsed.year, qe, d, 23, 59, 59, 999999)
         elif reso == 'day':
-            st = datetime(parsed.year, parsed.month, parsed.day)
-            return (Timestamp(st, tz=self.tz),
-                    Timestamp(Timestamp(st + offsets.Day(),
-                                        tz=self.tz).value - 1))
+            start = Timestamp(parsed.year, parsed.month, parsed.day)
+            end = start + timedelta(days=1) - Nano(1)
         elif reso == 'hour':
-            st = datetime(parsed.year, parsed.month, parsed.day,
-                          hour=parsed.hour)
-            return (Timestamp(st, tz=self.tz),
-                    Timestamp(Timestamp(st + offsets.Hour(),
-                                        tz=self.tz).value - 1))
+            start = Timestamp(parsed.year, parsed.month, parsed.day,
+                              parsed.hour)
+            end = start + timedelta(hours=1) - Nano(1)
         elif reso == 'minute':
-            st = datetime(parsed.year, parsed.month, parsed.day,
-                          hour=parsed.hour, minute=parsed.minute)
-            return (Timestamp(st, tz=self.tz),
-                    Timestamp(Timestamp(st + offsets.Minute(),
-                                        tz=self.tz).value - 1))
+            start = Timestamp(parsed.year, parsed.month, parsed.day,
+                              parsed.hour, parsed.minute)
+            end = start + timedelta(minutes=1) - Nano(1)
         elif reso == 'second':
-            st = datetime(parsed.year, parsed.month, parsed.day,
-                          hour=parsed.hour, minute=parsed.minute,
-                          second=parsed.second)
-            return (Timestamp(st, tz=self.tz),
-                    Timestamp(Timestamp(st + offsets.Second(),
-                                        tz=self.tz).value - 1))
+            start = Timestamp(parsed.year, parsed.month, parsed.day,
+                              parsed.hour, parsed.minute, parsed.second)
+            end = start + timedelta(seconds=1) - Nano(1)
         elif reso == 'microsecond':
-            st = datetime(parsed.year, parsed.month, parsed.day,
-                          parsed.hour, parsed.minute, parsed.second,
-                          parsed.microsecond)
-            return (Timestamp(st, tz=self.tz), Timestamp(st, tz=self.tz))
-        else:
-            raise KeyError
+            start = Timestamp(parsed.year, parsed.month, parsed.day,
+                              parsed.hour, parsed.minute, parsed.second,
+                              parsed.microsecond)
+            end = start + timedelta(microseconds=1) - Nano(1)
+        # GH 24076
+        # If an incoming date string contained a UTC offset, need to localize
+        # the parsed date to this offset first before aligning with the index's
+        # timezone
+        if parsed.tzinfo is not None:
+            if self.tz is None:
+                raise ValueError("The index must be timezone aware "
+                                 "when indexing with a date string with a "
+                                 "UTC offset")
+            start = start.tz_localize(parsed.tzinfo).tz_convert(self.tz)
+            end = end.tz_localize(parsed.tzinfo).tz_convert(self.tz)
+        elif self.tz is not None:
+            start = start.tz_localize(self.tz)
+            end = end.tz_localize(self.tz)
+        return start, end
 
     def _partial_date_slice(self, reso, parsed, use_lhs=True, use_rhs=True):
         is_monotonic = self.is_monotonic


Test patch:
diff --git a/pandas/tests/indexes/datetimes/test_datetime.py b/pandas/tests/indexes/datetimes/test_datetime.py
--- a/pandas/tests/indexes/datetimes/test_datetime.py
+++ b/pandas/tests/indexes/datetimes/test_datetime.py
@@ -102,7 +102,7 @@ def test_stringified_slice_with_tz(self):
         # GH#2658
         import datetime
         start = datetime.datetime.now()
-        idx = date_range(start=start, freq="1d", periods=10)
+        idx = date_range(start=start, freq="1d", periods=10, tz='US/Eastern')
         df = DataFrame(lrange(10), index=idx)
         df["2013-01-14 23:44:34.437768-05:00":]  # no exception here
 
diff --git a/pandas/tests/indexes/datetimes/test_partial_slicing.py b/pandas/tests/indexes/datetimes/test_partial_slicing.py
--- a/pandas/tests/indexes/datetimes/test_partial_slicing.py
+++ b/pandas/tests/indexes/datetimes/test_partial_slicing.py
@@ -396,3 +396,30 @@ def test_selection_by_datetimelike(self, datetimelike, op, expected):
         result = op(df.A, datetimelike)
         expected = Series(expected, name='A')
         tm.assert_series_equal(result, expected)
+
+    @pytest.mark.parametrize('start', [
+        '2018-12-02 21:50:00+00:00', pd.Timestamp('2018-12-02 21:50:00+00:00'),
+        pd.Timestamp('2018-12-02 21:50:00+00:00').to_pydatetime()
+    ])
+    @pytest.mark.parametrize('end', [
+        '2018-12-02 21:52:00+00:00', pd.Timestamp('2018-12-02 21:52:00+00:00'),
+        pd.Timestamp('2018-12-02 21:52:00+00:00').to_pydatetime()
+    ])
+    def test_getitem_with_datestring_with_UTC_offset(self, start, end):
+        # GH 24076
+        idx = pd.date_range(start='2018-12-02 14:50:00-07:00',
+                            end='2018-12-02 14:50:00-07:00', freq='1min')
+        df = pd.DataFrame(1, index=idx, columns=['A'])
+        result = df[start:end]
+        expected = df.iloc[0:3, :]
+        tm.assert_frame_equal(result, expected)
+
+        # GH 16785
+        start = str(start)
+        end = str(end)
+        with pytest.raises(ValueError, match="Both dates must"):
+            df[start:end[:-4] + '1:00']
+
+        with pytest.raises(ValueError, match="The index must be timezone"):
+            df = df.tz_localize(None)
+            df[start:end]