Repository: pandas-dev/pandas

Issue:
ENH: bad directive in to_datetime format - this uses std. strptime zone offset
this is really bad for me since almost all linux logfiles contain zone offset.

As far as I can see there is not workaround without parsing the timestamp twice.
#### Code Sample, a copy-pastable example if possible

_parse common Apache access log timestamp_

```
pd.to_datetime(['28/Jul/2006:10:22:04 -0300'], format='%d/%b/%Y:%H:%M:%S %z')

...
/usr/lib/python2.7/dist-packages/pandas/tseries/tools.pyc in _convert_listlike(arg, box, format, name)
    381                 return DatetimeIndex._simple_new(values, name=name, tz=tz)
    382             except (ValueError, TypeError):
--> 383                 raise e
    384 
    385     if arg is None:

ValueError: 'z' is a bad directive in format '%d/%b/%Y:%H:%M:%S %z'
```
#### Expected Output

_should parse timestamps with zone offset_
#### output of `pd.show_versions()`
## INSTALLED VERSIONS

commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-24-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.17.1
nose: 1.3.7
pip: 8.1.2
setuptools: 23.0.0
Cython: None
numpy: 1.11.0
scipy: 0.17.0
statsmodels: 0.6.1
IPython: 4.2.0
sphinx: 1.3.6
patsy: 0.4.1
dateutil: 2.4.2
pytz: 2014.10
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.4.3
matplotlib: 1.5.1
openpyxl: 2.3.0
xlrd: 0.9.4
xlwt: 0.7.5
xlsxwriter: None
lxml: 3.5.0
bs4: 4.4.1
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
Jinja2: None



Discussion:
Timezones `%z` directive are indeed not supported at the moment. 

Two possible workarounds (without parsing the date in two parts, the offset separately):
- use python 3 and there `datetime.datetime.strptime()` has gained `%z` support:
  
  ```
  In [2]: import datetime
  
  In [3]: datetime.datetime.strptime('28/Jul/2006:10:22:04 -0300','%d/%b/%Y:%H:%M:%S %z')
  Out[3]: datetime.datetime(2006, 7, 28, 10, 22, 4, tzinfo=datetime.timezone(datetime.timedelta(-1, 75600)))
  
  In [4]: import sys
  
  In [5]: sys.version
  Out[5]: '3.5.1 |Anaconda 2.4.0 (64-bit)| (default, Feb 16 2016, 09:49:46) [MSC v.1900 64 bit (AMD64)]'
  ```
- use dateutil. This is able to parse timezone offsets, but apparantly not the format as you have it. If you replace the first colon with a space, it works:
  
  ```
  In [10]: from dateutil import parser
  
  In [11]: parser.parse('28/Jul/2006:10:22:04 -0300')
  ...
  ValueError: Unknown string format
  
  In [12]: parser.parse('28/Jul/2006:10:22:04 -0300'.replace(':', ' ', 1))
  Out[12]: datetime.datetime(2006, 7, 28, 10, 22, 4, tzinfo=tzoffset(None, -10800))
  ```

Both methods you could apply on the values in your datetime column. But that will not necessarily be faster than parsing the timezone offset separately (if performance is a bottleneck)

See also http://stackoverflow.com/questions/1101508/how-to-parse-dates-with-0400-timezone-string-in-python

actually I don't think this would be very hard to add [here](https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L5164) (and should have the single whitespace be optional)

its a pretty regular regex, and we could just be passed in directly as the `tz` arg (which already knows how to handle fixed offsets).

@jorisvandenbossche thanks for the great advice! Obviously the python 2 docu is wrong since %z is actually not supported. I learned a lot! At least for now I go with the dateutil workaround.
@jreback I am not into pandas performance but I assume this would be the fastest option. I just assumed that pandas implements the strptime format which is obviously not the case. I can imagine that others bump into that, too.

@markfink 

> just assumed that pandas implements the strptime format which is obviously not the case.

pandas DOES implement the strptime format parsing, the `%z` is missing 


Resolving patch:
diff --git a/pandas/core/tools/datetimes.py b/pandas/core/tools/datetimes.py
--- a/pandas/core/tools/datetimes.py
+++ b/pandas/core/tools/datetimes.py
@@ -1,7 +1,8 @@
 from datetime import datetime, timedelta, time
-import numpy as np
 from collections import MutableMapping
 
+import numpy as np
+
 from pandas._libs import tslib
 from pandas._libs.tslibs.strptime import array_strptime
 from pandas._libs.tslibs import parsing, conversion
@@ -27,6 +28,7 @@
     ABCDataFrame)
 from pandas.core.dtypes.missing import notna
 from pandas.core import algorithms
+from pandas.compat import zip
 
 
 def _guess_datetime_format_for_array(arr, **kwargs):
@@ -103,6 +105,41 @@ def _convert_and_box_cache(arg, cache_array, box, errors, name=None):
     return result.values
 
 
+def _return_parsed_timezone_results(result, timezones, box, tz):
+    """
+    Return results from array_strptime if a %z or %Z directive was passed.
+
+    Parameters
+    ----------
+    result : ndarray
+        int64 date representations of the dates
+    timezones : ndarray
+        pytz timezone objects
+    box : boolean
+        True boxes result as an Index-like, False returns an ndarray
+    tz : object
+        None or pytz timezone object
+    Returns
+    -------
+    tz_result : ndarray of parsed dates with timezone
+        Returns:
+
+        - Index-like if box=True
+        - ndarray of Timestamps if box=False
+
+    """
+    if tz is not None:
+        raise ValueError("Cannot pass a tz argument when "
+                         "parsing strings with timezone "
+                         "information.")
+    tz_results = np.array([tslib.Timestamp(res).tz_localize(zone) for res, zone
+                           in zip(result, timezones)])
+    if box:
+        from pandas import Index
+        return Index(tz_results)
+    return tz_results
+
+
 def to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False,
                 utc=None, box=True, format=None, exact=True,
                 unit=None, infer_datetime_format=False, origin='unix',
@@ -343,8 +380,11 @@ def _convert_listlike(arg, box, format, name=None, tz=tz):
                 # fallback
                 if result is None:
                     try:
-                        result = array_strptime(arg, format, exact=exact,
-                                                errors=errors)
+                        result, timezones = array_strptime(
+                            arg, format, exact=exact, errors=errors)
+                        if '%Z' in format or '%z' in format:
+                            return _return_parsed_timezone_results(
+                                result, timezones, box, tz)
                     except tslib.OutOfBoundsDatetime:
                         if errors == 'raise':
                             raise


Test patch:
diff --git a/pandas/tests/indexes/datetimes/test_tools.py b/pandas/tests/indexes/datetimes/test_tools.py
--- a/pandas/tests/indexes/datetimes/test_tools.py
+++ b/pandas/tests/indexes/datetimes/test_tools.py
@@ -179,6 +179,59 @@ def test_to_datetime_format_weeks(self, cache):
         for s, format, dt in data:
             assert to_datetime(s, format=format, cache=cache) == dt
 
+    @pytest.mark.parametrize("box,const,assert_equal", [
+        [True, pd.Index, 'assert_index_equal'],
+        [False, np.array, 'assert_numpy_array_equal']])
+    @pytest.mark.parametrize("fmt,dates,expected_dates", [
+        ['%Y-%m-%d %H:%M:%S %Z',
+         ['2010-01-01 12:00:00 UTC'] * 2,
+         [pd.Timestamp('2010-01-01 12:00:00', tz='UTC')] * 2],
+        ['%Y-%m-%d %H:%M:%S %Z',
+         ['2010-01-01 12:00:00 UTC',
+          '2010-01-01 12:00:00 GMT',
+          '2010-01-01 12:00:00 US/Pacific'],
+         [pd.Timestamp('2010-01-01 12:00:00', tz='UTC'),
+          pd.Timestamp('2010-01-01 12:00:00', tz='GMT'),
+          pd.Timestamp('2010-01-01 12:00:00', tz='US/Pacific')]],
+        ['%Y-%m-%d %H:%M:%S%z',
+         ['2010-01-01 12:00:00+0100'] * 2,
+         [pd.Timestamp('2010-01-01 12:00:00',
+                       tzinfo=pytz.FixedOffset(60))] * 2],
+        ['%Y-%m-%d %H:%M:%S %z',
+         ['2010-01-01 12:00:00 +0100'] * 2,
+         [pd.Timestamp('2010-01-01 12:00:00',
+                       tzinfo=pytz.FixedOffset(60))] * 2],
+        ['%Y-%m-%d %H:%M:%S %z',
+         ['2010-01-01 12:00:00 +0100', '2010-01-01 12:00:00 -0100'],
+         [pd.Timestamp('2010-01-01 12:00:00',
+                       tzinfo=pytz.FixedOffset(60)),
+          pd.Timestamp('2010-01-01 12:00:00',
+                       tzinfo=pytz.FixedOffset(-60))]],
+        ['%Y-%m-%d %H:%M:%S %z',
+         ['2010-01-01 12:00:00 Z', '2010-01-01 12:00:00 Z'],
+         [pd.Timestamp('2010-01-01 12:00:00',
+                       tzinfo=pytz.FixedOffset(0)),  # pytz coerces to UTC
+          pd.Timestamp('2010-01-01 12:00:00',
+                       tzinfo=pytz.FixedOffset(0))]]])
+    def test_to_datetime_parse_tzname_or_tzoffset(self, box, const,
+                                                  assert_equal, fmt,
+                                                  dates, expected_dates):
+        # GH 13486
+        result = pd.to_datetime(dates, format=fmt, box=box)
+        expected = const(expected_dates)
+        getattr(tm, assert_equal)(result, expected)
+
+        with pytest.raises(ValueError):
+            pd.to_datetime(dates, format=fmt, box=box, utc=True)
+
+    @pytest.mark.parametrize('offset', [
+        '+0', '-1foo', 'UTCbar', ':10', '+01:000:01', ''])
+    def test_to_datetime_parse_timezone_malformed(self, offset):
+        fmt = '%Y-%m-%d %H:%M:%S %z'
+        date = '2010-01-01 12:00:00 ' + offset
+        with pytest.raises(ValueError):
+            pd.to_datetime([date], format=fmt)
+
 
 class TestToDatetime(object):
     def test_to_datetime_pydatetime(self):