Repository: pandas-dev/pandas

Issue:
BUG: DataFrame created with tzinfo cannot use to_dict(orient="records")
#### Code Sample, a copy-pastable example if possible

```python
import psycopg2
import datetime
import pandas as pd

data = [(datetime.datetime(2017, 11, 18, 21, 53, 0, 219225, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=60, name=None)),), (datetime.datetime(2017, 11, 18, 22, 6, 30, 61810, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=60, name=None)),)]

df = pd.DataFrame(list(x))

df.to_dict(orient='records') # fails

```
#### Problem description

The above code fails in pandas 0.21.0 with

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-34cf441e3f50> in <module>()
----> 1 df.to_dict(orient='records')

/Users/bolke/Documents/dev/airflow_env/lib/python2.7/site-packages/pandas-0.21.0-py2.7-macosx-10.12-x86_64.egg/pandas/core/frame.pyc in to_dict(self, orient)
    897             return [dict((k, _maybe_box_datetimelike(v))
    898                          for k, v in zip(self.columns, row))
--> 899                     for row in self.values]
    900         elif orient.lower().startswith('i'):
    901             return dict((k, v.to_dict()) for k, v in self.iterrows())

TypeError: izip argument #2 must support iteration
```

#### Expected Output

Dict of records

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Darwin
OS-release: 17.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.21.0
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.6.0
Cython: None
numpy: 1.13.3
scipy: None
pyarrow: None
xarray: None
IPython: 5.5.0
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0b10
sqlalchemy: 1.1.4
pymysql: None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>

BUG: DataFrame created with tzinfo cannot use to_dict(orient="records")
#### Code Sample, a copy-pastable example if possible

```python
import psycopg2
import datetime
import pandas as pd

data = [(datetime.datetime(2017, 11, 18, 21, 53, 0, 219225, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=60, name=None)),), (datetime.datetime(2017, 11, 18, 22, 6, 30, 61810, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=60, name=None)),)]

df = pd.DataFrame(list(x))

df.to_dict(orient='records') # fails

```
#### Problem description

The above code fails in pandas 0.21.0 with

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-34cf441e3f50> in <module>()
----> 1 df.to_dict(orient='records')

/Users/bolke/Documents/dev/airflow_env/lib/python2.7/site-packages/pandas-0.21.0-py2.7-macosx-10.12-x86_64.egg/pandas/core/frame.pyc in to_dict(self, orient)
    897             return [dict((k, _maybe_box_datetimelike(v))
    898                          for k, v in zip(self.columns, row))
--> 899                     for row in self.values]
    900         elif orient.lower().startswith('i'):
    901             return dict((k, v.to_dict()) for k, v in self.iterrows())

TypeError: izip argument #2 must support iteration
```

#### Expected Output

Dict of records

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Darwin
OS-release: 17.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.21.0
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.6.0
Cython: None
numpy: 1.13.3
scipy: None
pyarrow: None
xarray: None
IPython: 5.5.0
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0b10
sqlalchemy: 1.1.4
pymysql: None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
yeah this looks buggy. a PR to fix would be great!
If you can give me some hints where to look that would be appreciated. I ‘hacked’ the code in core/frame.py to check for iterable, but I don’t think that is really the fix (or is it?). 
see also https://github.com/pandas-dev/pandas/pull/18167/files

```
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 7145fa7..02a1cf4 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -994,7 +994,7 @@ class DataFrame(NDFrame):
         elif orient.lower().startswith('r'):
             return [into_c((k, _maybe_box_datetimelike(v))
                            for k, v in zip(self.columns, row))
-                    for row in self.values]
+                    for row in np.atleast_2d(self.values)]
         elif orient.lower().startswith('i'):
             return into_c((k, v.to_dict(into)) for k, v in self.iterrows())
         else:
```

The issue is that a *single* column of datetimetz returns a 1-d array (rather than a 2-d array) when using ``.values``. This is a more general issue. This will solve it here.
xref https://github.com/pandas-dev/pandas/issues/13407
Thx! I’ll have a look (to add tests) - or are you applying this patch yourself?
nope love for a PR (test and whatsnew note)
Will do. I will need to change your code slightly as it is not the proper fix, but it put me on the right path
yeah this looks buggy. a PR to fix would be great!
If you can give me some hints where to look that would be appreciated. I ‘hacked’ the code in core/frame.py to check for iterable, but I don’t think that is really the fix (or is it?). 
see also https://github.com/pandas-dev/pandas/pull/18167/files

```
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 7145fa7..02a1cf4 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -994,7 +994,7 @@ class DataFrame(NDFrame):
         elif orient.lower().startswith('r'):
             return [into_c((k, _maybe_box_datetimelike(v))
                            for k, v in zip(self.columns, row))
-                    for row in self.values]
+                    for row in np.atleast_2d(self.values)]
         elif orient.lower().startswith('i'):
             return into_c((k, v.to_dict(into)) for k, v in self.iterrows())
         else:
```

The issue is that a *single* column of datetimetz returns a 1-d array (rather than a 2-d array) when using ``.values``. This is a more general issue. This will solve it here.
xref https://github.com/pandas-dev/pandas/issues/13407
Thx! I’ll have a look (to add tests) - or are you applying this patch yourself?
nope love for a PR (test and whatsnew note)
Will do. I will need to change your code slightly as it is not the proper fix, but it put me on the right path

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -993,7 +993,7 @@ def to_dict(self, orient='dict', into=dict):
                           for k, v in compat.iteritems(self))
         elif orient.lower().startswith('r'):
             return [into_c((k, _maybe_box_datetimelike(v))
-                           for k, v in zip(self.columns, row))
+                           for k, v in zip(self.columns, np.atleast_1d(row)))
                     for row in self.values]
         elif orient.lower().startswith('i'):
             return into_c((k, v.to_dict(into)) for k, v in self.iterrows())


Test patch:
diff --git a/pandas/tests/frame/test_convert_to.py b/pandas/tests/frame/test_convert_to.py
--- a/pandas/tests/frame/test_convert_to.py
+++ b/pandas/tests/frame/test_convert_to.py
@@ -1,6 +1,9 @@
 # -*- coding: utf-8 -*-
 
+from datetime import datetime
+
 import pytest
+import pytz
 import collections
 import numpy as np
 
@@ -249,3 +252,18 @@ def test_to_dict_box_scalars(self):
 
         result = DataFrame(d).to_dict(orient='records')
         assert isinstance(result[0]['a'], (int, long))
+
+    def test_frame_to_dict_tz(self):
+        # GH18372 When converting to dict with orient='records' columns of
+        # datetime that are tz-aware were not converted to required arrays
+        data = [(datetime(2017, 11, 18, 21, 53, 0, 219225, tzinfo=pytz.utc),),
+                (datetime(2017, 11, 18, 22, 6, 30, 61810, tzinfo=pytz.utc,),)]
+        df = DataFrame(list(data), columns=["d", ])
+
+        result = df.to_dict(orient='records')
+        expected = [
+            {'d': Timestamp('2017-11-18 21:53:00.219225+0000', tz=pytz.utc)},
+            {'d': Timestamp('2017-11-18 22:06:30.061810+0000', tz=pytz.utc)},
+        ]
+        tm.assert_dict_equal(result[0], expected[0])
+        tm.assert_dict_equal(result[1], expected[1])