Repository: pandas-dev/pandas

Issue:
DataFrame.values not a 2D-array when constructed from timezone-aware datetimes
When a DataFrame column is constructed from timezone-aware datetime objects, its `values` attribute returns a `pandas.DatetimeIndex` instead of a 2D numpy array. This is problematic because the datetime index does not support all operations that a numpy array does.
#### Code Sample, a copy-pastable example if possible

``` python
import datetime
import dateutil
import pandas
import numpy as np
df = pandas.DataFrame()
df['Time'] = [datetime.datetime(2015,1,1,tzinfo=dateutil.tz.tzutc())]
df.dropna(axis=0) # raises ValueError: 'axis' entry is out of bounds
```

Also, `print df.values` returns `DatetimeIndex(['2015-01-01'], dtype='datetime64[ns, UTC]', freq=None)`.
#### Expected Output

The `df.dropna` call should be a no-op.

Compare this to the case when constructed using `df['Time'] = [datetime.datetime(2015,1,1)]`. In that case, `df.dropna` works as expected, and `df.values` is `array([['2014-12-31T16:00:00.000000000-0800']], dtype='datetime64[ns]')`.
#### output of `pd.show_versions()`

```
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 63 Stepping 2, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.18.1
nose: None
pip: 8.0.2
setuptools: 20.1.1
Cython: 0.23.4
numpy: 1.10.4
scipy: 0.17.0
statsmodels: 0.6.1
xarray: None
IPython: 4.1.1
sphinx: 1.3.5
patsy: 0.4.0
dateutil: 2.4.2
pytz: 2015.7
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.4.6
matplotlib: 1.5.1
openpyxl: 2.3.2
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: None
lxml: None
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.12
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None
```



Discussion:
This must be shortcutting on the 1-d case in lcd_types. If you have an additional column it will work.

```
In [10]: df['foo'] = 'bar'

In [11]: df.values
Out[11]: array([[Timestamp('2015-01-01 00:00:00+0000', tz='UTC'), 'bar']], dtype=object)
```

Note that using `.values` is pretty inefficient, nor does numpy support these extended dtypes.

This statement is non-sensical, as numpy barley understands timezones. And an `Index` API is a super-set of 1-d numpy operations.

> This is problematic because the datetime index does not support all operations that a numpy array does.

Agreed; I did not phrase that properly. What I meant is that some operations on a pandas dataframe, such as `dropna` along the 0-axis, fail, apparently because the index is a 1-D structure where a 2-D structure was expected.

Spent a long time debugging this today. It is very surprising to have `dropna` throw an AxisError exception in numpy when the column happens to be `datetime64[ns, UTC]` but not when it's `datetime64[ns]`.

As a workaround, I believe users can use `df[df.column_name.notnull()]` ([as described on StackOverflow](https://stackoverflow.com/questions/13413590/how-to-drop-rows-of-pandas-dataframe-whose-value-in-certain-columns-is-nan#13434501)) instead of `dropna(subset=['column name'])`.

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -6578,7 +6578,9 @@ def count(self, axis=0, level=None, numeric_only=False):
                 # column frames with an extension array
                 result = notna(frame).sum(axis=axis)
             else:
-                counts = notna(frame.values).sum(axis=axis)
+                # GH13407
+                series_counts = notna(frame).sum(axis=axis)
+                counts = series_counts.values
                 result = Series(counts, index=frame._get_agg_axis(axis))
 
         return result.astype('int64')


Test patch:
diff --git a/pandas/tests/frame/test_missing.py b/pandas/tests/frame/test_missing.py
--- a/pandas/tests/frame/test_missing.py
+++ b/pandas/tests/frame/test_missing.py
@@ -8,6 +8,9 @@
 from numpy import nan, random
 import numpy as np
 
+import datetime
+import dateutil
+
 from pandas.compat import lrange
 from pandas import (DataFrame, Series, Timestamp,
                     date_range, Categorical)
@@ -183,6 +186,26 @@ def test_dropna_multiple_axes(self):
         inp.dropna(how='all', axis=(0, 1), inplace=True)
         assert_frame_equal(inp, expected)
 
+    def test_dropna_tz_aware_datetime(self):
+        # GH13407
+        df = DataFrame()
+        dt1 = datetime.datetime(2015, 1, 1,
+                                tzinfo=dateutil.tz.tzutc())
+        dt2 = datetime.datetime(2015, 2, 2,
+                                tzinfo=dateutil.tz.tzutc())
+        df['Time'] = [dt1]
+        result = df.dropna(axis=0)
+        expected = DataFrame({'Time': [dt1]})
+        assert_frame_equal(result, expected)
+
+        # Ex2
+        df = DataFrame({'Time': [dt1, None, np.nan, dt2]})
+        result = df.dropna(axis=0)
+        expected = DataFrame([dt1, dt2],
+                             columns=['Time'],
+                             index=[0, 3])
+        assert_frame_equal(result, expected)
+
     def test_fillna(self):
         tf = self.tsframe
         tf.loc[tf.index[:5], 'A'] = nan