Repository: pandas-dev/pandas

Issue:
DataFrame[np.nan] raises TypeError with non-unique columns
#### Code Sample, a copy-pastable example if possible

```python
In [2]: pd.DataFrame(index=range(3), columns=[1, 2, float('nan'), 2])[float('nan')]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-86ff82017f2a> in <module>()
----> 1 pd.DataFrame(index=range(3), columns=[1, 2, float('nan'), 2])[float('nan')]

/home/nobackup/repo/pandas/pandas/core/frame.py in __getitem__(self, key)
   2685             return self._getitem_multilevel(key)
   2686         else:
-> 2687             return self._getitem_column(key)
   2688 
   2689     def _getitem_column(self, key):

/home/nobackup/repo/pandas/pandas/core/frame.py in _getitem_column(self, key)
   2695 
   2696         # duplicate columns & possible reduce dimensionality
-> 2697         result = self._constructor(self._data.get(key))
   2698         if result.columns.is_unique:
   2699             result = result[key]

/home/nobackup/repo/pandas/pandas/core/internals.py in get(self, item, fastpath)
   4128 
   4129             if isna(item):
-> 4130                 raise TypeError("cannot label index with a null key")
   4131 
   4132             indexer = self.items.get_indexer_for([item])

TypeError: cannot label index with a null key

In [3]: pd.DataFrame(index=range(3), columns=[1, 2, float('nan'), 4])[float('nan')]
Out[3]: 
0    NaN
1    NaN
2    NaN
Name: nan, dtype: object

In [4]: pd.Series(index=[1, 2, float('nan'), 2])[float('nan')]
Out[4]: nan

```

#### Problem description

The behavior of ``DataFrame[np.nan]`` on non-unique columns makes no particular sense and deviates from the behavior of ``Series[np.nan]``.

This is mistakenly tested here:
https://github.com/pandas-dev/pandas/blob/636dd01fdacba0c8f0e7b5aaa726165983fc861d/pandas/tests/frame/test_constructors.py#L1665

I will push a fix in few minutes.

#### Expected Output

Same as ``Out[3]:``.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: 415012f4f38ca0cf41717c51e49bd2349cba09a8
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-6-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.24.0.dev0+83.g415012f4f
pytest: 3.5.0
pip: 9.0.1
setuptools: 39.2.0
Cython: 0.25.2
numpy: 1.14.3
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.5.6
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.0dev
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.2.2.post1153+gff6786446
openpyxl: 2.3.0
xlrd: 1.0.0
xlwt: 1.3.0
xlsxwriter: 0.9.6
lxml: 4.1.1
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1

</details>

DataFrame[np.nan] raises TypeError with non-unique columns
#### Code Sample, a copy-pastable example if possible

```python
In [2]: pd.DataFrame(index=range(3), columns=[1, 2, float('nan'), 2])[float('nan')]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-86ff82017f2a> in <module>()
----> 1 pd.DataFrame(index=range(3), columns=[1, 2, float('nan'), 2])[float('nan')]

/home/nobackup/repo/pandas/pandas/core/frame.py in __getitem__(self, key)
   2685             return self._getitem_multilevel(key)
   2686         else:
-> 2687             return self._getitem_column(key)
   2688 
   2689     def _getitem_column(self, key):

/home/nobackup/repo/pandas/pandas/core/frame.py in _getitem_column(self, key)
   2695 
   2696         # duplicate columns & possible reduce dimensionality
-> 2697         result = self._constructor(self._data.get(key))
   2698         if result.columns.is_unique:
   2699             result = result[key]

/home/nobackup/repo/pandas/pandas/core/internals.py in get(self, item, fastpath)
   4128 
   4129             if isna(item):
-> 4130                 raise TypeError("cannot label index with a null key")
   4131 
   4132             indexer = self.items.get_indexer_for([item])

TypeError: cannot label index with a null key

In [3]: pd.DataFrame(index=range(3), columns=[1, 2, float('nan'), 4])[float('nan')]
Out[3]: 
0    NaN
1    NaN
2    NaN
Name: nan, dtype: object

In [4]: pd.Series(index=[1, 2, float('nan'), 2])[float('nan')]
Out[4]: nan

```

#### Problem description

The behavior of ``DataFrame[np.nan]`` on non-unique columns makes no particular sense and deviates from the behavior of ``Series[np.nan]``.

This is mistakenly tested here:
https://github.com/pandas-dev/pandas/blob/636dd01fdacba0c8f0e7b5aaa726165983fc861d/pandas/tests/frame/test_constructors.py#L1665

I will push a fix in few minutes.

#### Expected Output

Same as ``Out[3]:``.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: 415012f4f38ca0cf41717c51e49bd2349cba09a8
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-6-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.24.0.dev0+83.g415012f4f
pytest: 3.5.0
pip: 9.0.1
setuptools: 39.2.0
Cython: 0.25.2
numpy: 1.14.3
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.5.6
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.0dev
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.2.2.post1153+gff6786446
openpyxl: 2.3.0
xlrd: 1.0.0
xlwt: 1.3.0
xlsxwriter: 0.9.6
lxml: 4.1.1
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1

</details>



Discussion:

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -2670,68 +2670,80 @@ def _ixs(self, i, axis=0):
     def __getitem__(self, key):
         key = com._apply_if_callable(key, self)
 
-        # shortcut if we are an actual column
-        is_mi_columns = isinstance(self.columns, MultiIndex)
+        # shortcut if the key is in columns
         try:
-            if key in self.columns and not is_mi_columns:
-                return self._getitem_column(key)
-        except:
+            if self.columns.is_unique and key in self.columns:
+                if self.columns.nlevels > 1:
+                    return self._getitem_multilevel(key)
+                return self._get_item_cache(key)
+        except (TypeError, ValueError):
+            # The TypeError correctly catches non hashable "key" (e.g. list)
+            # The ValueError can be removed once GH #21729 is fixed
             pass
 
-        # see if we can slice the rows
+        # Do we have a slicer (on rows)?
         indexer = convert_to_index_sliceable(self, key)
         if indexer is not None:
-            return self._getitem_slice(indexer)
+            return self._slice(indexer, axis=0)
 
-        if isinstance(key, (Series, np.ndarray, Index, list)):
-            # either boolean or fancy integer index
-            return self._getitem_array(key)
-        elif isinstance(key, DataFrame):
+        # Do we have a (boolean) DataFrame?
+        if isinstance(key, DataFrame):
             return self._getitem_frame(key)
-        elif is_mi_columns:
-            return self._getitem_multilevel(key)
+
+        # Do we have a (boolean) 1d indexer?
+        if com.is_bool_indexer(key):
+            return self._getitem_bool_array(key)
+
+        # We are left with two options: a single key, and a collection of keys,
+        # We interpret tuples as collections only for non-MultiIndex
+        is_single_key = isinstance(key, tuple) or not is_list_like(key)
+
+        if is_single_key:
+            if self.columns.nlevels > 1:
+                return self._getitem_multilevel(key)
+            indexer = self.columns.get_loc(key)
+            if is_integer(indexer):
+                indexer = [indexer]
         else:
-            return self._getitem_column(key)
+            if is_iterator(key):
+                key = list(key)
+            indexer = self.loc._convert_to_indexer(key, axis=1,
+                                                   raise_missing=True)
 
-    def _getitem_column(self, key):
-        """ return the actual column """
+        # take() does not accept boolean indexers
+        if getattr(indexer, "dtype", None) == bool:
+            indexer = np.where(indexer)[0]
 
-        # get column
-        if self.columns.is_unique:
-            return self._get_item_cache(key)
+        data = self._take(indexer, axis=1)
 
-        # duplicate columns & possible reduce dimensionality
-        result = self._constructor(self._data.get(key))
-        if result.columns.is_unique:
-            result = result[key]
+        if is_single_key:
+            # What does looking for a single key in a non-unique index return?
+            # The behavior is inconsistent. It returns a Series, except when
+            # - the key itself is repeated (test on data.shape, #9519), or
+            # - we have a MultiIndex on columns (test on self.columns, #21309)
+            if data.shape[1] == 1 and not isinstance(self.columns, MultiIndex):
+                data = data[key]
 
-        return result
-
-    def _getitem_slice(self, key):
-        return self._slice(key, axis=0)
+        return data
 
-    def _getitem_array(self, key):
+    def _getitem_bool_array(self, key):
         # also raises Exception if object array with NA values
-        if com.is_bool_indexer(key):
-            # warning here just in case -- previously __setitem__ was
-            # reindexing but __getitem__ was not; it seems more reasonable to
-            # go with the __setitem__ behavior since that is more consistent
-            # with all other indexing behavior
-            if isinstance(key, Series) and not key.index.equals(self.index):
-                warnings.warn("Boolean Series key will be reindexed to match "
-                              "DataFrame index.", UserWarning, stacklevel=3)
-            elif len(key) != len(self.index):
-                raise ValueError('Item wrong length %d instead of %d.' %
-                                 (len(key), len(self.index)))
-            # check_bool_indexer will throw exception if Series key cannot
-            # be reindexed to match DataFrame rows
-            key = check_bool_indexer(self.index, key)
-            indexer = key.nonzero()[0]
-            return self._take(indexer, axis=0)
-        else:
-            indexer = self.loc._convert_to_indexer(key, axis=1,
-                                                   raise_missing=True)
-            return self._take(indexer, axis=1)
+        # warning here just in case -- previously __setitem__ was
+        # reindexing but __getitem__ was not; it seems more reasonable to
+        # go with the __setitem__ behavior since that is more consistent
+        # with all other indexing behavior
+        if isinstance(key, Series) and not key.index.equals(self.index):
+            warnings.warn("Boolean Series key will be reindexed to match "
+                          "DataFrame index.", UserWarning, stacklevel=3)
+        elif len(key) != len(self.index):
+            raise ValueError('Item wrong length %d instead of %d.' %
+                             (len(key), len(self.index)))
+
+        # check_bool_indexer will throw exception if Series key cannot
+        # be reindexed to match DataFrame rows
+        key = check_bool_indexer(self.index, key)
+        indexer = key.nonzero()[0]
+        return self._take(indexer, axis=0)
 
     def _getitem_multilevel(self, key):
         loc = self.columns.get_loc(key)
diff --git a/pandas/core/sparse/frame.py b/pandas/core/sparse/frame.py
--- a/pandas/core/sparse/frame.py
+++ b/pandas/core/sparse/frame.py
@@ -148,9 +148,10 @@ def _init_dict(self, data, index, columns, dtype=None):
         if index is None:
             index = extract_index(list(data.values()))
 
-        sp_maker = lambda x: SparseArray(x, kind=self._default_kind,
-                                         fill_value=self._default_fill_value,
-                                         copy=True, dtype=dtype)
+        def sp_maker(x):
+            return SparseArray(x, kind=self._default_kind,
+                               fill_value=self._default_fill_value,
+                               copy=True, dtype=dtype)
         sdict = {}
         for k, v in compat.iteritems(data):
             if isinstance(v, Series):
@@ -397,9 +398,10 @@ def _sanitize_column(self, key, value, **kwargs):
         sanitized_column : SparseArray
 
         """
-        sp_maker = lambda x, index=None: SparseArray(
-            x, index=index, fill_value=self._default_fill_value,
-            kind=self._default_kind)
+        def sp_maker(x, index=None):
+            return SparseArray(x, index=index,
+                               fill_value=self._default_fill_value,
+                               kind=self._default_kind)
         if isinstance(value, SparseSeries):
             clean = value.reindex(self.index).as_sparse_array(
                 fill_value=self._default_fill_value, kind=self._default_kind)
@@ -428,18 +430,6 @@ def _sanitize_column(self, key, value, **kwargs):
         # always return a SparseArray!
         return clean
 
-    def __getitem__(self, key):
-        """
-        Retrieve column or slice from DataFrame
-        """
-        if isinstance(key, slice):
-            date_rng = self.index[key]
-            return self.reindex(date_rng)
-        elif isinstance(key, (np.ndarray, list, Series)):
-            return self._getitem_array(key)
-        else:
-            return self._get_item_cache(key)
-
     def get_value(self, index, col, takeable=False):
         """
         Quickly retrieve single value at passed column and index


Test patch:
diff --git a/pandas/tests/frame/test_constructors.py b/pandas/tests/frame/test_constructors.py
--- a/pandas/tests/frame/test_constructors.py
+++ b/pandas/tests/frame/test_constructors.py
@@ -501,9 +501,11 @@ def test_constructor_dict_of_tuples(self):
         tm.assert_frame_equal(result, expected, check_dtype=False)
 
     def test_constructor_dict_multiindex(self):
-        check = lambda result, expected: tm.assert_frame_equal(
-            result, expected, check_dtype=True, check_index_type=True,
-            check_column_type=True, check_names=True)
+        def check(result, expected):
+            return tm.assert_frame_equal(result, expected, check_dtype=True,
+                                         check_index_type=True,
+                                         check_column_type=True,
+                                         check_names=True)
         d = {('a', 'a'): {('i', 'i'): 0, ('i', 'j'): 1, ('j', 'i'): 2},
              ('b', 'a'): {('i', 'i'): 6, ('i', 'j'): 5, ('j', 'i'): 4},
              ('b', 'c'): {('i', 'i'): 7, ('i', 'j'): 8, ('j', 'i'): 9}}
@@ -1655,19 +1657,21 @@ def check(df):
             for i in range(len(df.columns)):
                 df.iloc[:, i]
 
-            # allow single nans to succeed
             indexer = np.arange(len(df.columns))[isna(df.columns)]
 
-            if len(indexer) == 1:
-                tm.assert_series_equal(df.iloc[:, indexer[0]],
-                                       df.loc[:, np.nan])
-
-            # multiple nans should fail
-            else:
-
+            # No NaN found -> error
+            if len(indexer) == 0:
                 def f():
                     df.loc[:, np.nan]
                 pytest.raises(TypeError, f)
+            # single nan should result in Series
+            elif len(indexer) == 1:
+                tm.assert_series_equal(df.iloc[:, indexer[0]],
+                                       df.loc[:, np.nan])
+            # multiple nans should result in DataFrame
+            else:
+                tm.assert_frame_equal(df.iloc[:, indexer],
+                                      df.loc[:, np.nan])
 
         df = DataFrame([[1, 2, 3], [4, 5, 6]], index=[1, np.nan])
         check(df)
@@ -1683,6 +1687,11 @@ def f():
                        columns=[np.nan, 1.1, 2.2, np.nan])
         check(df)
 
+        # GH 21428 (non-unique columns)
+        df = DataFrame([[0.0, 1, 2, 3.0], [4, 5, 6, 7]],
+                       columns=[np.nan, 1, 2, 2])
+        check(df)
+
     def test_constructor_lists_to_object_dtype(self):
         # from #1074
         d = DataFrame({'a': [np.nan, False]})
diff --git a/pandas/tests/frame/test_indexing.py b/pandas/tests/frame/test_indexing.py
--- a/pandas/tests/frame/test_indexing.py
+++ b/pandas/tests/frame/test_indexing.py
@@ -92,45 +92,46 @@ def test_get(self):
             result = df.get(None)
             assert result is None
 
-    def test_getitem_iterator(self):
+    def test_loc_iterable(self):
         idx = iter(['A', 'B', 'C'])
         result = self.frame.loc[:, idx]
         expected = self.frame.loc[:, ['A', 'B', 'C']]
         assert_frame_equal(result, expected)
 
-        idx = iter(['A', 'B', 'C'])
-        result = self.frame.loc[:, idx]
-        expected = self.frame.loc[:, ['A', 'B', 'C']]
-        assert_frame_equal(result, expected)
+    @pytest.mark.parametrize(
+        "idx_type",
+        [list, iter, Index, set,
+         lambda l: dict(zip(l, range(len(l)))),
+         lambda l: dict(zip(l, range(len(l)))).keys()],
+        ids=["list", "iter", "Index", "set", "dict", "dict_keys"])
+    @pytest.mark.parametrize("levels", [1, 2])
+    def test_getitem_listlike(self, idx_type, levels):
+        # GH 21294
+
+        if levels == 1:
+            frame, missing = self.frame, 'food'
+        else:
+            # MultiIndex columns
+            frame = DataFrame(randn(8, 3),
+                              columns=Index([('foo', 'bar'), ('baz', 'qux'),
+                                             ('peek', 'aboo')],
+                                            name=('sth', 'sth2')))
+            missing = ('good', 'food')
 
-    def test_getitem_list(self):
-        self.frame.columns.name = 'foo'
+        keys = [frame.columns[1], frame.columns[0]]
+        idx = idx_type(keys)
+        idx_check = list(idx_type(keys))
 
-        result = self.frame[['B', 'A']]
-        result2 = self.frame[Index(['B', 'A'])]
+        result = frame[idx]
 
-        expected = self.frame.loc[:, ['B', 'A']]
-        expected.columns.name = 'foo'
+        expected = frame.loc[:, idx_check]
+        expected.columns.names = frame.columns.names
 
         assert_frame_equal(result, expected)
-        assert_frame_equal(result2, expected)
 
-        assert result.columns.name == 'foo'
-
-        with tm.assert_raises_regex(KeyError, 'not in index'):
-            self.frame[['B', 'A', 'food']]
+        idx = idx_type(keys + [missing])
         with tm.assert_raises_regex(KeyError, 'not in index'):
-            self.frame[Index(['B', 'A', 'foo'])]
-
-        # tuples
-        df = DataFrame(randn(8, 3),
-                       columns=Index([('foo', 'bar'), ('baz', 'qux'),
-                                      ('peek', 'aboo')], name=('sth', 'sth2')))
-
-        result = df[[('foo', 'bar'), ('baz', 'qux')]]
-        expected = df.iloc[:, :2]
-        assert_frame_equal(result, expected)
-        assert result.columns.names == ('sth', 'sth2')
+            frame[idx]
 
     def test_getitem_callable(self):
         # GH 12533
@@ -223,7 +224,8 @@ def test_setitem_callable(self):
 
     def test_setitem_other_callable(self):
         # GH 13299
-        inc = lambda x: x + 1
+        def inc(x):
+            return x + 1
 
         df = pd.DataFrame([[-1, 1], [1, -1]])
         df[df > 0] = inc
@@ -2082,7 +2084,8 @@ def test_reindex_level(self):
         icol = ['jim', 'joe', 'jolie']
 
         def verify_first_level(df, level, idx, check_index_type=True):
-            f = lambda val: np.nonzero(df[level] == val)[0]
+            def f(val):
+                return np.nonzero(df[level] == val)[0]
             i = np.concatenate(list(map(f, idx)))
             left = df.set_index(icol).reindex(idx, level=level)
             right = df.iloc[i].set_index(icol)