Repository: pandas-dev/pandas

Issue:
Regression in DataFrame.set_index with class instance column keys
The following code worked in Pandas 0.23.4 but not in Pandas 0.24.0 (I'm on Python 3.7.2).

```python
import pandas as pd

class Thing:
    # (Production code would also ensure a Thing instance's hash
    # and equality testing depended on name and color)

    def __init__(self, name, color):
        self.name = name
        self.color = color
    
    def __str__(self):
        return "<Thing %r>" % (self.name,)

thing1 = Thing('One', 'red')
thing2 = Thing('Two', 'blue')
df = pd.DataFrame({thing1: [0, 1], thing2: [2, 3]})
df.set_index([thing2])
```

In Pandas 0.23.4, I get the following correct result:

```
               <Thing 'One'>
<Thing 'Two'>               
2                          0
3                          1
```

In Pandas 0.24.0, I get the following error:

```Python-traceback
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../venv/lib/python3.7/site-packages/pandas/core/frame.py", line 4153, in set_index
    raise ValueError(err_msg)
ValueError: The parameter "keys" may be a column key, one-dimensional array, or a list containing only valid column keys and one-dimensional arrays.
```

After looking at Pandas 0.24.0's implementation of `DataFrame.set_index`:
https://github.com/pandas-dev/pandas/blob/83eb2428ceb6257042173582f3f436c2c887aa69/pandas/core/frame.py#L4144-L4153
I noticed that `is_scalar` returns `False` for `thing1` in Pandas 0.24.0:

```Python-console
>>> from pandas.core.dtypes.common import is_scalar
>>> is_scalar(thing1)
False
```
I suspect that it is incorrect to test DataFrame column keys using `is_scalar`.

# Output of ``pd.show_versions()``

<details>

## `pd.show_versions()` from Pandas 0.23.4
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.2.final.0
python-bits: 64
OS: Darwin
OS-release: 17.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 18.1
setuptools: 40.4.3
Cython: None
numpy: 1.16.0
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 1.1.2
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

## `pd.show_versions()` from Pandas 0.24.0
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.2.final.0
python-bits: 64
OS: Darwin
OS-release: 17.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.0
pytest: None
pip: 18.1
setuptools: 40.4.3
Cython: None
numpy: 1.16.0
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 1.1.2
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>



Discussion:
We have had quite some discussion lately about `set_index` (see eg https://github.com/pandas-dev/pandas/issues/24046), and the actual change (that started to use `is_scalar`, I think) that caused this regression is https://github.com/pandas-dev/pandas/pull/22486 and https://github.com/pandas-dev/pandas/pull/24762

In general the usage `is_scalar` gives problems with custom objects. Eg we also fixed this in fillna (https://github.com/pandas-dev/pandas/issues/20411).

cc @h-vetinari 
@jorisvandenbossche 

Does pandas support custom objects as labels? I think that's bound to break in many places. The code previously tried *everything* it got as a key, so in this sense this is a regression, yes. 

I'm a bit stumped as for how to deal with this. Column keys should IMO clearly be scalar (or tuples, grudgingly) - and that's the only reason `is_scalar` is there. CC @jreback

@wkschwartz 

Your object (at least the toy version) looks a bit like a tuple. As an immediate workaround, I'd suggest to try inheriting `Thing` from `tuple`, then the `isinstance(..., tuple)`-side should work at least.
In my production code, I use [dataclasses](https://docs.python.org/3/library/dataclasses.html) as custom objects in both column keys and row indices, which worked throughout Pandas 0.23.4. If Pandas 0.24 or later drop support for custom classes in row/column indices, I would be stuck at 0.23.4 forever. This is why I view the change as a regression. 
> Does pandas support custom objects as labels? 

We didn't disallow it previously, so yes.
This may have *happened* to work, but we don't support custom objects as labels explicity. Not against reverting this, but its buyer beware here.
I never could find anything in the documentation that takes a stance on what can or can’t be column keys, except the general notion that DataFrames are dict-like. From this I surmised that column keys should be hashable and immutable. Did I miss something in the documentation?


Resolving patch:
diff --git a/pandas/compat/__init__.py b/pandas/compat/__init__.py
--- a/pandas/compat/__init__.py
+++ b/pandas/compat/__init__.py
@@ -137,6 +137,7 @@ def lfilter(*args, **kwargs):
     reload = reload
     Hashable = collections.abc.Hashable
     Iterable = collections.abc.Iterable
+    Iterator = collections.abc.Iterator
     Mapping = collections.abc.Mapping
     MutableMapping = collections.abc.MutableMapping
     Sequence = collections.abc.Sequence
@@ -199,6 +200,7 @@ def get_range_parameters(data):
 
     Hashable = collections.Hashable
     Iterable = collections.Iterable
+    Iterator = collections.Iterator
     Mapping = collections.Mapping
     MutableMapping = collections.MutableMapping
     Sequence = collections.Sequence
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -33,7 +33,7 @@
 
 from pandas import compat
 from pandas.compat import (range, map, zip, lmap, lzip, StringIO, u,
-                           PY36, raise_with_traceback,
+                           PY36, raise_with_traceback, Iterator,
                            string_and_binary_types)
 from pandas.compat.numpy import function as nv
 from pandas.core.dtypes.cast import (
@@ -4025,7 +4025,8 @@ def set_index(self, keys, drop=True, append=False, inplace=False,
             This parameter can be either a single column key, a single array of
             the same length as the calling DataFrame, or a list containing an
             arbitrary combination of column keys and arrays. Here, "array"
-            encompasses :class:`Series`, :class:`Index` and ``np.ndarray``.
+            encompasses :class:`Series`, :class:`Index`, ``np.ndarray``, and
+            instances of :class:`abc.Iterator`.
         drop : bool, default True
             Delete columns to be used as the new index.
         append : bool, default False
@@ -4104,6 +4105,32 @@ def set_index(self, keys, drop=True, append=False, inplace=False,
         if not isinstance(keys, list):
             keys = [keys]
 
+        err_msg = ('The parameter "keys" may be a column key, one-dimensional '
+                   'array, or a list containing only valid column keys and '
+                   'one-dimensional arrays.')
+
+        missing = []
+        for col in keys:
+            if isinstance(col, (ABCIndexClass, ABCSeries, np.ndarray,
+                                list, Iterator)):
+                # arrays are fine as long as they are one-dimensional
+                # iterators get converted to list below
+                if getattr(col, 'ndim', 1) != 1:
+                    raise ValueError(err_msg)
+            else:
+                # everything else gets tried as a key; see GH 24969
+                try:
+                    found = col in self.columns
+                except TypeError:
+                    raise TypeError(err_msg + ' Received column of '
+                                    'type {}'.format(type(col)))
+                else:
+                    if not found:
+                        missing.append(col)
+
+        if missing:
+            raise KeyError('None of {} are in the columns'.format(missing))
+
         if inplace:
             frame = self
         else:
@@ -4132,6 +4159,9 @@ def set_index(self, keys, drop=True, append=False, inplace=False,
             elif isinstance(col, (list, np.ndarray)):
                 arrays.append(col)
                 names.append(None)
+            elif isinstance(col, Iterator):
+                arrays.append(list(col))
+                names.append(None)
             # from here, col can only be a column label
             else:
                 arrays.append(frame[col]._values)
@@ -4139,6 +4169,15 @@ def set_index(self, keys, drop=True, append=False, inplace=False,
                 if drop:
                     to_remove.append(col)
 
+            if len(arrays[-1]) != len(self):
+                # check newest element against length of calling frame, since
+                # ensure_index_from_sequences would not raise for append=False.
+                raise ValueError('Length mismatch: Expected {len_self} rows, '
+                                 'received array of length {len_col}'.format(
+                                     len_self=len(self),
+                                     len_col=len(arrays[-1])
+                                 ))
+
         index = ensure_index_from_sequences(arrays, names)
 
         if verify_integrity and not index.is_unique:


Test patch:
diff --git a/pandas/tests/frame/test_alter_axes.py b/pandas/tests/frame/test_alter_axes.py
--- a/pandas/tests/frame/test_alter_axes.py
+++ b/pandas/tests/frame/test_alter_axes.py
@@ -178,10 +178,10 @@ def test_set_index_pass_arrays(self, frame_of_index_cols,
     # MultiIndex constructor does not work directly on Series -> lambda
     # We also emulate a "constructor" for the label -> lambda
     # also test index name if append=True (name is duplicate here for A)
-    @pytest.mark.parametrize('box2', [Series, Index, np.array, list,
+    @pytest.mark.parametrize('box2', [Series, Index, np.array, list, iter,
                                       lambda x: MultiIndex.from_arrays([x]),
                                       lambda x: x.name])
-    @pytest.mark.parametrize('box1', [Series, Index, np.array, list,
+    @pytest.mark.parametrize('box1', [Series, Index, np.array, list, iter,
                                       lambda x: MultiIndex.from_arrays([x]),
                                       lambda x: x.name])
     @pytest.mark.parametrize('append, index_name', [(True, None),
@@ -195,6 +195,9 @@ def test_set_index_pass_arrays_duplicate(self, frame_of_index_cols, drop,
         keys = [box1(df['A']), box2(df['A'])]
         result = df.set_index(keys, drop=drop, append=append)
 
+        # if either box is iter, it has been consumed; re-read
+        keys = [box1(df['A']), box2(df['A'])]
+
         # need to adapt first drop for case that both keys are 'A' --
         # cannot drop the same column twice;
         # use "is" because == would give ambiguous Boolean error for containers
@@ -253,25 +256,48 @@ def test_set_index_raise_keys(self, frame_of_index_cols, drop, append):
             df.set_index(['A', df['A'], tuple(df['A'])],
                          drop=drop, append=append)
 
-    @pytest.mark.xfail(reason='broken due to revert, see GH 25085')
     @pytest.mark.parametrize('append', [True, False])
     @pytest.mark.parametrize('drop', [True, False])
-    @pytest.mark.parametrize('box', [set, iter, lambda x: (y for y in x)],
-                             ids=['set', 'iter', 'generator'])
+    @pytest.mark.parametrize('box', [set], ids=['set'])
     def test_set_index_raise_on_type(self, frame_of_index_cols, box,
                                      drop, append):
         df = frame_of_index_cols
 
         msg = 'The parameter "keys" may be a column key, .*'
-        # forbidden type, e.g. set/iter/generator
+        # forbidden type, e.g. set
         with pytest.raises(TypeError, match=msg):
             df.set_index(box(df['A']), drop=drop, append=append)
 
-        # forbidden type in list, e.g. set/iter/generator
+        # forbidden type in list, e.g. set
         with pytest.raises(TypeError, match=msg):
             df.set_index(['A', df['A'], box(df['A'])],
                          drop=drop, append=append)
 
+    # MultiIndex constructor does not work directly on Series -> lambda
+    @pytest.mark.parametrize('box', [Series, Index, np.array, iter,
+                                     lambda x: MultiIndex.from_arrays([x])],
+                             ids=['Series', 'Index', 'np.array',
+                                  'iter', 'MultiIndex'])
+    @pytest.mark.parametrize('length', [4, 6], ids=['too_short', 'too_long'])
+    @pytest.mark.parametrize('append', [True, False])
+    @pytest.mark.parametrize('drop', [True, False])
+    def test_set_index_raise_on_len(self, frame_of_index_cols, box, length,
+                                    drop, append):
+        # GH 24984
+        df = frame_of_index_cols  # has length 5
+
+        values = np.random.randint(0, 10, (length,))
+
+        msg = 'Length mismatch: Expected 5 rows, received array of length.*'
+
+        # wrong length directly
+        with pytest.raises(ValueError, match=msg):
+            df.set_index(box(values), drop=drop, append=append)
+
+        # wrong length in list
+        with pytest.raises(ValueError, match=msg):
+            df.set_index(['A', df.A, box(values)], drop=drop, append=append)
+
     def test_set_index_custom_label_type(self):
         # GH 24969
 
@@ -341,7 +367,7 @@ def __repr__(self):
 
         # missing key
         thing3 = Thing(['Three', 'pink'])
-        msg = '.*'  # due to revert, see GH 25085
+        msg = r"frozenset\(\{'Three', 'pink'\}\)"
         with pytest.raises(KeyError, match=msg):
             # missing label directly
             df.set_index(thing3)
@@ -366,7 +392,7 @@ def __str__(self):
         thing2 = Thing('Two', 'blue')
         df = DataFrame([[0, 2], [1, 3]], columns=[thing1, thing2])
 
-        msg = 'unhashable type.*'
+        msg = 'The parameter "keys" may be a column key, .*'
 
         with pytest.raises(TypeError, match=msg):
             # use custom label directly