Repository: pandas-dev/pandas

Issue:
Bug in Fancy/Boolean Indexing with nested lists
Fancy or Boolean indexing on a Series has two strange behaviors.  My examples only show the behavior with Fancy indexing, but it's the same for Boolean indexing.
## LHS vs RHS length

```
    >>> s = pd.Series(list('abc'))
    >>> s[[0,1,2]] = range(27)
    >>> list(s)
    [0, 1, 2]
```

I would have expected an error, similar to what I get with slice indexing

```
    >>> s = pd.Series(list('abc'))
    >>> s[0:3] = range(27)
    ValueError: cannot copy sequence with size 27 to array axis with dimension 3
```

An even odder behavior is when you have too few items in the RHS

```
    >>> s = pd.Series(list('abc'))
    >>> s[[0,1,2]] = range(2)
    >>> list(s)
    [0, 1, 0]
```

It seems to be using something like itertools.cycle which seems very arbitrary to me
## Nested RHS

This may seem like a strange use of pandas, but I need to store Python lists

```
    >>> s = pd.Series(list('abc'))
    >>> s[[0,1,2]] = [[100,200], [300,400], [500,600]]
    >>> list(s)
    [100, 200, 300]
```

Very strange.  It's like it flattens the input first.
But this flattening only happens if the nested levels are all the same size.

```
    >>> s = pd.Series(list('abc'))
    >>> s[[0,1,2]] = [[100,200], [300,400], [500,600, 601, 602]]
    >>> list(s)
    [[100,200], [300,400], [500,600, 601, 602]]
```

I know in numpy the array constructor would make a distinction between these two inputs, so maybe that's the reason for the difference, but I still don't see why ndarrays are being flattened.

I can work around the issue by converting the RHS to a 1-D array and passing that in.

```
    >>> s = pd.Series(list('abc'))
    >>> rhs = np.empty(3).astype('object')
    >>> rhs[:] = [[100,200], [300,400], [500,600]]
    >>> s[[0,1,2]] = rhs
    >>> list(s)
    [[100,200], [300,400], [500,600]]
```

Slice indexing doesn't have this problem at all

```
    >>> s = pd.Series(list('abc'))
    >>> s[0:3] = [[100,200], [300,400], [500,600]]
    >>> list(s)
    [[100,200], [300,400], [500,600]]
```

**My Question**: Are these behaviors a _bug_ or a "feature"?  I think Fancy/Boolean indexing should operate the same as slice indexing -- i.e. check for matching lengths and don't auto-convert to numpy array.



Discussion:
Oh boy. Hitting a bunch of buggy/underspecified NumPy stuff here. I'm having a look but may kick this can down the road

This is all NumPy behavior. It's going to be too much work for me to fix this anytime soon. I'm already completely fed up with the NumPy library so i would like to overhaul all this mess to make it consistent at some point in the future

You're right.  I just validated the same bugs on a plain ndarray.  Do you think there is any value in raising this issue on a NumPy forum?

Thanks for looking into these corner cases.  Pandas just keeps getting better and I find myself using it more and more when dealing with any non-trivial dataset.

@jreback is this resolved for pandas now that `Series` isn't an `ndarray` anymore?

Did I miss something? Series is no longer an NDFrame? 

I will take a look - haven't seen this issue before

@cpcloud whoops! miswrote - mean no longer an ndarray

@jtratner No worries! Figured it was something like that....just wanted to stay in the loop!

This is easy to make all of these act the same, just an extension in `where`. Right for `ndim==1` we basically handle a single element and a single list element on the rhs, as well as a boolean indexer that matches the rhs.

so this good (#2745)

```
In [3]: s = Series([1, 2])

In [4]: s[[True, False]] = [0, 1]

In [5]: s
Out[5]: 
0    0
1    2
dtype: int64

```

else it is converted to a `ndarray`. So just need to deal with shorter/longer ones and raise a ValueError.
https://github.com/pydata/pandas/blob/master/pandas/core/generic.py#L2285


Resolving patch:
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -547,10 +547,41 @@ def setitem(self, indexer, value):
             dtype, _ = com._maybe_promote(arr_value.dtype)
             values = values.astype(dtype)
 
+        transf = (lambda x: x.T) if self.ndim == 2 else (lambda x: x)
+        values = transf(values)
+        l = len(values)
+
+        # length checking
+        # boolean with truth values == len of the value is ok too
+        if isinstance(indexer, (np.ndarray, list)):
+            if is_list_like(value) and len(indexer) != len(value):
+                if not (isinstance(indexer, np.ndarray) and indexer.dtype == np.bool_ and len(indexer[indexer]) == len(value)):
+                    raise ValueError("cannot set using a list-like indexer with a different length than the value")
+
+        # slice
+        elif isinstance(indexer, slice):
+
+            if is_list_like(value) and l:
+                start = indexer.start
+                stop = indexer.stop
+                step = indexer.step
+                if start is None:
+                    start = 0
+                elif start < 0:
+                    start += l
+                if stop is None or stop > l:
+                    stop = len(values)
+                elif stop < 0:
+                    stop += l
+                if step is None:
+                    step = 1
+                elif step < 0:
+                    step = abs(step)
+                if (stop-start) / step != len(value):
+                    raise ValueError("cannot set using a slice indexer with a different length than the value")
+
         try:
             # set and return a block
-            transf = (lambda x: x.T) if self.ndim == 2 else (lambda x: x)
-            values = transf(values)
             values[indexer] = value
 
             # coerce and try to infer the dtypes of the result
@@ -561,7 +592,9 @@ def setitem(self, indexer, value):
             values = self._try_coerce_result(values)
             values = self._try_cast_result(values, dtype)
             return [make_block(transf(values), self.items, self.ref_items, ndim=self.ndim, fastpath=True)]
-        except:
+        except (ValueError, TypeError) as detail:
+            raise
+        except (Exception) as detail:
             pass
 
         return [ self ]
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -1096,10 +1096,9 @@ def _set_labels(self, key, value):
         self._set_values(indexer, value)
 
     def _set_values(self, key, value):
-        values = self.values
         if isinstance(key, Series):
             key = key.values
-        values[key] = _index.convert_scalar(values, value)
+        self._data = self._data.setitem(key,value)
 
     # help out SparseSeries
     _get_val_at = ndarray.__getitem__


Test patch:
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -1284,6 +1284,59 @@ def f():
         expected = Series(np.nan,index=[9])
         assert_series_equal(result, expected)
 
+    def test_where_setitem_invalid(self):
+
+        # GH 2702
+        # make sure correct exceptions are raised on invalid list assignment
+
+        # slice
+        s = Series(list('abc'))
+        def f():
+            s[0:3] = list(range(27))
+        self.assertRaises(ValueError, f)
+
+        s[0:3] = list(range(3))
+        expected = Series([0,1,2])
+        assert_series_equal(s, expected)
+
+        # slice with step
+        s = Series(list('abcdef'))
+        def f():
+            s[0:4:2] = list(range(27))
+        self.assertRaises(ValueError, f)
+
+        s = Series(list('abcdef'))
+        s[0:4:2] = list(range(2))
+        expected = Series([0,'b',1,'d','e','f'])
+        assert_series_equal(s, expected)
+
+        # neg slices
+        s = Series(list('abcdef'))
+        def f():
+            s[:-1] = list(range(27))
+        self.assertRaises(ValueError, f)
+
+        s[-3:-1] = list(range(2))
+        expected = Series(['a','b','c',0,1,'f'])
+        assert_series_equal(s, expected)
+
+        # list
+        s = Series(list('abc'))
+        def f():
+            s[[0,1,2]] = list(range(27))
+        self.assertRaises(ValueError, f)
+
+        s = Series(list('abc'))
+        def f():
+            s[[0,1,2]] = list(range(2))
+        self.assertRaises(ValueError, f)
+
+        # scalar
+        s = Series(list('abc'))
+        s[0] = list(range(10))
+        expected = Series([list(range(10)),'b','c'])
+        assert_series_equal(s, expected)
+
     def test_where_broadcast(self):
         # Test a variety of differently sized series
         for size in range(2, 6):
@@ -2550,22 +2603,23 @@ def test_between(self):
         expected = s[5:16].dropna()
         assert_series_equal(result, expected)
 
-    def test_setitem_na_exception(self):
-        def testme1():
-            s = Series([2, 3, 4, 5, 6, 7, 8, 9, 10])
-            s[::2] = np.nan
-
-        def testme2():
-            s = Series([True, True, False, False])
-            s[::2] = np.nan
+    def test_setitem_na(self):
+        # these induce dtype changes
+        expected = Series([np.nan, 3, np.nan, 5, np.nan, 7, np.nan, 9, np.nan])
+        s = Series([2, 3, 4, 5, 6, 7, 8, 9, 10])
+        s[::2] = np.nan
+        assert_series_equal(s, expected)
 
-        def testme3():
-            s = Series(np.arange(10))
-            s[:5] = np.nan
+        # get's coerced to float, right?
+        expected = Series([np.nan, 1, np.nan, 0])
+        s = Series([True, True, False, False])
+        s[::2] = np.nan
+        assert_series_equal(s, expected)
 
-        self.assertRaises(Exception, testme1)
-        self.assertRaises(Exception, testme2)
-        self.assertRaises(Exception, testme3)
+        expected = Series([np.nan, np.nan, np.nan, np.nan, np.nan, 5, 6, 7, 8, 9])
+        s = Series(np.arange(10))
+        s[:5] = np.nan
+        assert_series_equal(s, expected)
 
     def test_scalar_na_cmp_corners(self):
         s = Series([2, 3, 4, 5, 6, 7, 8, 9, 10])