Repository: pandas-dev/pandas

Issue:
'ValueError: Cannot assign nan to integer series" when calling `where` on Series with non-unique index.
Create a Series with non-unique index:

``` python
>>> import pandas as pd
>>> pd.__version__
'0.12.0'
>>> s1 = pd.Series(range(3))
>>> s2 = pd.Series(range(3))
>>> comb = pd.concat([s1,s2])
>>> comb
0    0
1    1
2    2
0    0
1    1
2    2
dtype: int64
```

As of https://github.com/pydata/pandas/issues/4548 I cannot use `comb[comb<2] =+ 10`, so I tried working with `where` (according to http://pandas.pydata.org/pandas-docs/stable/indexing.html#where-and-masking, stating "To guarantee that selection output has the same shape as the original data, you can use the where method in Series and DataFrame"). But already calling `where` on `comb` is problematic:

``` python
>>> comb.where(comb < 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/projects/bioinfp_apps/Python-2.7.3/lib/python2.7/site-packages/pandas/core/series.py", line 745, in where
    ser._set_with(~cond, other)
  File "/projects/bioinfp_apps/Python-2.7.3/lib/python2.7/site-packages/pandas/core/series.py", line 886, in _set_with
    self._set_values(key, value)
  File "/projects/bioinfp_apps/Python-2.7.3/lib/python2.7/site-packages/pandas/core/series.py", line 904, in _set_values
    values[key] = _index.convert_scalar(values, value)
  File "index.pyx", line 547, in pandas.index.convert_scalar (pandas/index.c:9752)
  File "index.pyx", line 560, in pandas.index.convert_scalar (pandas/index.c:9639)
ValueError: Cannot assign nan to integer series
>>> 
```

Is this expected?



Discussion:
yes this is expected; you cannot (ATM) change the dtype in place (from integer to float), which is what assigning with a `nan` entails. Series act somewhat differently that DataFrames because they have a different super-class, this is slated to be fixed in 0.13.

This type of assignment is more supported in `DataFrames` (as well as dealing with non-unique indices)

```
In [16]: df = DataFrame(dict(a = [0,1,2,0,1,2]))

In [17]: df
Out[17]: 
   a
0  0
1  1
2  2
3  0
4  1
5  2

In [18]: df[df['a']<2]
Out[18]: 
   a
0  0
1  1
3  0
4  1

In [19]: df[df['a']<2] += 2

In [20]: df
Out[20]: 
   a
0  2
1  3
2  2
3  2
4  3
```

If it's expected, it's fine.

But honestly, from the perspective of the normal user, I don't see an "assignment" in the line `comb.where(comb < 2)` at all. The semantics of this line say that we just _read_ and not _write_. This makes it especially hard to see where the "data type change" should be. So the error message comes really surprising. When reading the docs, the user just sees that something like `s.where(s > 0)` should work _or_ provide an understandable error message.

`where` is not strictly an indexing operation, it is shape preserving,which means `nan`s are possibly introduced meaning a dtype change (in this case) is required. It think the docs are pretty clear on this: http://pandas.pydata.org/pandas-docs/dev/indexing.html#where-and-masking.

Integer not supporting `nan` is a general problem (stemming from numpy), see here: http://pandas.pydata.org/pandas-docs/dev/gotchas.html#nan-integer-na-values-and-na-type-promotions

It is important to understand dtype changes; your operations could do unexpected things.

all that said, this is fixed in #3482 :)


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -7,7 +7,7 @@
 
 import pandas as pd
 from pandas.core.base import PandasObject
-from pandas.core.index import Index, MultiIndex, _ensure_index
+from pandas.core.index import Index, MultiIndex, _ensure_index, InvalidIndexError
 import pandas.core.indexing as indexing
 from pandas.core.indexing import _maybe_convert_indices
 from pandas.tseries.index import DatetimeIndex
@@ -2308,6 +2308,10 @@ def where(self, cond, other=np.nan, inplace=False, try_cast=False, raise_on_erro
 
                 _, other = self.align(other, join='left', fill_value=np.nan)
 
+                # if we are NOT aligned, raise as we cannot where index
+                if not all([ other._get_axis(i).equals(ax) for i, ax in enumerate(self.axes) ]):
+                    raise InvalidIndexError
+
             # slice me out of the other
             else:
                 raise NotImplemented
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -1037,9 +1037,13 @@ def __setitem__(self, key, value):
 
         if _is_bool_indexer(key):
             key = _check_bool_indexer(self.index, key)
-            self.where(~key, value, inplace=True)
-        else:
-            self._set_with(key, value)
+            try:
+                self.where(~key, value, inplace=True)
+                return
+            except (InvalidIndexError):
+                pass
+
+        self._set_with(key, value)
 
     def _set_with_engine(self, key, value):
         values = self.values


Test patch:
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -1373,6 +1373,26 @@ def test_where_inplace(self):
         rs.where(cond, -s, inplace=True)
         assert_series_equal(rs, s.where(cond, -s))
 
+    def test_where_dups(self):
+        # GH 4550
+        # where crashes with dups in index
+        s1 = Series(list(range(3)))
+        s2 = Series(list(range(3)))
+        comb = pd.concat([s1,s2])
+        result = comb.where(comb < 2)
+        expected = Series([0,1,np.nan,0,1,np.nan],index=[0,1,2,0,1,2])
+        assert_series_equal(result, expected)
+
+        # GH 4548
+        # inplace updating not working with dups
+        comb[comb<1] = 5
+        expected = Series([5,1,2,5,1,2],index=[0,1,2,0,1,2])
+        assert_series_equal(comb, expected)
+
+        comb[comb<2] += 10
+        expected = Series([5,11,2,5,11,2],index=[0,1,2,0,1,2])
+        assert_series_equal(comb, expected)
+
     def test_mask(self):
         s = Series(np.random.randn(5))
         cond = s > 0