Repository: pandas-dev/pandas

Issue:
fillna() does not work when value parameter is a list
Should raise on a passed list to `value`

The results from the fillna() method are very strange when the value parameter is given a list. 

For example, using a simple example DataFrame:

> df = pandas.DataFrame({'A': [numpy.nan, 1, 2], 'B': [10, numpy.nan, 12], 'C': [[20, 21, 22], [23, 24, 25], numpy.nan]})
> df
>     A   B             C
> 0 NaN  10  [20, 21, 22]
> 1   1 NaN  [23, 24, 25]
> 2   2  12           NaN
> 
> df.fillna(value=[100, 101, 102])
>      A    B             C
> 0  100   10  [20, 21, 22]
> 1    1  101  [23, 24, 25]
> 2    2   12           102

So it appears the values in the list are used to fill the 'holes' in order, if the list has the same length as number of holes. But if the the list is shorter than the number of holes, the behavior changes to using only the first value in the list:

>  df.fillna(value=[100, 101])
>      A    B             C
> 0  100   10  [20, 21, 22]
> 1    1  100  [23, 24, 25]
> 2    2   12           100

If the list is longer than the number of holes, you get something even more odd:

> df.fillna(value=[100, 101, 102, 103])
>      A    B             C
> 0  100   10  [20, 21, 22]
> 1    1  100  [23, 24, 25]
> 2    2   12           102

If you specify provide a dict that specifies the fill values by column, the values from the list are used within that column only:

> df.fillna(value={'C': [100, 101]})
>     A   B             C
> 0 NaN  10  [20, 21, 22]
> 1   1 NaN  [23, 24, 25]
> 2   2  12           100

Since it's not always practical to know the number of NaN values a priori, or to customize the length of the value list to match it, this is problematic. Furthermore, some desired values get over-interpreted and cannot be used:

For example, if you want to actually replace all NaN instances in a single column with the same list (either empty or non-empty), I can't figure out how to do it:

> df.fillna(value={'C': [[100,101]]})
>     A   B             C
> 0 NaN  10  [20, 21, 22]
> 1   1 NaN  [23, 24, 25]
> 2   2  12           100

Indeed, if you specify the empty list nothing is filled:

> df.fillna(value={'C': list()})
>     A   B             C
> 0 NaN  10  [20, 21, 22]
> 1   1 NaN  [23, 24, 25]
> 2   2  12           NaN

But a dict works fine:

> f.fillna(value={'C': {0: 1}})
>     A   B             C
> 0 NaN  10  [20, 21, 22]
> 1   1 NaN  [23, 24, 25]
> 2   2  12        {0: 1}
> 
> df.fillna(value={'C': dict()})
>     A   B             C
> 0 NaN  10  [20, 21, 22]
> 1   1 NaN  [23, 24, 25]
> 2   2  12            {}

So it appears the fillna() is making a lot of decisions about how the fill values should be applied, and certain desired outcomes can't be achieved because it's being too 'clever'.



Discussion:
lists are not allowed (for the reasons you show), should raise on this (only scalar or dict are valid)

FYI keeping lists in a frame, while allowed, it not efficient at all, what excatly are you trying to accomplish?

Good question. I am creating a DataFrame containing a number of key elements of information on a daily process - some of those elements are singular (floats, integers, strings), but some are multiple - and the number of elements can vary day by day from 0 to n. I'm storing those elements currently as lists.  

For example, something like the dummy data frame I used in the notes on the Issue.

If you have any suggestions for alternative approaches, I'd be glad to hear them.

Thanks
Iain

On Tuesday, April 23, 2013 at 12:50 PM, jreback wrote:

> FYI keeping lists in a frame, while allowed, it not efficient at all, what excatly are you trying to accomplish?
> 
> —
> Reply to this email directly or view it on GitHub (https://github.com/pydata/pandas/issues/3435#issuecomment-16870881).

I would use multiple df's in this case, maybe indexed by a common element
(and then wrap a class around it to manage it)

for your singular elements it looks like a single df is good
for the multiple ones

use another frame that is indexed 0..n (could be along index or columns whatever makes sense)

when you are mixing hierarchical and non-hierarchial (singular data) better 2 use different objects


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -3319,7 +3319,7 @@ def fillna(self, value=None, method=None, axis=0, inplace=False,
         value : scalar or dict
             Value to use to fill holes (e.g. 0), alternately a dict of values
             specifying which value to use for each column (columns not in the
-            dict will not be filled)
+            dict will not be filled). This value cannot be a list.
         axis : {0, 1}, default 0
             0: fill column-by-column
             1: fill row-by-row
@@ -3341,6 +3341,9 @@ def fillna(self, value=None, method=None, axis=0, inplace=False,
         -------
         filled : DataFrame
         """
+        if isinstance(value, (list, tuple)):
+            raise TypeError('"value" parameter must be a scalar or dict, but '
+                            'you passed a "{0}"'.format(type(value).__name__))
         self._consolidate_inplace()
 
         axis = self._get_axis_number(axis)
diff --git a/pandas/core/panel.py b/pandas/core/panel.py
--- a/pandas/core/panel.py
+++ b/pandas/core/panel.py
@@ -1007,6 +1007,9 @@ def fillna(self, value=None, method=None):
         --------
         DataFrame.reindex, DataFrame.asfreq
         """
+        if isinstance(value, (list, tuple)):
+            raise TypeError('"value" parameter must be a scalar or dict, but '
+                            'you passed a "{0}"'.format(type(value).__name__))
         if value is None:
             if method is None:
                 raise ValueError('must specify a fill method or value')
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -2729,6 +2729,9 @@ def fillna(self, value=None, method=None, inplace=False,
         -------
         filled : Series
         """
+        if isinstance(value, (list, tuple)):
+            raise TypeError('"value" parameter must be a scalar or dict, but '
+                            'you passed a "{0}"'.format(type(value).__name__))
         if not self._can_hold_na:
             return self.copy() if not inplace else None
 


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -6039,6 +6039,12 @@ def test_fillna_invalid_method(self):
         except ValueError, inst:
             self.assert_('ffil' in str(inst))
 
+    def test_fillna_invalid_value(self):
+        # list
+        self.assertRaises(TypeError, self.frame.fillna, [1, 2])
+        # tuple
+        self.assertRaises(TypeError, self.frame.fillna, (1, 2))
+
     def test_replace_inplace(self):
         self.tsframe['A'][:5] = nan
         self.tsframe['A'][-5:] = nan
diff --git a/pandas/tests/test_panel.py b/pandas/tests/test_panel.py
--- a/pandas/tests/test_panel.py
+++ b/pandas/tests/test_panel.py
@@ -1099,6 +1099,9 @@ def test_fillna(self):
         self.assertRaises(ValueError, self.panel.fillna)
         self.assertRaises(ValueError, self.panel.fillna, 5, method='ffill')
 
+        self.assertRaises(TypeError, self.panel.fillna, [1, 2])
+        self.assertRaises(TypeError, self.panel.fillna, (1, 2))
+
     def test_ffill_bfill(self):
         assert_panel_equal(self.panel.ffill(),
                            self.panel.fillna(method='ffill'))
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -3822,6 +3822,11 @@ def test_fillna_int(self):
         s.fillna(method='ffill', inplace=True)
         assert_series_equal(s.fillna(method='ffill', inplace=False), s)
 
+    def test_fillna_raise(self):
+        s = Series(np.random.randint(-100, 100, 50))
+        self.assertRaises(TypeError, s.fillna, [1, 2])
+        self.assertRaises(TypeError, s.fillna, (1, 2))
+
 #------------------------------------------------------------------------------
 # TimeSeries-specific