Repository: dask/dask

Issue:
fillna with dask DataFrame raises ValueError
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
Attempted to use `a.fillna(b)` on a Dask DataFrame `a` with another Dask DataFrame `b` provided as the fill value. This raised the error
`ValueError: invalid fill value with a <class 'numpy.ndarray'>`

**What you expected to happen**:
A new Dask DataFrame returned containing the non-nan values of `a` and nan values replaced by the equivalent values from `b`

**Minimal Complete Verifiable Example**:

```python
import numpy as np
import pandas as pd
import dask.dataframe as dd

a=dd.from_pandas(pd.DataFrame({"A":[1,2,np.nan,4,5],"B":1}),npartitions=2)
b=dd.from_pandas(pd.DataFrame({"A":[1,2,3,4,5],"B":2}),npartitions=2)
a.fillna(b)
```

**Anything else we need to know?**:

**Environment**:

- Dask version: '2022.02.1'
- Python version: 3.8.10
- Operating System: Windows 10
- Install method (conda, pip, source): pip

<!-- If you are reporting an issue such as scale stability, cluster deadlock.
Please provide a cluster dump state with this issue, by running client.dump_cluster_state()

https://distributed.dask.org/en/stable/api.html?highlight=dump_cluster_state#distributed.Client.dump_cluster_state

-->


A workaround is to instead use
```python
a.fillna(b.compute())
```
but this seems sub-optimal for large datasets.


Discussion:
Thanks for writing this up @tangoed2whiskey! This seems like a reasonable case to support. As a workaround you can do:

```python
import numpy as np
import pandas as pd
import dask.dataframe as dd

a = dd.from_pandas(pd.DataFrame({"A": [1, 2, np.nan, 4, 5], "B": 1}), npartitions=2)
b = dd.from_pandas(pd.DataFrame({"A": [1, 2, 3, 4, 5], "B": 2}), npartitions=2)
a.where(a.notnull(), b).compute()
```

So if you are interested in opening a pull request to enhance `fillna` to accept dask dataframes, you can probably just pass the operation off to `where`.

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -1467,9 +1467,7 @@ def fillna(self, value=None, method=None, limit=None, axis=None):
         axis = self._validate_axis(axis)
         if method is None and limit is not None:
             raise NotImplementedError("fillna with set limit and method=None")
-        if isinstance(value, _Frame):
-            test_value = value._meta_nonempty.values[0]
-        elif isinstance(value, Scalar):
+        if isinstance(value, (_Frame, Scalar)):
             test_value = value._meta_nonempty
         else:
             test_value = value


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -2404,6 +2404,20 @@ def test_fillna_multi_dataframe():
     assert_eq(ddf.B.fillna(ddf.A), df.B.fillna(df.A))
 
 
+def test_fillna_dask_dataframe_input():
+    df = _compat.makeMissingDataframe()
+    df1 = _compat.makeMissingDataframe()
+    ddf = dd.from_pandas(df, npartitions=5)
+    ddf1 = dd.from_pandas(df1, npartitions=3)
+
+    assert_eq(ddf.fillna(ddf1), df.fillna(df1))
+
+    ddf_unknown = dd.from_pandas(df, npartitions=5, sort=False)
+    with pytest.raises(ValueError, match="Not all divisions are known"):
+        # Fails when divisions are unknown
+        assert_eq(ddf_unknown.fillna(ddf1), df.fillna(df1))
+
+
 def test_ffill_bfill():
     df = _compat.makeMissingDataframe()
     ddf = dd.from_pandas(df, npartitions=5, sort=False)