Repository: dask/dask

Issue:
cumsum() doesn't work on tables with >1 partition and columns with identical names
cumsum on a dask dataframe with multiple partitions and columns with identical names raise an exception

To reproduce:

```python
import dask
import dask.dataframe as dd
import pandas as pd

obj = pd.DataFrame([[0, 1, 2],
                    [3, 4, 5],
                    [6, 7, 8]], columns=['a', 'b', 'b'])
dask_df = dd.from_pandas(obj, npartitions=3)
dask_df.cumsum().compute()
```

Exception raised:

```
File ~\workspace\jupyter\jupyter\lib\site-packages\dask\dataframe\core.py:6851, in _take_last.<locals>._last_valid(s)
   6849 for i in range(1, min(10, len(s) + 1)):
   6850     val = s.iloc[-i]
-> 6851     if not pd.isnull(val):
   6852         return val
   6853 else:

File ~\workspace\jupyter\jupyter\lib\site-packages\pandas\core\generic.py:1527, in NDFrame.__nonzero__(self)
   1525 @final
   1526 def __nonzero__(self):
-> 1527     raise ValueError(
   1528         f"The truth value of a {type(self).__name__} is ambiguous. "
   1529         "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
   1530     )

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```

**Environment**:

- Dask version: 2022.05.0
- Python version: 3.9
- Operating System: Linux Centos
- Install method (conda, pip, source): pip



Discussion:
@faulaire Thanks for reporting! I can reproduce this.

Some observations:
- only happens when we have duplicate columns **and** `npartitions`>1
- other cumulative functions like `cummin`, `cummax`, `cumprod` also fail with the same error
This is because of the following code:

```python
    def _last_valid(s):
        for i in range(1, min(10, len(s) + 1)):
            val = s.iloc[-i]
            if not pd.isnull(val):
                return val
        else:
            nonnull = s[s.notna()]
            if not nonnull.empty:
                return nonnull.iloc[-1]
        return None

    if skipna is False:
        return a.iloc[-1]
    else:
        # take last valid value excluding NaN, NaN location may be different
        # in each column
        if is_dataframe_like(a):
            # create Series from appropriate backend dataframe library
            series_typ = type(a.iloc[0:1, 0])
            if a.empty:
                return series_typ([], dtype="float")
            return series_typ(
                {col: _last_valid(a[col]) for col in a.columns}, index=a.columns
            )
        else:
            return _last_valid(a)
```

Thus, if you do `dask_df.cumsum(skipna=False).compute()` it works, because it just calls `iloc`, which is robust to column non-unicity. 

However, with `skipna=True` it must look for the last non-nan value in each column, and here it assumes that column names are unique.

@faulaire Is there any good reason at all to have 2 equally-named columns in your `DataFrame`? This looks to me like a really unusual case, and a consequence of something that shouldn't happen from the beginning.

@pavithraes Perhaps we could raise a very specific error if trying to use a `cum` method with `skipna=True` when columns are repeated. But I'd really try to avoid this from the beginning: I can't think of any good reason to accept frames with non-unique columns; even something like `dask_df[col]`, which we expect to yield a `Series`, could yield a `DataFrame` if there are repeated columns, so personally I'd prevent this at initialization.

Another trivial example is doing `obj['b'].dtype`. This crashes in Pandas because `col` is non-unique, so it should be `.dtypes` instead of `.dtype`. However, with `a` it should be `obj['a'].dtype`.

Even if Pandas allows to do this because index values can be repeated and we can transpose frames with `.T` (and as a consequence columns can be repeated as well), Dask doesn't allow to do this. So I strongly suggest to prevent this at construction.
No good reasons to have 2 equally-names columns in my DataFrames. I was working on another issue #9092 and as you said, this case is handled by iloc that why I had this kind of DataFrames.

+1 to prevent this at construction.


This check is trivial to add in `from_pandas`, but I'm in doubt whether it should be a more general check at a deeper level, perhaps inside `_Frame.__init__()`, or in `__new__()`, or even in `new_dd_object`. 

@rjzamora @jsignell What happens if we try to build a `dd.DataFrame` from another object (let's say reading from Parquet/CSV files) which have repeated columns? Is that even possible? I'm nearly sure that writing a Parquet file with repeated columns isn't possible, because there'd be duplicate entries in the `schema`, but I'm not so sure while reading.

If this is controlled for such inputs, I'd add the check in `from_pandas` to prevent this when initializing from a `pd.DataFrame`, but if it's not controlled it should be done at a deeper level, shared by all possible `dd.DataFrame` construction options. Any thoughts?
It looks like pandas allows columns with duplicate column names, but read methods general coerce (csv) or fail (parquet) if duplicate columns exist. It kind of feels like duplicate column names should be forbidden in the constructor like you suggest, but I can see that being bad for backwards compatibility. It seems like duplicate column names would always be an accident though, so I'd support this being a breaking change as long as we do a deprecation cycle.
If there's a backwards compatibility problem, wouldn't that be caught by some test?

My approach (happy to contribute in this, btw) would be the following: 

1. Checking for duplicate columns in `from_pandas`, and raising an error if there are duplicities in `df` columns.
2. If that passes, trying to move the check to a deeper level (that is, to the functions called by `from_pandas` under the hood, such as `new_dd_object`).
3. Repeating this process till the check is placed as deep as possible in the function stack, subject to tests passing.

Do you agree with this?
> If there's a backwards compatibility problem, wouldn't that be caught by some test?

Not necessarily. This could just not be any test of this behavior. I am proposing explicitly not allowing duplicate column names in the constructor. If someone has that in downstream code, then they would get an error. It's ok to change functionality, we just need to give notice so people have time to adjust and understand the change. 

> 1. Checking for duplicate columns in `from_pandas`, and raising an error if there are duplicates in `df` columns.

This seems correct, and this is something that could happen during graph construction aka before the user calls `.compute()`

> 2. If that passes, trying to move the check to a deeper level (that is, to the functions called by `from_pandas` under the hood, such as `new_dd_object`).

I think this will be unnecessary since there is already a check to make sure that the columns in each partition match the columns of the `_meta` and we are already checking the `_meta` in step 1.

So basically you should only need step 1. But ideally all the IO methods, not just `from_pandas`.
[This pull request](https://github.com/dask/dask/pull/9422) should avoid this case, as it's not possible anymore to have 2 equally-named columns.
I suspect that changing this line: https://github.com/dask/dask/blob/43fede5413e9fad1499880a09e3d823a8b386a15/dask/dataframe/core.py#L7209

with this diff

```diff
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
index 6eb2ca491..8487a51da 100644
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -7206,7 +7206,7 @@ def _take_last(a, skipna=True):
             if a.empty:
                 return series_typ([], dtype="float")
             return series_typ(
-                {col: _last_valid(a[col]) for col in a.columns}, index=a.columns
+                [_last_valid(a.iloc[:, i]) for i in range(len(a.columns))], index=a.columns
             )
         else:
             return _last_valid(a)
```

appears to fix things

```python
In [1]: import dask
   ...: import dask.dataframe as dd
   ...: import pandas as pd
   ...:
   ...: obj = pd.DataFrame([[0, 1, 2],
   ...:                     [3, 4, 5],
   ...:                     [6, 7, 8]], columns=['a', 'b', 'b'])
   ...: dask_df = dd.from_pandas(obj, npartitions=3)
   ...: dask_df.cumsum().compute()
Out[1]:
   a   b   b
0  0   1   2
1  3   5   7
2  9  12  15
```

I'll mark this as a good first issue, if anyone wants to finish it off by writing a test to cover this and make sure this change doesn't break anything else.

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -7216,7 +7216,8 @@ def _last_valid(s):
             if a.empty:
                 return series_typ([], dtype="float")
             return series_typ(
-                {col: _last_valid(a[col]) for col in a.columns}, index=a.columns
+                [_last_valid(a.iloc[:, i]) for i in range(len(a.columns))],
+                index=a.columns,
             )
         else:
             return _last_valid(a)


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -717,6 +717,30 @@ def test_cumulative():
     assert_eq(df.cummax(axis=1, skipna=False), ddf.cummax(axis=1, skipna=False))
     assert_eq(df.cumprod(axis=1, skipna=False), ddf.cumprod(axis=1, skipna=False))
 
+    # With duplicate columns
+    df = pd.DataFrame(np.random.randn(100, 3), columns=list("abb"))
+    ddf = dd.from_pandas(df, 3)
+
+    assert_eq(df.cumsum(), ddf.cumsum())
+    assert_eq(df.cummin(), ddf.cummin())
+    assert_eq(df.cummax(), ddf.cummax())
+    assert_eq(df.cumprod(), ddf.cumprod())
+
+    assert_eq(df.cumsum(skipna=False), ddf.cumsum(skipna=False))
+    assert_eq(df.cummin(skipna=False), ddf.cummin(skipna=False))
+    assert_eq(df.cummax(skipna=False), ddf.cummax(skipna=False))
+    assert_eq(df.cumprod(skipna=False), ddf.cumprod(skipna=False))
+
+    assert_eq(df.cumsum(axis=1), ddf.cumsum(axis=1))
+    assert_eq(df.cummin(axis=1), ddf.cummin(axis=1))
+    assert_eq(df.cummax(axis=1), ddf.cummax(axis=1))
+    assert_eq(df.cumprod(axis=1), ddf.cumprod(axis=1))
+
+    assert_eq(df.cumsum(axis=1, skipna=False), ddf.cumsum(axis=1, skipna=False))
+    assert_eq(df.cummin(axis=1, skipna=False), ddf.cummin(axis=1, skipna=False))
+    assert_eq(df.cummax(axis=1, skipna=False), ddf.cummax(axis=1, skipna=False))
+    assert_eq(df.cumprod(axis=1, skipna=False), ddf.cumprod(axis=1, skipna=False))
+
 
 @pytest.mark.parametrize(
     "func",