Repository: dask/dask

Issue:
DataFrame.__setitem__ fails for index
```python
In [10]: import pandas as pd

In [11]: import dask.dataframe as dd

In [12]: df = pd.DataFrame({"A": [1, 2], "B": [3, 4]})

In [13]: ddf = dd.from_pandas(df, 2)

In [14]: df[df.columns] = 1

In [15]: ddf[ddf.columns] = 1
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-736f0f46dec1> in <module>()
----> 1 ddf[ddf.columns] = 1

~/sandbox/dask/dask/dataframe/core.py in __setitem__(self, key, value)
   2430                                 for k, c in zip(key, value.columns)})
   2431         else:
-> 2432             df = self.assign(**{key: value})
   2433
   2434         self.dask = df.dask

~/Envs/dask-dev/lib/python3.6/site-packages/pandas/core/indexes/base.py in __hash__(self)
   2060
   2061     def __hash__(self):
-> 2062         raise TypeError("unhashable type: %r" % type(self).__name__)
   2063
   2064     def __setitem__(self, key, value):

TypeError: unhashable type: 'Index'
```

pandas index objects aren't hashable, putting them in a dictionary fails. We should probably special Index objects in `DataFrame.__setitem__` and convert them to lists.


Discussion:
Can I take up the task? is anyone looking into this? is there any guide for contributors?
Please feel free to take it. http://dask.pydata.org/en/latest/develop.html

________________________________
From: Sriharsha Hatwar <notifications@github.com>
Sent: Saturday, August 11, 2018 12:58:46 PM
To: dask/dask
Cc: Tom Augspurger; Author
Subject: Re: [dask/dask] DataFrame.__setitem__ fails for index (#3818)


Can I take up the task? is anyone looking into this? is there any guide for contributors?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/dask/dask/issues/3818#issuecomment-412291625>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHInh6IFCiDqWHY7oGDQT-eZj1lNG3ks5uPxtWgaJpZM4ViPTz>.

Hey @TomAugspurger, could you please elaborate this statement ?

<<<
pandas index objects aren't hashable, putting them in a dictionary fails. We should probably special Index objects in DataFrame.__setitem__ and convert them to lists.
<<<

And also let me know if there are already existing test cases to validate the fix or we need to write a new one.

do you think, 
                    `assert_eq(df.sum().sum(), ddf.sum().sum())`.
Can be used as testcase? 

The problem is in https://github.com/dask/dask/blob/b88b805633051df046009650c400a865b5a638af/dask/dataframe/core.py#L2435-L2440

In that `else` block, I think should add a 

```
if isinstance(key, pd.Index)
    key = list(key)
```


Tests can go in dask/dataframe/tests/test_dataframe.py. A good test would be taking my example and ensure the dask and pandas operations give the same output

```python
df = ...
ddf = ...

df[df.columns] = 1
ddf[ddf.columns] = 1
assert_eq(df, ddf)
```
Hey @TomAugspurger thanks for the explanation. Even list is not hashable right? so I am still getting TypeError Unhashable type.
What I think is, remove the else block as anyway we are converting the **key** into a list. just add the 

`if isinstance(key, pd.Index)  
        key = list(key)`

Inside the if block itself and then rest will follow?

or we need to change the 

`self.assign()` function as per the type of the key by sending a new parameter?

PS: Was not able place to second line of code in the new line. 
Hmm yes... I'm a bit confused by the current implementation. I suppose we
want to go down the `if` block, but that seems to expect `value` to be a
dataframe, which it might not be.

On Sun, Aug 19, 2018 at 1:18 PM Sriharsha Hatwar <notifications@github.com>
wrote:

> Hey @TomAugspurger <https://github.com/TomAugspurger> thanks for the
> explanation. Even list is not hashable right? so I am still getting
> TypeError Unhashable type.
> What I think is, remove the else block as anyway we are converting the
> *key* into a list. just add the
>
> if isinstance(key, pd.Index) key = list(key)
>
> Inside the if block itself and then rest will follow?
>
> PS: Was not able to second line of code in the new line.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3818#issuecomment-414145469>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIk4HG9KlcfLYU2Z20PlbUUvvuLTtks5uSawLgaJpZM4ViPTz>
> .
>

Do you have any implementation in mind for this?
Not off the top of my head, no.

________________________________
From: Sriharsha Hatwar <notifications@github.com>
Sent: Tuesday, August 21, 2018 1:30:05 PM
To: dask/dask
Cc: Tom Augspurger; Mention
Subject: Re: [dask/dask] DataFrame.__setitem__ fails for index (#3818)


So do you have any implementation in mind for this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/dask/dask/issues/3818#issuecomment-414776158>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIvgXvM_LL8f5C-kHVc03GK7JGa_Vks5uTFGtgaJpZM4ViPTz>.

hey @TomAugspurger  and @mrocklin, do you think this is still a good first issue kind of fix? do you think I need to dive deeper into the way dask creates the partition for the objects?

Not sure. I haven't had time to look more closely.

On Tue, Sep 4, 2018 at 11:37 AM Sriharsha Hatwar <notifications@github.com>
wrote:

> hey @TomAugspurger <https://github.com/TomAugspurger> and @mrocklin
> <https://github.com/mrocklin>, do you think this is still a good first
> issue kind of fix? do you think I need to dive deeper into the way dask
> creates the partition for the objects?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3818#issuecomment-418435837>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIgHSTsJ1gCNUD3phRoxBtABkYQ32ks5uXqxcgaJpZM4ViPTz>
> .
>

I will have a look at this over the weekend and will submit a PR.

On Wed 12 Sep, 2018, 03:06 Tom Augspurger, <notifications@github.com> wrote:

> Not sure. I haven't had time to look more closely.
>
> On Tue, Sep 4, 2018 at 11:37 AM Sriharsha Hatwar <notifications@github.com
> >
> wrote:
>
> > hey @TomAugspurger <https://github.com/TomAugspurger> and @mrocklin
> > <https://github.com/mrocklin>, do you think this is still a good first
> > issue kind of fix? do you think I need to dive deeper into the way dask
> > creates the partition for the objects?
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub
> > <https://github.com/dask/dask/issues/3818#issuecomment-418435837>, or
> mute
> > the thread
> > <
> https://github.com/notifications/unsubscribe-auth/ABQHIgHSTsJ1gCNUD3phRoxBtABkYQ32ks5uXqxcgaJpZM4ViPTz
> >
> > .
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3818#issuecomment-420433924>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AOI8J6KiUvR5BMyD8WGffLOeVxU1RWBsks5uaCzagaJpZM4ViPTz>
> .
>

I will think on this over the weekend, will ask for assistance if I am not able to do it.

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -2490,9 +2490,13 @@ def __getitem__(self, key):
         raise NotImplementedError(key)
 
     def __setitem__(self, key, value):
-        if isinstance(key, (tuple, list)):
+        if isinstance(key, (tuple, list)) and isinstance(value, DataFrame):
             df = self.assign(**{k: value[c]
                                 for k, c in zip(key, value.columns)})
+
+        elif isinstance(key, pd.Index) and not isinstance(value, DataFrame):
+            key = list(key)
+            df = self.assign(**{k: value for k in key})
         else:
             df = self.assign(**{key: value})
 


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -3320,3 +3320,11 @@ def test_partitions_indexer():
     assert ddf.x.partitions[:3].npartitions == 3
 
     assert ddf.x.partitions[::2].compute().tolist() == [0, 1, 4, 5, 8, 9]
+
+
+def test_setitem():
+    df = pd.DataFrame({"A": [1, 2], "B": [3, 4]})
+    ddf = dd.from_pandas(df.copy(), 2)
+    df[df.columns] = 1
+    ddf[ddf.columns] = 1
+    assert_eq(df, ddf)