Repository: dask/dask

Issue:
Dask DataFrame set_index does not work when column name matches index name
**What happened:**

Calling set_index providing a column name that matches the name of the index does not perform the set_index.

**What you expected to happen:**

The provided column becomes the index, as was the case prior to Dask==2022.05.1

**Minimal Complete Verifiable Example:**

```python
import dask.dataframe as dd
import pandas as pd

df = dd.from_pandas(pd.DataFrame([
    {'a': 1, 'b': 2},
    {'a': 2, 'b': 1}]
), npartitions=1)
df = df.set_index('a')
df = df.rename(columns={'b': 'a'})
df.set_index('a').head()
```

**Anything else we need to know?:**

This is the [branch](https://github.com/dask/dask/blob/0b637ab775fa177d0af737e4648a89484aee5c74/dask/dataframe/core.py#L4816) responsible for the behavior, it was introduced in a commit that seems [unrelated](https://github.com/dask/dask/commit/d458533b5f04494c8eb070adaff6470cea2cf6b0). Dask versions prior to 2022.05.1 are unaffected.

**Environment**

Dask version: 2022.5.1
Python version: 3.10.5
Operating System: Ubuntu 20.04
Install method (conda, pip, source): pip


Discussion:
At the moment it looks like Dask chooses to do nothing if the column name matches the index name:
https://github.com/dask/dask/blob/0b637ab775fa177d0af737e4648a89484aee5c74/dask/dataframe/core.py#L4813-L4817

@jorloplaz do you have any thoughts whether this optimization was intended for a particular workflow ?
IMHO, a `DataFrame` should never have 2 columns with the same name, and this should be controlled at all times (see the discussion regarding #9122). This includes as a particular case the one currently acting as the index, so in your example the call to  `rename` should raise an error, as there would be two columns called `'a'`.  

Think about it in simpler terms: if you want to use `'b'` as index, why not using `set_index('b')` directly?

Assuming the above holds, once you have a column acting as index, calling `set_index` on that same column should do nothing (be it in string form or in `Series` form; so doing `set_index(df.index)` should also be a no-op). 

The only particular case where it could trigger some action is if the index is found to be unsorted on runtime (e.g., because `known_divisions=False` or `df.index.is_monotonic_increasing` happens to be `False`). 

@jsignell @pavithraes Any thoughts on this?
Given that currently nothing prevents you from having multiple columns with the same name, perhaps it's better to check whether there exists a column that matches the index name in `set_index` before assuming it should be a no-op?
I don't think that solves the problem. What happens above if you do first `reset_index()` and then `set_index('a')`. My guess is that, since there's another column called `'a'` when resetting, it is reset with the name `'index'`, and then you end up with two columns `'a'` and `'index'` instead of `'b'` and `'a'`, which was the original situation.

Even if Dask doesn't currently prevent you now from having multiple columns with the same name, there's no good reason why you should do it. In my view, name clashes show a flaw in the processing pipeline.

Pandas does allow non-unicity because in any moment you can transpose the DataFrame with `.T`, so columns can become the index, and the index can for sure have duplicate values. But Dask is very different, because it is on the index that is partitioned, but not in the columns, so a frame isn't transposable (i.e., rows and columns are not interchangeable).
Why don't we give the user a warning when such a situation occurs as well as make a note in the documentation?
Mmm, as I said, if you have a column `'a'` acting as index, it's because that's one of the original columns of your dataframe. So once it is the index, doing `.set_index('a')` on that frame makes sense it is a no-op (recall that `.set_index()` requires computing quantiles and shuffling, which is expensive, so it should be done with care).

If you do have another column in the frame with that same name, it shows some error on your side. Because that wouldn't be possible from the beginning: if you load your frame from an SQL table, there cannot be any columns with the same name, same goes for Parquet files, etc.

So you did something between loading your data and calling `.set_index()` that introduced a duplicate column. In the snippet above it is `.rename()`. Why would you do such a thing? I cannot think of any good reason...

And it's not only `.set_index()` the one affected. What would you expect the output from `.reset_index()['a']` to be? A series with the renamed column? A series with what used to be the index? A frame with both columns, both named `'a'`?

This is why Dask should always prevent you from introducing some duplicate column name.
I agree in principle with @jorloplaz that this is an odd behavior and that duplicate column names are never a good idea. However since they are still possible to create, it makes sense to at least give a warning in this case. 

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -4809,12 +4809,18 @@ def set_index(
         if isinstance(other, Series):
             # If it's already the index, there's nothing to do
             if other._name == self.index._name:
+                warnings.warn(
+                    "New index has same name as existing, this is a no-op.", UserWarning
+                )
                 return self
 
         # If the name of a column/index
         else:
             # With the same name as the index, there's nothing to do either
             if other == self.index.name:
+                warnings.warn(
+                    "New index has same name as existing, this is a no-op.", UserWarning
+                )
                 return self
 
             # If a missing column, KeyError


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -3724,9 +3724,11 @@ def test_set_index_with_index():
     "Setting the index with the existing index is a no-op"
     df = pd.DataFrame({"x": [1, 2, 3, 4], "y": [1, 0, 1, 0]}).set_index("x")
     ddf = dd.from_pandas(df, npartitions=2)
-    ddf2 = ddf.set_index(ddf.index)
+    with pytest.warns(UserWarning, match="this is a no-op"):
+        ddf2 = ddf.set_index(ddf.index)
     assert ddf2 is ddf
-    ddf = ddf.set_index("x", drop=False)
+    with pytest.warns(UserWarning, match="this is a no-op"):
+        ddf = ddf.set_index("x", drop=False)
     assert ddf2 is ddf
 
 
diff --git a/dask/dataframe/tests/test_shuffle.py b/dask/dataframe/tests/test_shuffle.py
--- a/dask/dataframe/tests/test_shuffle.py
+++ b/dask/dataframe/tests/test_shuffle.py
@@ -193,7 +193,6 @@ def test_set_index_general(npartitions, shuffle_method):
         df.set_index(df.x + df.y), ddf.set_index(ddf.x + ddf.y, shuffle=shuffle_method)
     )
     assert_eq(df.set_index(df.x + 1), ddf.set_index(ddf.x + 1, shuffle=shuffle_method))
-    assert_eq(df.set_index(df.index), ddf.set_index(ddf.index, shuffle=shuffle_method))
 
 
 def test_set_index_self_index(shuffle_method):
@@ -203,7 +202,8 @@ def test_set_index_self_index(shuffle_method):
     )
 
     a = dd.from_pandas(df, npartitions=4)
-    b = a.set_index(a.index, shuffle=shuffle_method)
+    with pytest.warns(UserWarning, match="this is a no-op"):
+        b = a.set_index(a.index, shuffle=shuffle_method)
     assert a is b
 
     assert_eq(b, df.set_index(df.index))