Repository: modin-project/modin

Issue:
BUG: Falling back to standard Pandas implementation when assigning a dataframe to a columnar selection
### Modin version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest released version of Modin.

- [X] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-main-branch).)


### Reproducible Example

```python
import sys
import modin
import modin.pandas as pd

data = {
    'A': [1.234, 5.678],
    'B': [1, 2],
    'C': ['one', 'two']
}

df = pd.DataFrame(data)
df_selected = df[['A', 'C']]
df[df_selected.columns] = df_selected
```

### Issue Description

The example outputs the following warning

```
UserWarning: `DataFrame.setitem_unhashable_key` is not currently supported by PandasOnRay, defaulting to pandas implementation.
Please refer to https://modin.readthedocs.io/en/stable/supported_apis/defaulting_to_pandas.html for explanation.
UserWarning: Distributing <class 'pandas.core.frame.DataFrame'> object. This may take some time.
```

The same behavior occurs when using `Dask` engine. 



### Expected Behavior

I would expect to be able to make this assignment without returning the dataset to pandas then back to Modin Dataset, since I'm working with very large datasets and this operation can become very costly (cpu/io wise).

There is no code to show an expected behavior. 

### Error Logs

<details>

```python-traceback

UserWarning: `DataFrame.setitem_unhashable_key` is not currently supported by PandasOnRay, defaulting to pandas implementation.
Please refer to https://modin.readthedocs.io/en/stable/supported_apis/defaulting_to_pandas.html for explanation.
UserWarning: Distributing <class 'pandas.core.frame.DataFrame'> object. This may take some time.
```

</details>


### Installed Versions

INSTALLED VERSIONS
------------------
commit                : e9dbcc127913db77473a83936e8b6bb94ef84f0d
python                : 3.10.13.final.0
python-bits           : 64
OS                    : Linux
OS-release            : 5.10.0-28-cloud-amd64
Version               : #1 SMP Debian 5.10.209-2 (2024-01-31)
machine               : x86_64
processor             : 
byteorder             : little
LC_ALL                : None
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

Modin dependencies
------------------
modin                 : 0.29.0+9.ge9dbcc12
ray                   : 2.9.3
dask                  : 2024.4.1
distributed           : 2024.4.1
hdk                   : None

pandas dependencies
-------------------
pandas                : 2.2.1
numpy                 : 1.25.2
pytz                  : 2024.1
dateutil              : 2.9.0.post0
setuptools            : 69.1.1
pip                   : 24.0
Cython                : None
pytest                : 8.1.1
hypothesis            : None
sphinx                : None
blosc                 : None
feather               : None
xlsxwriter            : None
lxml.etree            : None
html5lib              : None
pymysql               : None
psycopg2              : None
jinja2                : 3.1.3
IPython               : 8.21.0
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.3
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : None
fsspec                : 2024.2.0
gcsfs                 : 2024.2.0
matplotlib            : 3.8.3
numba                 : 0.58.1
numexpr               : None
odfpy                 : None
openpyxl              : None
pandas_gbq            : 0.22.0
pyarrow               : 15.0.1
pyreadstat            : None
python-calamine       : None
pyxlsb                : None
s3fs                  : None
scipy                 : 1.11.4
sqlalchemy            : 2.0.28
tables                : None
tabulate              : None
xarray                : 2024.3.0
xlrd                  : None
zstandard             : 0.22.0
tzdata                : 2024.1
qtpy                  : None
pyqt5                 : None


Discussion:
Hi @cw-igormorgado, thanks for opening this issue! I can confirm that we are defaulting to pandas in this case on latest main. I'll open a PR shortly to add the ability for assigning a dataframe to a columnar selection. Meanwhile, you could convert df_selected.columns to a list as a workaround and the things should work without emiting a warning.

Resolving patch:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
@@ -1511,16 +1511,19 @@ def _join_arrow_columns(self, other_modin_frames):
                 raise NotImplementedError("Duplicate column names")
             max_len = max(len(t) for t in tables)
             columns = [c for t in tables for c in t.columns]
+            new_dtypes = [dt for frame in frames for dt in frame.dtypes]
             # Make all columns of the same length, if required.
             for i, col in enumerate(columns):
                 if len(col) < max_len:
                     columns[i] = pyarrow.chunked_array(
                         col.chunks + [pyarrow.nulls(max_len - len(col), col.type)]
                     )
+                    new_dtypes[i] = arrow_type_to_pandas(columns[i].type)
             return self.from_arrow(
                 at=pyarrow.table(columns, column_names),
                 columns=[c for f in frames for c in f.columns],
                 encode_col_names=False,
+                new_dtypes=new_dtypes,
             )
         return None
 
@@ -2955,7 +2958,11 @@ def from_pandas(cls, df):
             len(new_index) == 0
             and not isinstance(new_index, MultiIndex)
             and new_index.name is None
-        ) or (len(new_columns) != 0 and is_trivial_index(new_index)):
+        ) or (
+            new_index.name is None
+            and len(new_columns) != 0
+            and is_trivial_index(new_index)
+        ):
             index_cols = None
         else:
             orig_index_names = new_index.names
@@ -3009,7 +3016,13 @@ def encoder(n):
 
     @classmethod
     def from_arrow(
-        cls, at, index_cols=None, index=None, columns=None, encode_col_names=True
+        cls,
+        at,
+        index_cols=None,
+        index=None,
+        columns=None,
+        encode_col_names=True,
+        new_dtypes=None,
     ):
         """
         Build a frame from an Arrow table.
@@ -3028,6 +3041,8 @@ def from_arrow(
             Column labels to use for resulting frame.
         encode_col_names : bool, default: True
             Encode column names.
+        new_dtypes : pandas.Index or list, optional
+            Column data types.
 
         Returns
         -------
@@ -3057,20 +3072,21 @@ def from_arrow(
 
         dtype_index = [] if index_cols is None else list(index_cols)
         dtype_index.extend(new_columns)
-        new_dtypes = []
-
-        for col in at.columns:
-            if pyarrow.types.is_dictionary(col.type):
-                new_dtypes.append(
-                    LazyProxyCategoricalDtype._build_proxy(
-                        parent=at,
-                        column_name=col._name,
-                        materializer=build_categorical_from_at,
-                        dtype=arrow_type_to_pandas(col.type.value_type),
+
+        if new_dtypes is None:
+            new_dtypes = []
+            for col in at.columns:
+                if pyarrow.types.is_dictionary(col.type):
+                    new_dtypes.append(
+                        LazyProxyCategoricalDtype._build_proxy(
+                            parent=at,
+                            column_name=col._name,
+                            materializer=build_categorical_from_at,
+                            dtype=arrow_type_to_pandas(col.type.value_type),
+                        )
                     )
-                )
-            else:
-                new_dtypes.append(cls._arrow_type_to_dtype(col.type))
+                else:
+                    new_dtypes.append(cls._arrow_type_to_dtype(col.type))
 
         if len(unsupported_cols) > 0:
             ErrorMessage.single_warning(
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -2671,7 +2671,9 @@ def __setitem__(self, key, value) -> None:
                     key = self.__constructor__(key, columns=self.columns)
                 return self.mask(key, value, inplace=True)
 
-            if isinstance(key, list) and all((x in self.columns for x in key)):
+            if isinstance(key, (list, pandas.Index)) and all(
+                (x in self.columns for x in key)
+            ):
                 if is_list_like(value):
                     if not (hasattr(value, "shape") and hasattr(value, "ndim")):
                         value = np.array(value)


Test patch:
diff --git a/modin/tests/pandas/dataframe/test_indexing.py b/modin/tests/pandas/dataframe/test_indexing.py
--- a/modin/tests/pandas/dataframe/test_indexing.py
+++ b/modin/tests/pandas/dataframe/test_indexing.py
@@ -1771,19 +1771,7 @@ def test_reset_index_with_multi_index_drop(
     )
 
 
-@pytest.mark.parametrize(
-    "test_async_reset_index",
-    [
-        False,
-        pytest.param(
-            True,
-            marks=pytest.mark.xfail(
-                StorageFormat.get() == "Hdk",
-                reason="HDK does not store trivial indexes.",
-            ),
-        ),
-    ],
-)
+@pytest.mark.parametrize("test_async_reset_index", [False, True])
 @pytest.mark.parametrize("index_levels_names_max_levels", [0, 1, 2])
 def test_reset_index_with_named_index(
     index_levels_names_max_levels, test_async_reset_index
@@ -2176,6 +2164,11 @@ def test___setitem__(data):
 
     df_equals(modin_df, pandas_df)
 
+    # Test df assignment to a columns selection
+    modin_df[modin_df.columns[[0, -1]]] = modin_df[modin_df.columns[[0, -1]]]
+    pandas_df[pandas_df.columns[[0, -1]]] = pandas_df[pandas_df.columns[[0, -1]]]
+    df_equals(modin_df, pandas_df)
+
     # Test series assignment to column
     modin_df = pd.DataFrame(columns=modin_df.columns)
     pandas_df = pandas.DataFrame(columns=pandas_df.columns)