Repository: modin-project/modin

Issue:
Allow set_index to take an object to be handled by the backend (a backend index, etc)
As discussed in #1780, the set_index API assumes that we either have a set of column names/objects and that the set of columns passed in matches the set of known columns. We then call set_index_from_columns to let the backend handle the operation. A problem occurs when we need to call set_index with a backend type which the modin layer does not know about:

```
df.set_index(MyBackendIndex)
```

`MyBackendIndex` is not a normal Pandas type, nor a type which modin knows about, nor an actual column in the dataframe. In this case we want the backend to be responsible for as much of the handling here as possible. We *also* want to retain the nice error handling already present in set_index to get for missing columns.

One solution I have is to get to see if keys is non-"primitive" and when it is considered missing from the columns we raising an error and throw it over the wall to the query_compiler. Another method might be to agree on an attribute or method set on each backend which can tell the dataframe set_index method to circumvent the column checks.

*dataframe.set_index* approach:
```
-        if missing:
+        # If the missing column is a "primitive", return the errors.
+        # Otherwise we let the query compiler figure out what to do with
+        # the keys
+        if missing and not hasattr(missing[0], "__dict__"):
+            # The keys are a primitive type
             raise KeyError(f"None of {missing} are in the columns")
```
 
https://github.com/modin-project/modin/issues/1780


Discussion:

Resolving patch:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -1967,7 +1967,11 @@ def set_index(
             else:
                 if not found:
                     missing.append(col)
-        if missing:
+        # If the missing column is a "primitive", return the errors.
+        # Otherwise we let the query compiler figure out what to do with
+        # the keys
+        if missing and not hasattr(missing[0], "__dict__"):
+            # The keys are a primitive type
             raise KeyError(f"None of {missing} are in the columns")
 
         new_query_compiler = self._query_compiler.set_index_from_columns(


Test patch:
diff --git a/modin/pandas/test/dataframe/test_indexing.py b/modin/pandas/test/dataframe/test_indexing.py
--- a/modin/pandas/test/dataframe/test_indexing.py
+++ b/modin/pandas/test/dataframe/test_indexing.py
@@ -1232,6 +1232,23 @@ def test_rename_bug():
     df_equals(modin_df, df)
 
 
+def test_index_to_datetime_using_set_index():
+    data = {"YEAR": ["1992", "1993", "1994"], "ALIENS": [1, 99, 1]}
+    modin_df_years = pd.DataFrame(data=data)
+    df_years = pandas.DataFrame(data=data)
+    modin_df_years = modin_df_years.set_index("YEAR")
+    df_years = df_years.set_index("YEAR")
+    modin_datetime_index = pd.to_datetime(modin_df_years.index, format="%Y")
+    pandas_datetime_index = pandas.to_datetime(df_years.index, format="%Y")
+
+    modin_df_years.index = modin_datetime_index
+    df_years.index = pandas_datetime_index
+
+    modin_df_years.set_index(modin_datetime_index)
+    df_years.set_index(pandas_datetime_index)
+    df_equals(modin_df_years, df_years)
+
+
 def test_rename_axis():
     data = {"num_legs": [4, 4, 2], "num_arms": [0, 0, 2]}
     index = ["dog", "cat", "monkey"]