Repository: ibis-project/ibis

Issue:
bug: ibis asof join when left and right table has the same time predicate column, the join result (right table time column result)is incorrect
### What happened?

Below screenshot shows the wrong right table time column result after asof join. I believe the bug only occur when the left and right table join keys are the same.

![image](https://user-images.githubusercontent.com/2775108/234201380-9edf55a5-966c-44ab-a329-efe4931895ba.png)

Below screenshot shows exactly the same code, but when left/right table has different time column, the result looks ok.
![image](https://user-images.githubusercontent.com/2775108/234201591-ef1e11e1-fc22-4800-aa3d-389625c7be12.png)


### What version of ibis are you using?

5.1.0

### What backend(s) are you using, if any?

pandas

### Relevant log output

```sh
N/A
```


### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Hey @zhenzhongxu -- thanks for reporting this and sorry for the delay in response!  You opened this when all of us were traveling and this fell through the cracks a little bit. 

Could you post the text of the screenshots you have above so it's easier for us to cut and paste?

Thank you!
@gforsyth thanks for looking into this. Here you go:
```
import pandas as pd

import ibis
print(f"Ibis version: {ibis.__version__}")

l_time_col="time"
r_time_col="time_different_col"

left_df = pd.DataFrame({
    l_time_col: pd.to_datetime(['2023-04-16 9:30:00', '2023-04-16 10:30:00', '2023-04-16 11:30:00', '2023-04-16 12:30:00']),
    'cc_num': [1, 1, 2, 3]
})
right_df = pd.DataFrame({
    r_time_col: pd.to_datetime(['2023-04-16 9:00:00', '2023-04-16 10:00:00', '2023-04-16 11:00:00', '2023-04-16 12:00:00']),
    'value1': [10.0, 20.0, 30.0, 40.0],
    'cc_num': [1, 1, 2, 3]
})

backend = ibis.pandas.connect({"left_df": left_df, "right_df": right_df})
l_expr = backend.table("left_df")
r_expr = backend.table("right_df")

expr = l_expr.asof_join(
    r_expr, 
    [(l_time_col, r_time_col)], 
    by=["cc_num"]
    )[l_expr, r_expr[r_time_col].name("time_right_tbl"), r_expr.value1]

expr.execute()
```
Hey @zhenzhongxu -- so part of this is due to Pandas behavior in [`merge_asof`](https://pandas.pydata.org/docs/reference/api/pandas.merge_asof.html)

```python
[ins] In [7]: left_df = pd.DataFrame(
         ...:     {
         ...:         "time": pd.to_datetime(
         ...:             [
         ...:                 "2023-04-16 9:30:00",
         ...:                 "2023-04-16 10:30:00",
         ...:                 "2023-04-16 11:30:00",
         ...:                 "2023-04-16 12:30:00",
         ...:             ]
         ...:         ),
         ...:         "cc_num": [1, 1, 2, 3],
         ...:     }
         ...: )
         ...: right_df = pd.DataFrame(
         ...:     {
         ...:         "time": pd.to_datetime(
         ...:             [
         ...:                 "2023-04-16 9:00:00",
         ...:                 "2023-04-16 10:00:00",
         ...:                 "2023-04-16 11:00:00",
         ...:                 "2023-04-16 12:00:00",
         ...:             ]
         ...:         ),
         ...:         "value1": [10.0, 20.0, 30.0, 40.0],
         ...:         "cc_num": [1, 1, 2, 3],
         ...:     }
         ...: )

[ins] In [8]: pd.merge_asof(
         ...:     left_df,
         ...:     right_df,
         ...:     left_on="time",
         ...:     right_on="time",
         ...:     left_by="cc_num",
         ...:     right_by="cc_num",
         ...: )
Out[8]: 
                 time  cc_num  value1
0 2023-04-16 09:30:00       1    10.0
1 2023-04-16 10:30:00       1    20.0
2 2023-04-16 11:30:00       2    30.0
3 2023-04-16 12:30:00       3    40.0

[ins] In [9]: left_df = pd.DataFrame(
         ...:     {
         ...:         "time": pd.to_datetime(
         ...:             [
         ...:                 "2023-04-16 9:30:00",
         ...:                 "2023-04-16 10:30:00",
         ...:                 "2023-04-16 11:30:00",
         ...:                 "2023-04-16 12:30:00",
         ...:             ]
         ...:         ),
         ...:         "cc_num": [1, 1, 2, 3],
         ...:     }
         ...: )
         ...: right_df = pd.DataFrame(
         ...:     {
         ...:         "time_different_col": pd.to_datetime(
         ...:             [
         ...:                 "2023-04-16 9:00:00",
         ...:                 "2023-04-16 10:00:00",
         ...:                 "2023-04-16 11:00:00",
         ...:                 "2023-04-16 12:00:00",
         ...:             ]
         ...:         ),
         ...:         "value1": [10.0, 20.0, 30.0, 40.0],
         ...:         "cc_num": [1, 1, 2, 3],
         ...:     }
         ...: )

[ins] In [10]: pd.merge_asof(
          ...:     left_df,
          ...:     right_df,
          ...:     left_on="time",
          ...:     right_on="time_different_col",
          ...:     left_by="cc_num",
          ...:     right_by="cc_num",
          ...: )
Out[10]: 
                 time  cc_num  time_different_col  value1
0 2023-04-16 09:30:00       1 2023-04-16 09:00:00    10.0
1 2023-04-16 10:30:00       1 2023-04-16 10:00:00    20.0
2 2023-04-16 11:30:00       2 2023-04-16 11:00:00    30.0
3 2023-04-16 12:30:00       3 2023-04-16 12:00:00    40.0
```

I'm not super familiar with the pandas backend, so I'm tracking down where we reconstitute the `time_right` column for the case when the column names are the same.  We could possibly remove that column when the two `time` columns share the same name, but I don't think we can include the original RHS time column if pandas isn't returning it (unless there's sufficient information in the asof_join result to then perform a subsequent join, but that seems like a bad idea).
Well, that's a deep :rabbit: :hole:

After the pandas asofjoin, we end up in `ibis/backends/pandas/execution/selection.py::build_df_from_projection()` which then calls `compute_projection` for every selection expression, which extracts each column individually from some interim result.

That interim `result` is the output of `pd.merge_asof` and the original content of the RHS time column is already lost.  `compute_projection` creates a duplicate of the LHS time column via the `remap_overlapping_column_names` method and then somewhere later on we tack on the usual join suffix for colliding column names (`_right`).

I see a few possible paths forward:
1. Close this as a `wontfix` because of the upstream behavior
2. Warn / error if the asof_join predicate column has a name collision because of this unexpected behavior
3. Mimic `pandas` behavior and drop the RHS time column if there's a name collision.
I've confirmed that this bug / behavior goes back as far as ibis 3.2 (probably further, but those are the environments I have set up right now), so this was not introduced during the removal of intermediate ops (or any of the other subsequent large internals refactors)
@gforsyth, thanks for looking into this! Super detailed investigation, now I understand why the behavior happens. 

Regarding the three options you proposed, I personally would vote to start with option 2 to address the user confusion. However, to address the root concern, besides  option 3, there may be another potential option:

4. Whenever there is a collision on the time columns between LHS and RHS, Ibis internally always renames the two columns by adding a postfix (users can override but would default to something "_left" and "_right", just as an example). When user chooses to provide the same postfix for both LHS and RHS, then we can revert back to the behavior outlined in option 3. From my impression, this approach seems to be consistent with some other existing implementations in ibis. 

Between options 3 and 4, I personally prefer 4 better because it's more explicit. 

Resolving patch:
diff --git a/ibis/backends/pandas/execution/join.py b/ibis/backends/pandas/execution/join.py
--- a/ibis/backends/pandas/execution/join.py
+++ b/ibis/backends/pandas/execution/join.py
@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+import itertools
+
 import pandas as pd
 
 import ibis.expr.analysis as an
@@ -130,13 +132,39 @@ def execute_asof_join(op, left, right, by, tolerance, predicates, **kwargs):
     left_on, right_on = _extract_predicate_names(predicates)
     left_by, right_by = _extract_predicate_names(by)
 
+    # Add default join suffixes to predicates and groups and rename the
+    # corresponding columns before the `merge_asof`. If we don't do this and the
+    # predicates have the same column name, we lose the original RHS column
+    # values in the output. Instead, the RHS values are copies of the LHS values.
+    # xref https://github.com/ibis-project/ibis/issues/6080
+    left_on_suffixed = [x + constants.JOIN_SUFFIXES[0] for x in left_on]
+    right_on_suffixed = [x + constants.JOIN_SUFFIXES[1] for x in right_on]
+
+    left_by_suffixed = [x + constants.JOIN_SUFFIXES[0] for x in left_by]
+    right_by_suffixed = [x + constants.JOIN_SUFFIXES[1] for x in right_by]
+
+    left = left.rename(
+        columns=dict(
+            itertools.chain(
+                zip(left_on, left_on_suffixed), zip(left_by, left_by_suffixed)
+            )
+        )
+    )
+    right = right.rename(
+        columns=dict(
+            itertools.chain(
+                zip(right_on, right_on_suffixed), zip(right_by, right_by_suffixed)
+            )
+        )
+    )
+
     return pd.merge_asof(
         left=left,
         right=right,
-        left_on=left_on,
-        right_on=right_on,
-        left_by=left_by or None,
-        right_by=right_by or None,
+        left_on=left_on_suffixed,
+        right_on=right_on_suffixed,
+        left_by=left_by_suffixed or None,
+        right_by=right_by_suffixed or None,
         tolerance=tolerance,
         suffixes=constants.JOIN_SUFFIXES,
     )


Test patch:
diff --git a/ibis/backends/pandas/tests/execution/test_join.py b/ibis/backends/pandas/tests/execution/test_join.py
--- a/ibis/backends/pandas/tests/execution/test_join.py
+++ b/ibis/backends/pandas/tests/execution/test_join.py
@@ -256,32 +256,36 @@ def test_join_with_window_function(players_base, players_df, batting, batting_df
 
 @merge_asof_minversion
 def test_asof_join(time_left, time_right, time_df1, time_df2):
-    expr = time_left.asof_join(time_right, 'time')[time_left, time_right.other_value]
+    expr = time_left.asof_join(time_right, 'time')
     result = expr.execute()
     expected = pd.merge_asof(time_df1, time_df2, on='time')
     tm.assert_frame_equal(result[expected.columns], expected)
+    with pytest.raises(AssertionError):
+        tm.assert_series_equal(result["time"], result["time_right"])
 
 
 @merge_asof_minversion
 def test_asof_join_predicate(time_left, time_right, time_df1, time_df2):
-    expr = time_left.asof_join(time_right, time_left.time == time_right.time)[
-        time_left, time_right.other_value
-    ]
+    expr = time_left.asof_join(time_right, time_left.time == time_right.time)
     result = expr.execute()
     expected = pd.merge_asof(time_df1, time_df2, on='time')
     tm.assert_frame_equal(result[expected.columns], expected)
+    with pytest.raises(AssertionError):
+        tm.assert_series_equal(result["time"], result["time_right"])
 
 
 @merge_asof_minversion
 def test_keyed_asof_join(
     time_keyed_left, time_keyed_right, time_keyed_df1, time_keyed_df2
 ):
-    expr = time_keyed_left.asof_join(time_keyed_right, 'time', by='key')[
-        time_keyed_left, time_keyed_right.other_value
-    ]
+    expr = time_keyed_left.asof_join(time_keyed_right, 'time', by='key')
     result = expr.execute()
     expected = pd.merge_asof(time_keyed_df1, time_keyed_df2, on='time', by='key')
     tm.assert_frame_equal(result[expected.columns], expected)
+    with pytest.raises(AssertionError):
+        tm.assert_series_equal(result["time"], result["time_right"])
+    with pytest.raises(AssertionError):
+        tm.assert_series_equal(result["key"], result["key_right"])
 
 
 @merge_asof_minversion
@@ -290,7 +294,7 @@ def test_keyed_asof_join_with_tolerance(
 ):
     expr = time_keyed_left.asof_join(
         time_keyed_right, 'time', by='key', tolerance=2 * ibis.interval(days=1)
-    )[time_keyed_left, time_keyed_right.other_value]
+    )
     result = expr.execute()
     expected = pd.merge_asof(
         time_keyed_df1,
@@ -300,6 +304,10 @@ def test_keyed_asof_join_with_tolerance(
         tolerance=pd.Timedelta('2D'),
     )
     tm.assert_frame_equal(result[expected.columns], expected)
+    with pytest.raises(AssertionError):
+        tm.assert_series_equal(result["time"], result["time_right"])
+    with pytest.raises(AssertionError):
+        tm.assert_series_equal(result["key"], result["key_right"])
 
 
 @merge_asof_minversion