Repository: databricks/koalas

Issue:
join/merge problems
I'm trying to merge 2 kdf that actually are subsets of the same kdf.
The main kdf contains all the log events of the all _transport_order_number_ : 
E.g.,:  transport_order_number== 696530708053


```python
kdf=ks.DataFrame({'transport_order_number': {11059585: ('696530708053'),  36538499: '696530708053',  41914814: '696530708053',  58878846: '696530708053',  83502171: '696530708053',  87335732: '696530708053',  89651819: '696530708053'},
 'event_description': {11059585: 'PIEZA EN RUTA AL DESTINATARIO',  36538499: 'TRANSFERENCIA RUTA (OTBCS)',  41914814: 'RECEPCION TRANS. PIEZA',  58878846: 'RETIRO DESDE PDT',  83502171: 'RECEPCION TRANS. CONT.',  87335732: 'RECEPCIONADA',  89651819: 'PIEZA ENTREGADA A DESTINATARIO'},
 'event_date': {11059585: ('2020-12-15 09:05:12.743000'),  36538499: ('2020-12-15 06:42:22.477000'),  41914814: ('2020-12-15 06:42:34.083000'),  58878846: ('2020-12-14 13:41:00'),  83502171: ('2020-12-15 06:42:00'),  87335732: ('2020-12-14 14:41:00'),
  89651819: ('2020-12-15 12:53:00')}})
```

so I made  two subset using `ks.loc[]` to get just the 'RECEPCIONADA' AND 'RETIRO DESDE PDT ' events with: 

```python
recepcion=kdf.loc[kdf['event_description']=='RECEPCIONADA']
```
![image](https://user-images.githubusercontent.com/78232922/108151825-5bb6d680-70b6-11eb-8b73-9bd61a8dfffb.png)

```python
retiro=kdf.loc[kdf['event_description']=='RETIRO DESDE PDT']
```
![image](https://user-images.githubusercontent.com/78232922/108151840-64a7a800-70b6-11eb-9ed8-3251e296df27.png)


Then when I try to marge this 2 subset (recepcion & retiro) using 'how=outer' I got only NaN values ...
```python
ks.merge(fecha_recepcion, fecha_retiro, on='transport_order_number', how='outer', suffixes=('_recepcion','_retiro'))
```

![image](https://user-images.githubusercontent.com/78232922/108010455-ab789d80-6fe3-11eb-8516-0d378db928b5.png)


Using Pandas I got the expected results

Pandas DF:
```python
pd.merge(kdf1, kdf2, on='transport_order_number', how='outer')
```
![image](https://user-images.githubusercontent.com/78232922/108010515-d105a700-6fe3-11eb-8640-177ba23afed9.png)






Discussion:
In order to avoid this problem, I had to recall the data 2 times to create the subsets.

```Python
A_kdf=ks.DataFrame({'transport_order_number': {11059585: ('696530708053'), 36538499: '696530708053', 41914814: '696530708053', 58878846: '696530708053', 83502171: '696530708053', 87335732: '696530708053', 89651819: '696530708053'}, 'event_description': {11059585: 'PIEZA EN RUTA AL DESTINATARIO', 36538499: 'TRANSFERENCIA RUTA (OTBCS)', 41914814: 'RECEPCION TRANS. PIEZA', 58878846: 'RETIRO DESDE PDT', 83502171: 'RECEPCION TRANS. CONT.', 87335732: 'RECEPCIONADA', 89651819: 'PIEZA ENTREGADA A DESTINATARIO'}, 'event_date': {11059585: ('2020-12-15 09:05:12.743000'), 36538499: ('2020-12-15 06:42:22.477000'), 41914814: ('2020-12-15 06:42:34.083000'), 58878846: ('2020-12-14 13:41:00'), 83502171: ('2020-12-15 06:42:00'), 87335732: ('2020-12-14 14:41:00'), 89651819: ('2020-12-15 12:53:00')}})

B_kdf=ks.DataFrame({'transport_order_number': {11059585: ('696530708053'), 36538499: '696530708053', 41914814: '696530708053', 58878846: '696530708053', 83502171: '696530708053', 87335732: '696530708053', 89651819: '696530708053'}, 'event_description': {11059585: 'PIEZA EN RUTA AL DESTINATARIO', 36538499: 'TRANSFERENCIA RUTA (OTBCS)', 41914814: 'RECEPCION TRANS. PIEZA', 58878846: 'RETIRO DESDE PDT', 83502171: 'RECEPCION TRANS. CONT.', 87335732: 'RECEPCIONADA', 89651819: 'PIEZA ENTREGADA A DESTINATARIO'}, 'event_date': {11059585: ('2020-12-15 09:05:12.743000'), 36538499: ('2020-12-15 06:42:22.477000'), 41914814: ('2020-12-15 06:42:34.083000'), 58878846: ('2020-12-14 13:41:00'), 83502171: ('2020-12-15 06:42:00'), 87335732: ('2020-12-14 14:41:00'), 89651819: ('2020-12-15 12:53:00')}})

A_kdf_retiro=A_kdf.loc[A_kdf['event_description']=='RETIRO DESDE PDT']
B_kdf_recepcion=B_kdf.loc[B_kdf['event_description']=='RECEPCIONADA']

B_kdf_recepcion.merge(A_kdf_retiro, on='transport_order_number', how='outer')
```


![image](https://user-images.githubusercontent.com/78232922/108154235-5019de80-70bb-11eb-9d87-c6d7a9908730.png)


Sorry, could you clarify what the problem is and what the expected result is?

I tried what you described:

```py
>>> kdf=ks.DataFrame({'transport_order_number': {11059585: ('696530708053'),  36538499: '696530708053',  41914814: '696530708053',  58878846: '696530708053',  83502171: '696530708053',  87335732: '696530708053',  89651819: '696530708053'},
...  'event_description': {11059585: 'PIEZA EN RUTA AL DESTINATARIO',  36538499: 'TRANSFERENCIA RUTA (OTBCS)',  41914814: 'RECEPCION TRANS. PIEZA',  58878846: 'RETIRO DESDE PDT',  83502171: 'RECEPCION TRANS. CONT.',  87335732: 'RECEPCIONADA',  89651819: 'PIEZA ENTREGADA A DESTINATARIO'},
...  'event_date': {11059585: ('2020-12-15 09:05:12.743000'),  36538499: ('2020-12-15 06:42:22.477000'),  41914814: ('2020-12-15 06:42:34.083000'),  58878846: ('2020-12-14 13:41:00'),  83502171: ('2020-12-15 06:42:00'),  87335732: ('2020-12-14 14:41:00'),
...   89651819: ('2020-12-15 12:53:00')}})
>>> kdf
         transport_order_number               event_description                  event_date
11059585           696530708053   PIEZA EN RUTA AL DESTINATARIO  2020-12-15 09:05:12.743000
36538499           696530708053      TRANSFERENCIA RUTA (OTBCS)  2020-12-15 06:42:22.477000
41914814           696530708053          RECEPCION TRANS. PIEZA  2020-12-15 06:42:34.083000
58878846           696530708053                RETIRO DESDE PDT         2020-12-14 13:41:00
83502171           696530708053          RECEPCION TRANS. CONT.         2020-12-15 06:42:00
87335732           696530708053                    RECEPCIONADA         2020-12-14 14:41:00
89651819           696530708053  PIEZA ENTREGADA A DESTINATARIO         2020-12-15 12:53:00

>>> recepcion=kdf.loc[kdf['event_description']=='RECEPCIONADA']
>>> recepcion
         transport_order_number event_description           event_date
87335732           696530708053      RECEPCIONADA  2020-12-14 14:41:00

>>> retiro=kdf.loc[kdf['event_description']=='RETIRO DESDE PDT']
>>> retiro
         transport_order_number event_description           event_date
58878846           696530708053  RETIRO DESDE PDT  2020-12-14 13:41:00

>>> ks.merge(recepcion, retiro, on='transport_order_number', how='outer', suffixes=('_recepcion','_retiro'))
  transport_order_number event_description_recepcion event_date_recepcion event_description_retiro    event_date_retiro
0           696530708053                RECEPCIONADA  2020-12-14 14:41:00             RECEPCIONADA  2020-12-14 14:41:00
```

or from the pandas' example:

```py
>>> df1 = ks.DataFrame({'transport_order_number': ['696530708053', '696530708055'], 'event_description': ['RECEPCIONADA', 'RECEPCIONADA'], 'event_date': ['2020-01-02', '2020-01-02']})
>>> df1
  transport_order_number event_description  event_date
0           696530708053      RECEPCIONADA  2020-01-02
1           696530708055      RECEPCIONADA  2020-01-02
>>> df2 = ks.DataFrame({'transport_order_number': ['696530708053', '696530708056'], 'event_description': ['RETIRO DESDE PDT', 'RETIRO DESDE PDT'], 'event_date': ['2020-01-03', '2020-01-03']})
>>> df2
  transport_order_number event_description  event_date
0           696530708053  RETIRO DESDE PDT  2020-01-03
1           696530708056  RETIRO DESDE PDT  2020-01-03

>>> ks.merge(df1, df2, on='transport_order_number', how='outer').sort_values('transport_order_number')
  transport_order_number event_description_x event_date_x event_description_y event_date_y
2           696530708053        RECEPCIONADA   2020-01-02    RETIRO DESDE PDT   2020-01-03
0           696530708055        RECEPCIONADA   2020-01-02                None         None
1           696530708056                None         None    RETIRO DESDE PDT   2020-01-03
```

This seems the same as the pandas' example result?
Ho @ueshin,
Don't you see a litle strange that the ks.merge return the same event ( RECEPCIONADA -  RECEPCIONADA ) on both sides, when it shouldn't be possible 'cause df1 and df2 were filtered for different "events"  (RECEPCIONADA - RETIRO DESDE PDT, respectively).

![image](https://user-images.githubusercontent.com/78232922/108449169-ca746b00-7241-11eb-9365-c800efdb6fbf.png)

Ah, I see what you mean! Yeah, that's right, it's weird.. Let us investigate it.
Thanks for the report!

Resolving patch:
diff --git a/databricks/koalas/frame.py b/databricks/koalas/frame.py
--- a/databricks/koalas/frame.py
+++ b/databricks/koalas/frame.py
@@ -7260,10 +7260,37 @@ def to_list(os: Optional[Union[Any, List[Any], Tuple, List[Tuple]]]) -> List[Tup
             if len(left_key_names) != len(right_key_names):
                 raise ValueError("len(left_keys) must equal len(right_keys)")
 
+        # We should distinguish the name to avoid ambiguous column name after merging.
+        right_prefix = "__right_"
+        right_key_names = [right_prefix + right_key_name for right_key_name in right_key_names]
+
         how = validate_how(how)
 
+        def resolve(internal, side):
+            rename = lambda col: "__{}_{}".format(side, col)
+            internal = internal.resolved_copy
+            sdf = internal.spark_frame
+            sdf = internal.spark_frame.select(
+                [
+                    scol_for(sdf, col).alias(rename(col))
+                    for col in sdf.columns
+                    if col not in HIDDEN_COLUMNS
+                ]
+                + list(HIDDEN_COLUMNS)
+            )
+            return internal.copy(
+                spark_frame=sdf,
+                index_spark_columns=[
+                    scol_for(sdf, rename(col)) for col in internal.index_spark_column_names
+                ],
+                data_spark_columns=[
+                    scol_for(sdf, rename(col)) for col in internal.data_spark_column_names
+                ],
+                preserve_dtypes=True,
+            )
+
         left_internal = self._internal.resolved_copy
-        right_internal = right._internal.resolved_copy
+        right_internal = resolve(right._internal, "right")
 
         left_table = left_internal.spark_frame.alias("left_table")
         right_table = right_internal.spark_frame.alias("right_table")
@@ -7301,10 +7328,13 @@ def to_list(os: Optional[Union[Any, List[Any], Tuple, List[Tuple]]]) -> List[Tup
             scol = left_scol_for(label)
             if label in duplicate_columns:
                 spark_column_name = left_internal.spark_column_name_for(label)
-                if spark_column_name in left_key_names and spark_column_name in right_key_names:
+                if (
+                    spark_column_name in left_key_names
+                    and (right_prefix + spark_column_name) in right_key_names
+                ):
                     right_scol = right_scol_for(label)
                     if how == "right":
-                        scol = right_scol
+                        scol = right_scol.alias(col)
                     elif how == "full":
                         scol = F.when(scol.isNotNull(), scol).otherwise(right_scol).alias(col)
                     else:
@@ -7317,11 +7347,15 @@ def to_list(os: Optional[Union[Any, List[Any], Tuple, List[Tuple]]]) -> List[Tup
             data_columns.append(col)
             column_labels.append(label)
         for label in right_internal.column_labels:
-            col = right_internal.spark_column_name_for(label)
-            scol = right_scol_for(label)
+            # recover `right_prefix` here.
+            col = right_internal.spark_column_name_for(label)[len(right_prefix) :]
+            scol = right_scol_for(label).alias(col)
             if label in duplicate_columns:
                 spark_column_name = left_internal.spark_column_name_for(label)
-                if spark_column_name in left_key_names and spark_column_name in right_key_names:
+                if (
+                    spark_column_name in left_key_names
+                    and (right_prefix + spark_column_name) in right_key_names
+                ):
                     continue
                 else:
                     col = col + right_suffix


Test patch:
diff --git a/databricks/koalas/tests/test_dataframe.py b/databricks/koalas/tests/test_dataframe.py
--- a/databricks/koalas/tests/test_dataframe.py
+++ b/databricks/koalas/tests/test_dataframe.py
@@ -2030,6 +2030,54 @@ def check(op, right_kdf=right_kdf, right_pdf=right_pdf):
             )
         )
 
+    def test_merge_same_anchor(self):
+        pdf = pd.DataFrame(
+            {
+                "lkey": ["foo", "bar", "baz", "foo", "bar", "l"],
+                "rkey": ["baz", "foo", "bar", "baz", "foo", "r"],
+                "value": [1, 1, 3, 5, 6, 7],
+                "x": list("abcdef"),
+                "y": list("efghij"),
+            },
+            columns=["lkey", "rkey", "value", "x", "y"],
+        )
+        kdf = ks.from_pandas(pdf)
+
+        left_pdf = pdf[["lkey", "value", "x"]]
+        right_pdf = pdf[["rkey", "value", "y"]]
+        left_kdf = kdf[["lkey", "value", "x"]]
+        right_kdf = kdf[["rkey", "value", "y"]]
+
+        def check(op, right_kdf=right_kdf, right_pdf=right_pdf):
+            k_res = op(left_kdf, right_kdf)
+            k_res = k_res.to_pandas()
+            k_res = k_res.sort_values(by=list(k_res.columns))
+            k_res = k_res.reset_index(drop=True)
+            p_res = op(left_pdf, right_pdf)
+            p_res = p_res.sort_values(by=list(p_res.columns))
+            p_res = p_res.reset_index(drop=True)
+            self.assert_eq(k_res, p_res)
+
+        check(lambda left, right: left.merge(right))
+        check(lambda left, right: left.merge(right, on="value"))
+        check(lambda left, right: left.merge(right, left_on="lkey", right_on="rkey"))
+        check(lambda left, right: left.set_index("lkey").merge(right.set_index("rkey")))
+        check(
+            lambda left, right: left.set_index("lkey").merge(
+                right, left_index=True, right_on="rkey"
+            )
+        )
+        check(
+            lambda left, right: left.merge(
+                right.set_index("rkey"), left_on="lkey", right_index=True
+            )
+        )
+        check(
+            lambda left, right: left.set_index("lkey").merge(
+                right.set_index("rkey"), left_index=True, right_index=True
+            )
+        )
+
     def test_merge_retains_indices(self):
         left_pdf = pd.DataFrame({"A": [0, 1]})
         right_pdf = pd.DataFrame({"B": [1, 2]}, index=[1, 2])