Bug: Inconsistent Behavior Between PySpark DataFrame API's `eqNullSafe` Join and SQL Null-Safe Equality `<=>`
Status: Open | Priority: Trivial | Resolution: 
Created: 09/Sep/24 20:21
Affects Version/s: 3.3.0

h3. Summary:
h3. 
When performing a self inner join with null safety in PySpark using the `eqNullSafe` method, the resulting output is inconsistent with the equivalent SQL query using the null-safe equality operator `<=>`. Specifically, the results of the join differ, as the DataFrame API produces columns containing wrong `NULL` values compared to the SQL-based approach.

h3. Steps to Reproduce:
h3. 
The following code reproduces the issue:


{code:python}
from datetime import date, datetime
import pyspark.sql.functions as f
from pyspark.sql import SparkSession

spark = SparkSession.Builder().getOrCreate()

x = spark.createDataFrame(
    [
        (1, "A", date(2021, 1, 1), datetime(2021, 1, 1, 10), 20.5, True),
        (None, None, None, None, None, None),
        (2, "B", date(2021, 1, 2), datetime(2021, 1, 2, 12), 10.8, False),
    ],
    schema="id int, name string, init date, init_t timestamp, amount double, is_active boolean",
)

# Self inner join using PySpark DataFrame API with null-safe equality
x.alias("x1").join(
    x.alias("x2"), how="inner", on=f.col("x1.id").eqNullSafe(f.col("x2.id"))
).show()

# Self inner join using SQL null-safe equality
x.createOrReplaceTempView("x")

spark.sql(
    """
    select * 
    from x x1
    inner join x x2 on x1.id <=> x2.id
    """
).show()
{code}


h3. Expected Behavior:
h3. 
The results from both the PySpark DataFrame API (`eqNullSafe`) and the SQL-based (`<=>`) null-safe equality join should match. Specifically, both joins should handle `NULL` values in the same way, returning identical rows.

For the dataset provided above, both joins are expected to output:


{noformat}
+----+----+----------+-------------------+------+---------+----+----+----------+-------------------+------+---------+
|  id|name|      init|             init_t|amount|is_active|  id|name|      init|             init_t|amount|is_active|
+----+----+----------+-------------------+------+---------+----+----+----------+-------------------+------+---------+
|null|null|      null|               null|  null|     null|null|null|      null|               null|  null|     null|
|   1|   A|2021-01-01|2021-01-01 10:00:00|  20.5|     true|   1|   A|2021-01-01|2021-01-01 10:00:00|  20.5|     true|
|   2|   B|2021-01-02|2021-01-02 12:00:00|  10.8|    false|   2|   B|2021-01-02|2021-01-02 12:00:00|  10.8|    false|
+----+----+----------+-------------------+------+---------+----+----+----------+-------------------+------+---------+
{noformat}


h3. Actual Behavior:
h3. 
The PySpark DataFrame API's `eqNullSafe` join produces unexpected additional rows with `NULL` values, which are absent in the SQL-based join:

*PySpark DataFrame API (`eqNullSafe`) result:*

{noformat}
+----+----+----+------+------+---------+----+----+----------+-------------------+------+---------+
|  id|name|init|init_t|amount|is_active|  id|name|      init|             init_t|amount|is_active|
+----+----+----+------+------+---------+----+----+----------+-------------------+------+---------+
|null|null|null|  null|  null|     null|   1|   A|2021-01-01|2021-01-01 10:00:00|  20.5|     true|
|null|null|null|  null|  null|     null|null|null|      null|               null|  null|     null|
|null|null|null|  null|  null|     null|   2|   B|2021-01-02|2021-01-02 12:00:00|  10.8|    false|
+----+----+----+------+------+---------+----+----+----------+-------------------+------+---------+
{noformat}

*SQL-based (`<=>`) result:*

{noformat}
+----+----+----------+-------------------+------+---------+----+----+----------+-------------------+------+---------+
|  id|name|      init|             init_t|amount|is_active|  id|name|      init|             init_t|amount|is_active|
+----+----+----------+-------------------+------+---------+----+----+----------+-------------------+------+---------+
|null|null|      null|               null|  null|     null|null|null|      null|               null|  null|     null|
|   1|   A|2021-01-01|2021-01-01 10:00:00|  20.5|     true|   1|   A|2021-01-01|2021-01-01 10:00:00|  20.5|     true|
|   2|   B|2021-01-02|2021-01-02 12:00:00|  10.8|    false|   2|   B|2021-01-02|2021-01-02 12:00:00|  10.8|    false|
+----+----+----------+-------------------+------+---------+----+----+----------+-------------------+------+---------+
{noformat}

h3. Impact:
h3. 
This inconsistency causes confusion and could lead to incorrect results when performing joins involving `NULL` values in PySpark. Users might expect the same behavior between the PySpark DataFrame API and SQL queries for null-safe equality joins.

h3. Suggested Fix:
h3. 
Ensure that the behavior of `eqNullSafe` in the PySpark DataFrame API matches the behavior of the SQL `<=>` operator, so that both produce consistent results when joining on nullable columns.