Repository: databricks/koalas

Issue:
Fix comparison operators to treat NULL as False
This PR proposes to:
- Fix column comparison operators to treat NULL as False
- Resolve #999.

## pandas
pandas treats NULL as False
```python
>>> pser
0    0.0
1    1.0
2    2.0
3    NaN
dtype: float64

>>> pser == 0
0     True
1    False
2    False
3    False  <- bool
dtype: bool
```

## Koalas
Koalas currently treats NULL as NULL
```python
>>> kser
0    0.0
1    1.0
2    2.0
3    NaN
Name: 0, dtype: float64

# CURRENT
>>> kser == 0
0     True
1    False
2    False
3     None  <- not bool
Name: 0, dtype: object

# PROPOSED
>>> kser == 0
0     True
1    False
2    False
3    False  <- bool
Name: 0, dtype: bool
```


Discussion:
One of the failed doctests. It seems to be working.
```
_________________ [doctest] databricks.koalas.series.Series.eq _________________
532         ...                   index=['a', 'b', 'c', 'd'], columns=['a', 'b'])
533 
534         >>> df.a == 1
535         a     True
536         b    False
537         c    False
538         d    False
539         Name: a, dtype: bool
540 
541         >>> df.b.eq(1)
Differences (unified diff with -expected +actual):
    @@ -1,5 +1,5 @@
    -a    True
    -b    None
    -c    True
    -d    None
    -Name: b, dtype: object
    +a     True
    +b    False
    +c     True
    +d    False
    +Name: b, dtype: bool
```
https://travis-ci.com/databricks/koalas/jobs/254303993#L2298
# [Codecov](https://codecov.io/gh/databricks/koalas/pull/1029?src=pr&el=h1) Report
> Merging [#1029](https://codecov.io/gh/databricks/koalas/pull/1029?src=pr&el=desc) into [master](https://codecov.io/gh/databricks/koalas/commit/42758e68a664c0850ead749fde65759ecbf25047?src=pr&el=desc) will **increase** coverage by `0.13%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/databricks/koalas/pull/1029/graphs/tree.svg?width=650&token=EtnrpE8dVx&height=150&src=pr)](https://codecov.io/gh/databricks/koalas/pull/1029?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1029      +/-   ##
==========================================
+ Coverage   94.92%   95.05%   +0.13%     
==========================================
  Files          34       34              
  Lines        6697     6778      +81     
==========================================
+ Hits         6357     6443      +86     
+ Misses        340      335       -5
```


| [Impacted Files](https://codecov.io/gh/databricks/koalas/pull/1029?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [databricks/koalas/series.py](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvc2VyaWVzLnB5) | `96.41% <ø> (ø)` | :arrow_up: |
| [databricks/koalas/frame.py](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvZnJhbWUucHk=) | `96.5% <ø> (+0.16%)` | :arrow_up: |
| [databricks/koalas/base.py](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvYmFzZS5weQ==) | `95.23% <100%> (+0.33%)` | :arrow_up: |
| [databricks/koalas/\_\_init\_\_.py](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvX19pbml0X18ucHk=) | `85.1% <0%> (-2.13%)` | :arrow_down: |
| [databricks/conftest.py](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9jb25mdGVzdC5weQ==) | `96% <0%> (-2%)` | :arrow_down: |
| [databricks/koalas/indexes.py](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvaW5kZXhlcy5weQ==) | `95.91% <0%> (-0.5%)` | :arrow_down: |
| [databricks/koalas/generic.py](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvZ2VuZXJpYy5weQ==) | `95.26% <0%> (-0.48%)` | :arrow_down: |
| [databricks/koalas/groupby.py](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvZ3JvdXBieS5weQ==) | `91.18% <0%> (-0.21%)` | :arrow_down: |
| [databricks/koalas/missing/indexes.py](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvbWlzc2luZy9pbmRleGVzLnB5) | `100% <0%> (ø)` | :arrow_up: |
| ... and [7 more](https://codecov.io/gh/databricks/koalas/pull/1029/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/databricks/koalas/pull/1029?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/databricks/koalas/pull/1029?src=pr&el=footer). Last update [42758e6...000d3fd](https://codecov.io/gh/databricks/koalas/pull/1029?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

What about https://github.com/databricks/koalas/pull/1029#discussion_r343992874?
Should we also address here? cc @HyukjinKwon 
@HyukjinKwon @ueshin 
This change makes `koalas` behave differently from `pyspark`. Is that really ok?
Actually I'm not quite sure whether we should follow pandas' behavior here.
How about adding a config to switch the null semantics between pandas and SQL.
cc @HyukjinKwon 
https://github.com/databricks/koalas/pull/1029#discussion_r343992874 this behaivour at least looks consistent with Python native comparison.
I think it's fine to fix ... for now ...
@HyukjinKwon 
> I think it's fine to fix ... for now ...

We don't need to fix this weird behavior for now?
I meant let's fix to be consistent with pandas/Python comparison for now since that's basically what Koalas needs to do.
@HyukjinKwon ok, which one should I address, pandas or python?
For `['eq', 'ne', 'lt', 'le', 'ge', 'gt']`, my impression was that both are same. Are they different?
Let's stick to pandas for now if they are different ... maybe we should have a config as @ueshin said but I think it's fine for now ... 
@HyukjinKwon 
> For ['eq', 'ne', 'lt', 'le', 'ge', 'gt'], my impression was that both are same. Are they different?

No, they are same.
Let's stick to pandas for now.
Is `logistic operators` correct name? It should be `comparison operators`, shouldn't it?
https://github.com/databricks/koalas/blob/e469d540d8a14059a2706d8f2fdc4afcca9be747/databricks/koalas/base.py#L185-L190

https://github.com/databricks/koalas/blob/master/databricks/koalas/base.py#L185
Yup, I think so.
Fixed it
### Softagram Impact Report for pull/1029 (head commit: 000d3fdc74fd)


### :star: Change Overview


![Showing the changed files, dependency changes and the impact - click for full size](https://opensource.softagram.com/cdn/impact/dd6f973b-a9da-477d-8f74-1a29916573a5_koalas_1029_impact_change_graph_IDqu2eWqyH82xQZ8cIyzLEllJzVe62.png)
*(Open in [Softagram Desktop](https://softagram.com/desktop) for full details)*






### :bulb: Insights
- Co-change Alert: You modified frame.py. Often test_dataframe.py (koalas/tests) is modified at the same time.


### :page_facing_up: Full report

* Permalink: [Full report for pull/1029](https://opensource.softagram.com/dashboard/teams/7ff2e81c-61e0-4605-b4ac-d496bc4c37c7/projects/dd6f973b-a9da-477d-8f74-1a29916573a5/impact-reports/4859c1d0-fb45-4e80-a660-8b74ec88a947)

#
[Impact Report explained.](http://help.softagram.com/en/articles/2847546-pull-merge-request-report-explained) Give feedback on this report to support@softagram.com

Merged to master.
Thanks, @harupy 
@HyukjinKwon thanks!

Resolving patch:
diff --git a/databricks/koalas/base.py b/databricks/koalas/base.py
--- a/databricks/koalas/base.py
+++ b/databricks/koalas/base.py
@@ -36,6 +36,27 @@
 from databricks.koalas.utils import align_diff_series, scol_for, validate_axis
 
 
+def booleanize_null(left_scol, scol, f):
+    """
+    Booleanize Null in Spark Column
+    """
+    comp_ops = [getattr(spark.Column, '__{}__'.format(comp_op))
+                for comp_op in ['eq', 'ne', 'lt', 'le', 'ge', 'gt']]
+
+    if f in comp_ops:
+        # if `f` is "!=", fill null with True otherwise False
+        filler = f == spark.Column.__ne__
+        scol = F.when(scol.isNull(), filler).otherwise(scol)
+
+    elif f == spark.Column.__or__:
+        scol = F.when(left_scol.isNull() | scol.isNull(), False).otherwise(scol)
+
+    elif f == spark.Column.__and__:
+        scol = F.when(scol.isNull(), False).otherwise(scol)
+
+    return scol
+
+
 def _column_op(f):
     """
     A decorator that wraps APIs taking/returning Spark Column so that Koalas Series can be
@@ -57,27 +78,14 @@ def wrapper(self, *args):
             # Same DataFrame anchors
             args = [arg._scol if isinstance(arg, IndexOpsMixin) else arg for arg in args]
             scol = f(self._scol, *args)
-
-            # check if `f` is a comparison operator
-            comp_ops = ['eq', 'ne', 'lt', 'le', 'ge', 'gt']
-            is_comp_op = any(f == getattr(spark.Column, '__{}__'.format(comp_op))
-                             for comp_op in comp_ops)
-
-            if is_comp_op:
-                filler = f == spark.Column.__ne__
-                scol = F.when(scol.isNull(), filler).otherwise(scol)
-
-            elif f == spark.Column.__or__:
-                scol = F.when(self._scol.isNull() | scol.isNull(), False).otherwise(scol)
-
-            elif f == spark.Column.__and__:
-                scol = F.when(scol.isNull(), False).otherwise(scol)
+            scol = booleanize_null(self._scol, scol, f)
 
             return self._with_new_scol(scol)
         else:
             # Different DataFrame anchors
             def apply_func(this_column, *that_columns):
-                return f(this_column, *that_columns)
+                scol = f(this_column, *that_columns)
+                return booleanize_null(this_column, scol, f)
 
             return align_diff_series(apply_func, self, *args, how="full")
 


Test patch:
diff --git a/databricks/koalas/tests/test_ops_on_diff_frames.py b/databricks/koalas/tests/test_ops_on_diff_frames.py
--- a/databricks/koalas/tests/test_ops_on_diff_frames.py
+++ b/databricks/koalas/tests/test_ops_on_diff_frames.py
@@ -14,6 +14,7 @@
 # limitations under the License.
 #
 import pandas as pd
+import numpy as np
 
 from databricks import koalas as ks
 from databricks.koalas.config import set_option, reset_option
@@ -216,6 +217,23 @@ def test_arithmetic_chain(self):
             (kdf1 + kdf2 - kdf3).sort_index(),
             (pdf1 + pdf2 - pdf3).sort_index(), almost=True)
 
+    def test_bitwise(self):
+        pser1 = pd.Series([True, False, True, False, np.nan, np.nan, True, False, np.nan])
+        pser2 = pd.Series([True, False, False, True, True, False, np.nan, np.nan, np.nan])
+        kser1 = ks.from_pandas(pser1)
+        kser2 = ks.from_pandas(pser2)
+
+        self.assert_eq(pser1 | pser2, (kser1 | kser2).sort_index())
+        self.assert_eq(pser1 & pser2, (kser1 & kser2).sort_index())
+
+        pser1 = pd.Series([True, False, np.nan], index=list('ABC'))
+        pser2 = pd.Series([False, True, np.nan], index=list('DEF'))
+        kser1 = ks.from_pandas(pser1)
+        kser2 = ks.from_pandas(pser2)
+
+        self.assert_eq(pser1 | pser2, (kser1 | kser2).sort_index())
+        self.assert_eq(pser1 & pser2, (kser1 & kser2).sort_index())
+
     def test_different_columns(self):
         kdf1 = self.kdf1
         kdf4 = self.kdf4