Repository: databricks/koalas

Issue:
Support MultiIndex for `loc` indexer with slice as `row_sel`.
We should support MultiIndex for `loc` indexer with slice as `row_sel`.

```py
>>> pser
x  a    1
   b    2
y  c    3
   d    4
z  e    5
dtype: int64
>>> pser.loc['y':]
y  c    3
   d    4
z  e    5
dtype: int64
>>> pser.loc[:'y']
x  a    1
   b    2
y  c    3
   d    4
dtype: int64
>>> pser.loc[('x','b'):]
x  b    2
y  c    3
   d    4
z  e    5
dtype: int64
>>> pser.loc[:('y','c')]
x  a    1
   b    2
y  c    3
dtype: int64
>>> pser.loc[('x','b'):('y','c')]
x  b    2
y  c    3
dtype: int64
>>> pser.loc['x':('y','c')]
x  a    1
   b    2
y  c    3
dtype: int64
>>> pser.loc[('x','b'):'y']
x  b    2
y  c    3
   d    4
dtype: int64
```
[WIP] Support MultiIndex for `loc` indexer with slice as `row_sel`.
Resolves #1175 

```python
>>> kser
x  a    1
   b    2
y  c    3
   d    4
z  e    5
Name: 0, dtype: int64

>>> kser.loc['y':]
y  c    3
   d    4
z  e    5
Name: 0, dtype: int64

>>> kser.loc[:'y']
x  a    1
   b    2
y  c    3
   d    4
Name: 0, dtype: int64

>>> kser.loc[('x', 'b'):]
x  b    2
y  c    3
   d    4
z  e    5
Name: 0, dtype: int64

>>> kser.loc[:('y', 'c')]
x  a    1
   b    2
y  c    3
Name: 0, dtype: int64

>>> kser.loc[('x', 'b'):('y', 'c')]
x  b    2
y  c    3
Name: 0, dtype: int64

>>> kser.loc['x':('y', 'c')]
x  a    1
   b    2
y  c    3
Name: 0, dtype: int64

>>> kser.loc[('x', 'b'):'y']
x  b    2
y  c    3
   d    4
Name: 0, dtype: int64
```


Discussion:
@itholic, are you interested in this?
@HyukjinKwon Oh, I was reviewing the various indexers and considering implementing this part. I will be happy to.
i'll take this and #1176  also
While i'm here, i'm going to add codes for handling exception such like `KeyError` when given key is not included in `MultiIndex` and their test cases.

FYI: for this, we need #1183 
### Softagram Impact Report for pull/1182 (head commit: b863ba8dc962)



### :warning: Copy paste found
:information_source: test_indexing.py: *Copy paste fragment* inside the same file on lines 600, 618:
```                        [False, True, True],
                        slice(0, 1)]:
            self.assert_eq(kdf.iloc[:, indexer], pdf.iloc[:, indexer])
            self.assert_eq(kdf.iloc[:1, indexe...(truncated 198 chars)
```

 _Now that you are on the file, it would be easier to pay back some tech. debt._


### :star: Change Overview


![Showing the changed files, dependency changes and the impact - click for full size](https://opensource.softagram.com/cdn/impact/dd6f973b-a9da-477d-8f74-1a29916573a5_koalas_1182_impact_change_graph_6zyBVsAVMPvBMIt83VCFms9HmPcmQI.png)
*(Open in [Softagram Desktop](https://softagram.com/desktop) for full details)*






### :page_facing_up: Full report

* Permalink: [Full report for pull/1182](https://opensource.softagram.com/dashboard/teams/7ff2e81c-61e0-4605-b4ac-d496bc4c37c7/projects/dd6f973b-a9da-477d-8f74-1a29916573a5/impact-reports/51f54cbc-76de-4dbc-badb-8f6508e8ece9)

#
[Impact Report explained.](http://help.softagram.com/en/articles/2847546-pull-merge-request-report-explained) Give feedback on this report to support@softagram.com

hmm.. pandas doesn't support below case.
(if `MultiIndex` is not sorted, it seems they doesn't support)

```python
>>> pser
z  a    1
   b    2
y  c    3
   d    4
x  e    5
dtype: int64

>>> pser.loc['z':]
Traceback (most recent call last):
...
pandas.errors.UnsortedIndexError: 'Key length (1) was greater than MultiIndex lexsort depth (0)'

>>> pser.loc[('z', 'a'):]
Traceback (most recent call last):
...
pandas.errors.UnsortedIndexError: 'Key length (2) was greater than MultiIndex lexsort depth (0)'
```

we can support above, but i think we'd better to block this for now to follow pandas' behavior.
# [Codecov](https://codecov.io/gh/databricks/koalas/pull/1182?src=pr&el=h1) Report
> Merging [#1182](https://codecov.io/gh/databricks/koalas/pull/1182?src=pr&el=desc) into [master](https://codecov.io/gh/databricks/koalas/commit/15a6406da7f92e0464639031eb818c40e874b8ad?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/databricks/koalas/pull/1182/graphs/tree.svg?width=650&token=EtnrpE8dVx&height=150&src=pr)](https://codecov.io/gh/databricks/koalas/pull/1182?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1182      +/-   ##
==========================================
+ Coverage   95.24%   95.24%   +<.01%     
==========================================
  Files          35       35              
  Lines        7105     7135      +30     
==========================================
+ Hits         6767     6796      +29     
- Misses        338      339       +1
```


| [Impacted Files](https://codecov.io/gh/databricks/koalas/pull/1182?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [databricks/koalas/indexing.py](https://codecov.io/gh/databricks/koalas/pull/1182/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvaW5kZXhpbmcucHk=) | `96.02% <100%> (+0.06%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/databricks/koalas/pull/1182?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/databricks/koalas/pull/1182?src=pr&el=footer). Last update [15a6406...b863ba8](https://codecov.io/gh/databricks/koalas/pull/1182?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

since #1183 has merged, i visit this work again for now 😄 
@itholic I'll take this over.
@ueshin Sure, go ahead. Sorry for the delay :crying_cat_face: 

Resolving patch:
diff --git a/databricks/koalas/indexes.py b/databricks/koalas/indexes.py
--- a/databricks/koalas/indexes.py
+++ b/databricks/koalas/indexes.py
@@ -1914,19 +1914,23 @@ def levshape(self):
         ).collect()[0]
         return tuple(result)
 
+    @staticmethod
+    def _comparator_for_monotonic_increasing(data_type):
+        if isinstance(data_type, BooleanType):
+            return compare_allow_null
+        else:
+            return compare_null_last
+
     def _is_monotonic(self):
-        col = self._scol
+        scol = self._scol
         window = Window.orderBy(NATURAL_ORDER_COLUMN_NAME).rowsBetween(-1, -1)
-        prev = F.lag(col, 1).over(window)
+        prev = F.lag(scol, 1).over(window)
 
         cond = F.lit(True)
         for field in self.spark_type[::-1]:
-            left = col.getField(field.name)
+            left = scol.getField(field.name)
             right = prev.getField(field.name)
-            if isinstance(field.dataType, BooleanType):
-                compare = compare_allow_null
-            else:
-                compare = compare_null_last
+            compare = MultiIndex._comparator_for_monotonic_increasing(field.dataType)
             cond = F.when(left.eqNullSafe(right), cond).otherwise(
                 compare(left, right, spark.Column.__gt__)
             )
@@ -1942,23 +1946,27 @@ def _is_monotonic(self):
 
         return _col(DataFrame(internal))
 
+    @staticmethod
+    def _comparator_for_monotonic_decreasing(data_type):
+        if isinstance(data_type, StringType):
+            return compare_disallow_null
+        elif isinstance(data_type, BooleanType):
+            return compare_allow_null
+        elif isinstance(data_type, NumericType):
+            return compare_null_last
+        else:
+            return compare_null_first
+
     def _is_monotonic_decreasing(self):
-        col = self._scol
+        scol = self._scol
         window = Window.orderBy(NATURAL_ORDER_COLUMN_NAME).rowsBetween(-1, -1)
-        prev = F.lag(col, 1).over(window)
+        prev = F.lag(scol, 1).over(window)
 
         cond = F.lit(True)
         for field in self.spark_type[::-1]:
-            left = col.getField(field.name)
+            left = scol.getField(field.name)
             right = prev.getField(field.name)
-            if isinstance(field.dataType, StringType):
-                compare = compare_disallow_null
-            elif isinstance(field.dataType, BooleanType):
-                compare = compare_allow_null
-            elif isinstance(field.dataType, NumericType):
-                compare = compare_null_last
-            else:
-                compare = compare_null_first
+            compare = MultiIndex._comparator_for_monotonic_decreasing(field.dataType)
             cond = F.when(left.eqNullSafe(right), cond).otherwise(
                 compare(left, right, spark.Column.__lt__)
             )
diff --git a/databricks/koalas/indexing.py b/databricks/koalas/indexing.py
--- a/databricks/koalas/indexing.py
+++ b/databricks/koalas/indexing.py
@@ -519,6 +519,7 @@ def _raiseNotImplemented(description):
         )
 
     def _select_rows(self, rows_sel):
+        from databricks.koalas.indexes import MultiIndex
         from databricks.koalas.series import Series
 
         if isinstance(rows_sel, Series):
@@ -595,10 +596,62 @@ def _select_rows(self, rows_sel):
                         else:
                             raise KeyError(rows_sel.stop)
 
-                if len(cond) > 0:
-                    return reduce(lambda x, y: x & y, cond), None, None
+                return reduce(lambda x, y: x & y, cond), None, None
             else:
-                LocIndexer._raiseNotImplemented("Cannot use slice for MultiIndex with Spark.")
+                index = self._kdf_or_kser.index
+                index_data_type = [f.dataType for f in index.to_series().spark_type]
+
+                start = rows_sel.start
+                if start is not None:
+                    if not isinstance(start, tuple):
+                        start = (start,)
+                    if len(start) == 0:
+                        start = None
+                stop = rows_sel.stop
+                if stop is not None:
+                    if not isinstance(stop, tuple):
+                        stop = (stop,)
+                    if len(stop) == 0:
+                        stop = None
+
+                depth = max(
+                    len(start) if start is not None else 0, len(stop) if stop is not None else 0
+                )
+                if depth == 0:
+                    return None, None, None
+                elif (
+                    depth > len(self._internal.index_map)
+                    or not index.droplevel(
+                        list(range(len(self._internal.index_map))[depth:])
+                    ).is_monotonic
+                ):
+                    raise KeyError(
+                        "Key length ({}) was greater than MultiIndex sort depth".format(depth)
+                    )
+
+                conds = []
+                if start is not None:
+                    cond = F.lit(True)
+                    for scol, value, dt in list(
+                        zip(self._internal.index_spark_columns, start, index_data_type)
+                    )[::-1]:
+                        compare = MultiIndex._comparator_for_monotonic_increasing(dt)
+                        cond = F.when(scol.eqNullSafe(F.lit(value).cast(dt)), cond).otherwise(
+                            compare(scol, F.lit(value).cast(dt), spark.Column.__gt__)
+                        )
+                    conds.append(cond)
+                if stop is not None:
+                    cond = F.lit(True)
+                    for scol, value, dt in list(
+                        zip(self._internal.index_spark_columns, stop, index_data_type)
+                    )[::-1]:
+                        compare = MultiIndex._comparator_for_monotonic_increasing(dt)
+                        cond = F.when(scol.eqNullSafe(F.lit(value).cast(dt)), cond).otherwise(
+                            compare(scol, F.lit(value).cast(dt), spark.Column.__lt__)
+                        )
+                    conds.append(cond)
+
+                return reduce(lambda x, y: x & y, conds), None, None
         elif is_list_like(rows_sel) and not isinstance(rows_sel, tuple):
             rows_sel = list(rows_sel)
             if len(rows_sel) == 0:


Test patch:
diff --git a/databricks/koalas/tests/test_indexes.py b/databricks/koalas/tests/test_indexes.py
--- a/databricks/koalas/tests/test_indexes.py
+++ b/databricks/koalas/tests/test_indexes.py
@@ -878,10 +878,11 @@ def test_monotonic(self):
         datas.append([("x", "d", "o"), ("y", "c", None), ("y", "c", None), ("z", "a", "r")])
 
         for data in datas:
-            kmidx = ks.MultiIndex.from_tuples(data)
-            pmidx = kmidx.to_pandas()
-            self.assert_eq(kmidx.is_monotonic_increasing, pmidx.is_monotonic_increasing)
-            self.assert_eq(kmidx.is_monotonic_decreasing, pmidx.is_monotonic_decreasing)
+            with self.subTest(data=data):
+                kmidx = ks.MultiIndex.from_tuples(data)
+                pmidx = kmidx.to_pandas()
+                self.assert_eq(kmidx.is_monotonic_increasing, pmidx.is_monotonic_increasing)
+                self.assert_eq(kmidx.is_monotonic_decreasing, pmidx.is_monotonic_decreasing)
 
         # The datas below are showing different result depends on pandas version.
         # Because the behavior of handling null values is changed in pandas >= 1.0.0.
@@ -894,13 +895,14 @@ def test_monotonic(self):
         datas.append([("x", "d", "o"), ("y", "c", None), ("y", "c", "q"), ("z", "a", "r")])
 
         for data in datas:
-            kmidx = ks.MultiIndex.from_tuples(data)
-            pmidx = kmidx.to_pandas()
-            expected_increasing_result = pmidx.is_monotonic_increasing
-            if LooseVersion(pd.__version__) < LooseVersion("1.0.0"):
-                expected_increasing_result = not expected_increasing_result
-            self.assert_eq(kmidx.is_monotonic_increasing, expected_increasing_result)
-            self.assert_eq(kmidx.is_monotonic_decreasing, pmidx.is_monotonic_decreasing)
+            with self.subTest(data=data):
+                kmidx = ks.MultiIndex.from_tuples(data)
+                pmidx = kmidx.to_pandas()
+                expected_increasing_result = pmidx.is_monotonic_increasing
+                if LooseVersion(pd.__version__) < LooseVersion("1.0.0"):
+                    expected_increasing_result = not expected_increasing_result
+                self.assert_eq(kmidx.is_monotonic_increasing, expected_increasing_result)
+                self.assert_eq(kmidx.is_monotonic_decreasing, pmidx.is_monotonic_decreasing)
 
     def test_difference(self):
         # Index
diff --git a/databricks/koalas/tests/test_indexing.py b/databricks/koalas/tests/test_indexing.py
--- a/databricks/koalas/tests/test_indexing.py
+++ b/databricks/koalas/tests/test_indexing.py
@@ -382,7 +382,8 @@ def test_loc_multiindex(self):
         pdf = pdf.set_index("b", append=True)
 
         self.assert_eq(kdf.loc[:], pdf.loc[:])
-        self.assertRaises(NotImplementedError, lambda: kdf.loc[5:5])
+        self.assert_eq(kdf.loc[5:5], pdf.loc[5:5])
+        self.assert_eq(kdf.loc[5:9], pdf.loc[5:9])
 
         self.assert_eq(kdf.loc[5], pdf.loc[5])
         self.assert_eq(kdf.loc[9], pdf.loc[9])
@@ -393,6 +394,80 @@ def test_loc_multiindex(self):
         self.assertTrue((kdf.a.loc[(5, 3)] == pdf.a.loc[(5, 3)]).all())
         self.assert_eq(kdf.a.loc[(9, 0)], pdf.a.loc[(9, 0)])
 
+        # monotonically increasing index test
+        pdf = pd.DataFrame(
+            {"a": [1, 2, 3, 4, 5]},
+            index=pd.MultiIndex.from_tuples(
+                [("x", "a"), ("x", "b"), ("y", "c"), ("y", "d"), ("z", "e")]
+            ),
+        )
+        kdf = ks.from_pandas(pdf)
+
+        for rows_sel in [
+            slice(None),
+            slice("y", None),
+            slice(None, "y"),
+            slice(("x", "b"), None),
+            slice(None, ("y", "c")),
+            slice(("x", "b"), ("y", "c")),
+            slice("x", ("y", "c")),
+            slice(("x", "b"), "y"),
+        ]:
+            with self.subTest("monotonically increasing", rows_sel=rows_sel):
+                self.assert_eq(kdf.loc[rows_sel], pdf.loc[rows_sel])
+                self.assert_eq(kdf.a.loc[rows_sel], pdf.a.loc[rows_sel])
+
+        # monotonically increasing first index test
+        pdf = pd.DataFrame(
+            {"a": [1, 2, 3, 4, 5]},
+            index=pd.MultiIndex.from_tuples(
+                [("x", "a"), ("x", "b"), ("y", "c"), ("y", "a"), ("z", "e")]
+            ),
+        )
+        kdf = ks.from_pandas(pdf)
+
+        for rows_sel in [
+            slice(None),
+            slice("y", None),
+            slice(None, "y"),
+        ]:
+            with self.subTest("monotonically increasing first index", rows_sel=rows_sel):
+                self.assert_eq(kdf.loc[rows_sel], pdf.loc[rows_sel])
+                self.assert_eq(kdf.a.loc[rows_sel], pdf.a.loc[rows_sel])
+
+        for rows_sel in [
+            slice(("x", "b"), None),
+            slice(None, ("y", "c")),
+            slice(("x", "b"), ("y", "c")),
+            slice("x", ("y", "c")),
+            slice(("x", "b"), "y"),
+        ]:
+            with self.subTest("monotonically increasing first index", rows_sel=rows_sel):
+                self.assertRaises(KeyError, lambda: kdf.loc[rows_sel])
+                self.assertRaises(KeyError, lambda: kdf.a.loc[rows_sel])
+
+        # not monotonically increasing index test
+        pdf = pd.DataFrame(
+            {"a": [1, 2, 3, 4, 5]},
+            index=pd.MultiIndex.from_tuples(
+                [("z", "e"), ("y", "d"), ("y", "c"), ("x", "b"), ("x", "a")]
+            ),
+        )
+        kdf = ks.from_pandas(pdf)
+
+        for rows_sel in [
+            slice("y", None),
+            slice(None, "y"),
+            slice(("x", "b"), None),
+            slice(None, ("y", "c")),
+            slice(("x", "b"), ("y", "c")),
+            slice("x", ("y", "c")),
+            slice(("x", "b"), "y"),
+        ]:
+            with self.subTest("monotonically decreasing", rows_sel=rows_sel):
+                self.assertRaises(KeyError, lambda: kdf.loc[rows_sel])
+                self.assertRaises(KeyError, lambda: kdf.a.loc[rows_sel])
+
     def test_loc2d_multiindex(self):
         kdf = self.kdf
         kdf = kdf.set_index("b", append=True)
@@ -401,7 +476,7 @@ def test_loc2d_multiindex(self):
 
         self.assert_eq(kdf.loc[:, :], pdf.loc[:, :])
         self.assert_eq(kdf.loc[:, "a"], pdf.loc[:, "a"])
-        self.assertRaises(NotImplementedError, lambda: kdf.loc[5:5, "a"])
+        self.assert_eq(kdf.loc[5:5, "a"], pdf.loc[5:5, "a"])
 
     def test_loc2d(self):
         kdf = self.kdf