Repository: databricks/koalas

Issue:
Implement missing functions for 10-minute tutorial with Koalas
This is the list of missing functions for **10-minute tutorial with Koalas** (#196) 

+ [ ] DataFrame.pivot_table
+ [ ] DataFrame.reindex
+ [ ] DataFrame.apply
+ [x] DataFrame.append (#388)
+ [x] DataFrame.at (#384)
+ [ ] DataFrame.T
+ [x] DataFrame.sub (#427)

Ignored functions
+ DataFrame.to_hdf
+ DataFrame.iat
+ DataFrame.shift

Some functions could be implemented for both DataFrame and Series. 

For contributors, please mention that you're working on some functions here and go ahead.



Discussion:
@HyukjinKwon any thoughts? 
Let's focus on: pivot_table, reindex, apply, append, at, T, to_hdf (in this order)

The other functions have big performance impact and will require more careful integration with spark, by using window functions under the hood for example.

@garawalid, if you are looking at implementing some of them, I suggest that we go in this order. What are your thoughts?
I can definitely start with `reindex`,  `append` and `at` as those three all seem pretty straight forward.

The others seem tricky to me for different reasons:
- `pivot_table` and `T` might require special care with their implementation as they potentially move around huge amounts of data between workers.
- `apply` should probably take UDFs since pandas/NumPy methods need to move the data from the Spark process to the Python process and back. Expecting the user to use PySpark UDFs however, would (again) break the separation between the (Py)Spark and pandas APIs.
- `to_hdf` should be easy to implement but requires a decision if the write to disk should happen from Spark or from Python.
Does anybody still use hdf these days? Maybe I'm not familiar enough with the pandas use cases. I feel maybe it's ok to skip hdf given people rarely use that in "big data".

@rxin, FWIW I think you are right that to_hdf is a very rarely used function. I'm sure there are edge cases, but I've never seen it used.
I've seen people (including myself) using HDF5 files to store Keras models but I've also never seen anyone using it with pandas or Spark. So probably it makes most sense to leave it out for now and wait until someone requests this feature (if ever).
@thunterdb `pivot_table` sounds a little bit tricky, I'll start with a simple version and we will improve it with different PR. 

Thanks @floscha, @rxin, @athena15 , @thunterdb . I'll update the list as you suggested. 
We will ignore `iat` because it needs the position of the row. 

Resolving patch:
diff --git a/databricks/koalas/frame.py b/databricks/koalas/frame.py
--- a/databricks/koalas/frame.py
+++ b/databricks/koalas/frame.py
@@ -2467,6 +2467,145 @@ def head(self, n=5):
 
         return DataFrame(self._internal.copy(sdf=self._sdf.limit(n)))
 
+    def pivot_table(self, values=None, index=None, columns=None,
+                    aggfunc='mean', fill_value=None):
+        """
+        Create a spreadsheet-style pivot table as a DataFrame. The levels in
+        the pivot table will be stored in MultiIndex objects (hierarchical
+        indexes) on the index and columns of the result DataFrame.
+
+        Parameters
+        ----------
+        values : column to aggregate.
+            They should be either a list of one column or a string. A list of columns
+            is not supported yet.
+        index : column (string) or list of columns
+            If an array is passed, it must be the same length as the data.
+            The list should contain string.
+        columns : column
+            Columns used in the pivot operation. Only one column is supported and
+            it should be a string.
+        aggfunc : function (string), dict, default mean
+            If dict is passed, the resulting pivot table will have
+            columns concatenated by "_" where the first part is the value
+            of columns and the second part is the column name in values
+            If dict is passed, the key is column to aggregate and value
+            is function or list of functions.
+        fill_value : scalar, default None
+            Value to replace missing values with.
+
+        Returns
+        -------
+        table : DataFrame
+
+        Examples
+        --------
+        >>> df = ks.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo",
+        ...                          "bar", "bar", "bar", "bar"],
+        ...                    "B": ["one", "one", "one", "two", "two",
+        ...                          "one", "one", "two", "two"],
+        ...                    "C": ["small", "large", "large", "small",
+        ...                          "small", "large", "small", "small",
+        ...                          "large"],
+        ...                    "D": [1, 2, 2, 3, 3, 4, 5, 6, 7],
+        ...                    "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]})
+        >>> df
+             A    B      C  D  E
+        0  foo  one  small  1  2
+        1  foo  one  large  2  4
+        2  foo  one  large  2  5
+        3  foo  two  small  3  5
+        4  foo  two  small  3  6
+        5  bar  one  large  4  6
+        6  bar  one  small  5  8
+        7  bar  two  small  6  9
+        8  bar  two  large  7  9
+
+        This first example aggregates values by taking the sum.
+
+        >>> table = df.pivot_table(values='D', index=['A', 'B'],
+        ...                         columns='C', aggfunc='sum')
+        >>> table  # doctest: +NORMALIZE_WHITESPACE
+                 large  small
+        A   B
+        foo one    4.0      1
+            two    NaN      6
+        bar two    7.0      6
+            one    4.0      5
+
+        We can also fill missing values using the `fill_value` parameter.
+
+        >>> table = df.pivot_table(values='D', index=['A', 'B'],
+        ...                         columns='C', aggfunc='sum', fill_value=0)
+        >>> table  # doctest: +NORMALIZE_WHITESPACE
+                 large  small
+        A   B
+        foo one      4      1
+            two      0      6
+        bar two      7      6
+            one      4      5
+
+        We can also calculate multiple types of aggregations for any given
+        value column.
+
+        >>> table = df.pivot_table(values = ['D'], index =['C'],
+        ...                         columns="A", aggfunc={'D':'mean'})
+        >>> table  # doctest: +NORMALIZE_WHITESPACE
+               bar       foo
+        C
+        small  5.5  2.333333
+        large  5.5  2.000000
+        """
+        if not isinstance(columns, str):
+            raise ValueError("columns should be string.")
+
+        if not isinstance(values, str) and not isinstance(values, list):
+            raise ValueError('values should be string or list of one column.')
+
+        if not isinstance(aggfunc, str) and (not isinstance(aggfunc, dict) or not all(
+                isinstance(key, str) and isinstance(value, str) for key, value in aggfunc.items())):
+            raise ValueError("aggfunc must be a dict mapping from column name (string) "
+                             "to aggregate functions (string).")
+
+        if isinstance(aggfunc, dict) and index is None:
+            raise NotImplementedError("pivot_table doesn't support aggfuct"
+                                      " as dict and without index.")
+
+        if isinstance(values, list) and len(values) > 1:
+            raise NotImplementedError('Values as list of columns is not implemented yet.')
+
+        if isinstance(aggfunc, str):
+            agg_cols = [F.expr('{1}({0}) as {0}'.format(values, aggfunc))]
+
+        elif isinstance(aggfunc, dict):
+            agg_cols = [F.expr('{1}({0}) as {0}'.format(key, value))
+                        for key, value in aggfunc.items()]
+            agg_columns = [key for key, value in aggfunc.items()]
+
+            if set(agg_columns) != set(values):
+                raise ValueError("Columns in aggfunc must be the same as values.")
+
+        if index is None:
+            sdf = self._sdf.groupBy().pivot(pivot_col=columns).agg(*agg_cols)
+
+        elif isinstance(index, list):
+            sdf = self._sdf.groupBy(index).pivot(pivot_col=columns).agg(*agg_cols)
+        else:
+            raise ValueError("index should be a None or a list of columns.")
+
+        if fill_value is not None and isinstance(fill_value, (int, float)):
+            sdf = sdf.fillna(fill_value)
+
+        if index is not None:
+            return DataFrame(sdf).set_index(index)
+        else:
+            if isinstance(values, list):
+                index_values = values[-1]
+            else:
+                index_values = values
+
+            return DataFrame(sdf.withColumn(columns, F.lit(index_values))).set_index(columns)
+
     @property
     def columns(self):
         """The column labels of the DataFrame."""
@@ -3310,7 +3449,7 @@ def merge(self, right: 'DataFrame', how: str = 'inner',
                 ).withColumnRenamed(
                     'left_table.%s' % left_index_col, left_index_col
                 ).drop(F.col('left_table.%s' % left_index_col))
-        if not(left_index and not right_index):
+        if not (left_index and not right_index):
             selected_columns = selected_columns.drop(*[F.col('right_table.%s' % right_index_col)
                                                        for right_index_col in right_index_columns
                                                        if right_index_col in left_index_columns])
diff --git a/databricks/koalas/missing/frame.py b/databricks/koalas/missing/frame.py
--- a/databricks/koalas/missing/frame.py
+++ b/databricks/koalas/missing/frame.py
@@ -100,7 +100,6 @@ class _MissingPandasLikeDataFrame(object):
     mode = unsupported_function('mode')
     pct_change = unsupported_function('pct_change')
     pivot = unsupported_function('pivot')
-    pivot_table = unsupported_function('pivot_table')
     pop = unsupported_function('pop')
     pow = unsupported_function('pow')
     prod = unsupported_function('prod')


Test patch:
diff --git a/databricks/koalas/tests/test_dataframe.py b/databricks/koalas/tests/test_dataframe.py
--- a/databricks/koalas/tests/test_dataframe.py
+++ b/databricks/koalas/tests/test_dataframe.py
@@ -728,7 +728,7 @@ def test_add_suffix(self):
     def test_join(self):
         # check basic function
         pdf1 = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
-                            'A': ['A0', 'A1', 'A2', 'A3']}, columns=['key', 'A'])
+                             'A': ['A0', 'A1', 'A2', 'A3']}, columns=['key', 'A'])
         pdf2 = pd.DataFrame({'key': ['K0', 'K1', 'K2'],
                              'B': ['B0', 'B1', 'B2']}, columns=['key', 'B'])
         kdf1 = ks.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
@@ -778,3 +778,102 @@ def get_data():
 
         with self.assertRaises(NotImplementedError):
             left_kdf.update(right_kdf, join='right')
+
+    def test_pivot_table_dtypes(self):
+        pdf = pd.DataFrame({'a': [4, 2, 3, 4, 8, 6],
+                            'b': [1, 2, 2, 4, 2, 4],
+                            'e': [1, 2, 2, 4, 2, 4],
+                            'c': [1, 2, 9, 4, 7, 4]},
+                           index=[10, 20, 30, 40, 50, 60])
+
+        kdf = ks.from_pandas(pdf)
+
+        # Skip columns comparison by reset_index
+        res_df = kdf.pivot_table(index=['c'], columns="a", values=['b'],
+                                 aggfunc={'b': 'mean'}) \
+            .dtypes.reset_index(drop=True)
+        exp_df = pdf.pivot_table(index=['c'], columns="a", values=['b'],
+                                 aggfunc={'b': 'mean'}) \
+            .dtypes.reset_index(drop=True)
+        self.assert_eq(res_df, exp_df)
+
+        # Results don't have the same column's name
+
+        # Todo: self.assert_eq(kdf.pivot_table(columns="a", values="b").dtypes,
+        #  pdf.pivot_table(columns="a", values="b").dtypes)
+
+        # Todo: self.assert_eq(kdf.pivot_table(index=['c'], columns="a", values="b").dtypes,
+        #  pdf.pivot_table(index=['c'], columns="a", values="b").dtypes)
+
+        # Todo: self.assert_eq(kdf.pivot_table(index=['e', 'c'], columns="a", values="b").dtypes,
+        #  pdf.pivot_table(index=['e', 'c'], columns="a", values="b").dtypes)
+
+        # Todo: self.assert_eq(kdf.pivot_table(index=['e', 'c'],
+        #  columns="a", values="b", fill_value=999).dtypes, pdf.pivot_table(index=['e', 'c'],
+        #  columns="a", values="b", fill_value=999).dtypes)
+
+    def test_pivot_table(self):
+        pdf = pd.DataFrame({'a': [4, 2, 3, 4, 8, 6],
+                            'b': [1, 2, 2, 4, 2, 4],
+                            'e': [1, 2, 2, 4, 2, 4],
+                            'c': [1, 2, 9, 4, 7, 4]},
+                           index=[10, 20, 30, 40, 50, 60])
+
+        kdf = ks.from_pandas(pdf)
+
+        # Checking if both DataFrames have the same results (Temporary)
+        np.testing.assert_equal(kdf.pivot_table(columns="a", values="b").to_numpy(),
+                                pdf.pivot_table(columns=["a"], values="b").to_numpy())
+
+        # Todo: self.assert_eq(kdf.pivot_table(columns="a", values="b"),
+        #  pdf.pivot_table(columns=["a"], values="b"))
+
+        # Todo: self.assert_eq(kdf.pivot_table(index=['c'], columns="a", values="b"),
+        #  pdf.pivot_table(index=['c'], columns=["a"], values="b"))
+
+        # Todo: self.assert_eq(kdf.pivot_table(index=['c'], columns="a", values=['b', 'e'],
+        #  aggfunc={'b': 'mean', 'e': 'sum'}), pdf.pivot_table(index=['c'], columns=["a"],
+        #  values=['b', 'e'], aggfunc={'b': 'mean', 'e': 'sum'}))
+
+        # Todo: self.assert_eq(kdf.pivot_table(index=['e', 'c'], columns="a", values="b"),
+        #  pdf.pivot_table(index=['e', 'c'], columns="a", values="b"))
+
+        # Todo: self.assert_eq(kdf.pivot_table(index=['e', 'c'], columns="a", values="b",
+        #  fill_value=999), pdf.pivot_table(index=['e', 'c'], columns="a", values="b",
+        #  fill_value=999))
+
+    def test_pivot_table_erros(self):
+
+        pdf = pd.DataFrame({'a': [4, 2, 3, 4, 8, 6],
+                            'b': [1, 2, 2, 4, 2, 4],
+                            'e': [1, 2, 2, 4, 2, 4],
+                            'c': [1, 2, 9, 4, 7, 4]},
+                           index=[10, 20, 30, 40, 50, 60])
+
+        kdf = ks.from_pandas(pdf)
+
+        msg = "values should be string or list of one column."
+        with self.assertRaisesRegex(ValueError, msg):
+            kdf.pivot_table(index=['c'], columns="a", values=5)
+
+        msg = "index should be a None or a list of columns."
+        with self.assertRaisesRegex(ValueError, msg):
+            kdf.pivot_table(index="c", columns="a", values="b")
+
+        msg = "pivot_table doesn't support aggfuct as dict and without index."
+        with self.assertRaisesRegex(NotImplementedError, msg):
+            kdf.pivot_table(columns="a", values=['b', 'e'], aggfunc={'b': 'mean', 'e': 'sum'})
+
+        msg = "columns should be string."
+        with self.assertRaisesRegex(ValueError, msg):
+            kdf.pivot_table(columns=["a"], values=['b'], aggfunc={'b': 'mean', 'e': 'sum'})
+
+        msg = "Columns in aggfunc must be the same as values."
+        with self.assertRaisesRegex(ValueError, msg):
+            kdf.pivot_table(index=['e', 'c'], columns="a", values='b',
+                            aggfunc={'b': 'mean', 'e': 'sum'})
+
+        msg = 'Values as list of columns is not implemented yet.'
+        with self.assertRaisesRegex(NotImplementedError, msg):
+            kdf.pivot_table(index=['c'], columns="a", values=['b', 'e'],
+                            aggfunc={'b': 'mean', 'e': 'sum'})