Repository: databricks/koalas

Issue:
pivot_table function with renaming columns
Here is the example from the documentation. 

```python
df = ks.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo",
                          "bar", "bar", "bar", "bar"],
                    "C": ["small", "large", "large", "small",
                          "small", "large", "small", "small",
                          "large"],
                    "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]},
                   columns=['A',  'C',  'E'])
df.pivot_table(values='E', index=['A'],
                        columns='C', aggfunc='sum')
```

The above code works as expected.  Then I rename the columns like 

```python
df.columns =['New_A','New_C','New_E']
print(df.columns)
```
This works fine with `applymap` and other functions like `to_datetime` but as soon as I use `pivot_table` , I got an error. 

```python
df.pivot_table(values='New_E', index=['New_A'],
                        columns='New_C', aggfunc='sum')
```
The above code will give the following error:

```python

KeyError: 'No StructField named New_E'

```

Is it not allowed to change the column name right now??  Thanks.


Discussion:
Seems like `df.columns = new_columns` doesn't change the columns in the internal spark dataframe. `pivot_table` tries to access the column  `New_E` which doesn't exist and raise an error.

```
>>> df._sdf.show()
+-----------------+---+-----+---+
|__index_level_0__|  A|    C|  E|
+-----------------+---+-----+---+
|                0|foo|small|  2|
|                1|foo|large|  4|
|                2|foo|large|  5|
|                3|foo|small|  5|
|                4|foo|small|  6|
|                5|bar|large|  6|
|                6|bar|small|  8|
|                7|bar|small|  9|
|                8|bar|large|  9|
+-----------------+---+-----+---+

```
@harupy Actually the root cause is the `pivot_table` doesn't handle metadata (`_InternalFrame`) properly. This is the same as `merge` case at #782. So far fixing `columns` is easier way, though, but it doesn't fix the case multi-index columns are used.
BTW, thanks for reporting this @thoo. the issue looks very easy to understand and reproduce.
Shall we fix the `columns` setter to rename the Spark columns for now?
@itholic @harupy Do you want to fix it?

I'll handle `merge` and `pivot_table` soon.
I'll work on this.
@harupy Great, thanks!
oh It seems pretty interesting, 
thanks for suggestion @ueshin ,
and go for it @harupy ! 😃 

Resolving patch:
diff --git a/databricks/koalas/frame.py b/databricks/koalas/frame.py
--- a/databricks/koalas/frame.py
+++ b/databricks/koalas/frame.py
@@ -4214,7 +4214,14 @@ def columns(self, columns):
                     "Length mismatch: Expected axis has %d elements, new values have %d elements"
                     % (len(old_names), len(column_index)))
             column_index_names = columns.names
-            self._internal = self._internal.copy(column_index=column_index,
+            data_columns = [str(idx) if len(idx) > 1 else idx[0] for idx in column_index]
+            sdf = self._sdf.select(
+                self._internal.index_scols +
+                [self._internal.scol_for(idx).alias(name)
+                 for idx, name in zip(self._internal.column_index, data_columns)])
+            self._internal = self._internal.copy(sdf=sdf,
+                                                 data_columns=data_columns,
+                                                 column_index=column_index,
                                                  column_index_names=column_index_names)
         else:
             old_names = self._internal.column_index
@@ -4227,7 +4234,14 @@ def columns(self, columns):
                 column_index_names = columns.names
             else:
                 column_index_names = None
-            self._internal = self._internal.copy(column_index=column_index,
+            data_columns = [str(idx) if len(idx) > 1 else idx[0] for idx in column_index]
+            sdf = self._sdf.select(
+                self._internal.index_scols +
+                [self._internal.scol_for(idx).alias(name)
+                 for idx, name in zip(self._internal.column_index, data_columns)])
+            self._internal = self._internal.copy(sdf=sdf,
+                                                 data_columns=data_columns,
+                                                 column_index=column_index,
                                                  column_index_names=column_index_names)
 
     @property


Test patch:
diff --git a/databricks/koalas/tests/test_dataframe.py b/databricks/koalas/tests/test_dataframe.py
--- a/databricks/koalas/tests/test_dataframe.py
+++ b/databricks/koalas/tests/test_dataframe.py
@@ -324,6 +324,8 @@ def test_rename_columns(self):
         pdf.columns = ['x', 'y']
         self.assert_eq(kdf.columns, pd.Index(['x', 'y']))
         self.assert_eq(kdf, pdf)
+        self.assert_eq(kdf._internal.data_columns, ['x', 'y'])
+        self.assert_eq(kdf._internal.spark_df.columns, ['x', 'y'])
 
         columns = pdf.columns
         columns.name = 'lvl_1'
@@ -348,17 +350,23 @@ def test_rename_columns(self):
         kdf.columns = ['x', 'y']
         self.assert_eq(kdf.columns, pd.Index(['x', 'y']))
         self.assert_eq(kdf, pdf)
+        self.assert_eq(kdf._internal.data_columns, ['x', 'y'])
+        self.assert_eq(kdf._internal.spark_df.columns, ['x', 'y'])
 
         pdf.columns = columns
         kdf.columns = columns
         self.assert_eq(kdf.columns, columns)
         self.assert_eq(kdf, pdf)
+        self.assert_eq(kdf._internal.data_columns, ["('A', '0')", "('B', 1)"])
+        self.assert_eq(kdf._internal.spark_df.columns, ["('A', '0')", "('B', 1)"])
 
         columns.names = ['lvl_1', 'lvl_2']
 
         kdf.columns = columns
         self.assert_eq(kdf.columns.names, ['lvl_1', 'lvl_2'])
         self.assert_eq(kdf, pdf)
+        self.assert_eq(kdf._internal.data_columns, ["('A', '0')", "('B', 1)"])
+        self.assert_eq(kdf._internal.spark_df.columns, ["('A', '0')", "('B', 1)"])
 
     def test_dot_in_column_name(self):
         self.assert_eq(