internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: databricks/koalas

Issue:
koalas.Series.mode() not returning sorted values
Koalas.Series.mode is not returning values in sorted order 
Pandas Version: 0.25.3
Koalas Version: 0.21.0

`pdf = pd.read_csv('/test/test.csv', index_col=0)`
`kdf = ks.from_pandas(pdf)`

Pandas Series mode function returns values in sorted order

`pdf.Numeric_float_col.mode().head()`
`0    58.351226`
`1    98.002419`
`2    98.023862`
`3    99.022165`
`4    99.039145`
`dtype: float64`

Whereas performing same operation using Koalas Series doesn't return sorted values as mentioned in the documentation
`kdf.Numeric_float_col.mode().head()`
`0     449.726629`
`1     149.175250`
`2     599.080985`
`3    1799.689583`
`4    1999.392477`
`Name: Numeric_float_col, dtype: float64`



Discussion:
Repost here too:
ah, we might need to sort the result values since the doc says it returns values in sorted order. here, also https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.mode.html Otherwise we should remove "in sorted order" from the doc.
Basically Koalas doesn't guarantee the row order without a special reason, though.
What do you think about this case? @HyukjinKwon 
Ah, thanks @ueshin. I missed that comment. @itholic can you make a PR to remove that sentence in the doc for now?

Resolving patch:
diff --git a/databricks/koalas/series.py b/databricks/koalas/series.py
--- a/databricks/koalas/series.py
+++ b/databricks/koalas/series.py
@@ -3386,7 +3386,7 @@ def mode(self, dropna=True) -> 'Series':
         Returns
         -------
         Series
-            Modes of the Series in sorted order.
+            Modes of the Series.
 
         Examples
         --------


Test patch:
Repository: databricks/koalas

Issue:
DataFrame.transform operates per-element when transform function is type-annotated
[`DataFrame.transform`](https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.transform.html#databricks.koalas.DataFrame.transform) is behaving differently than its pandas equivalent when its return type is annotated. Consider the example below.

```python
>>> import databricks.koalas as ks
>>> import numpy as np
>>> import pandas as pd
>>>
>>> pdf = pd.DataFrame([[1, 2, 3, 4], [2, 3, 4, 5]], columns=("a", "b", "c", "d"))
>>> kdf = ks.DataFrame([[1, 2, 3, 4], [2, 3, 4, 5]], columns=("a", "b", "c", "d"))
>>>
>>> def normalize(v):
...     return v / sum(v)
...
>>> pdf.transform(normalize)
          a    b         c         d
0  0.333333  0.4  0.428571  0.444444
1  0.666667  0.6  0.571429  0.555556
>>> kdf.transform(normalize)
          a    b         c         d                                            
0  0.333333  0.4  0.428571  0.444444
1  0.666667  0.6  0.571429  0.555556
>>>
>>> def typed_normalize(v) -> ks.Series[np.float64]:
...     return v / sum(v)
...
>>> pdf.transform(typed_normalize)
          a    b         c         d
0  0.333333  0.4  0.428571  0.444444
1  0.666667  0.6  0.571429  0.555556
>>> kdf.transform(typed_normalize)
     a    b    c    d                                                           
0  1.0  1.0  1.0  1.0
1  1.0  1.0  1.0  1.0
```

Koalas version: 0.24.0
Numpy version: 1.18.0
Pandas version: 0.25.3


Discussion:
I believe expecting global aggregation within the function doesn't work in Koalas:

```python
>>> def typed_normalize(v) -> ks.Series[np.float64]:
...     return v / sum(v)
```

because `v` will be multiple Series's as the segments of a whole Series `v`. I think the example above worked because it executed the shortcut path where it just uses pandas API (when the schema inference is required).

I think this limitation has to be clarified in the documentation at this moment.




Resolving patch:
diff --git a/databricks/koalas/frame.py b/databricks/koalas/frame.py
--- a/databricks/koalas/frame.py
+++ b/databricks/koalas/frame.py
@@ -1831,8 +1831,14 @@ def transform(self, func):
 
              Koalas uses return type hint and does not try to infer the type.
 
-        .. note:: the series within ``func`` is actually a pandas series, and
-            the length of each series is not guaranteed.
+        .. note:: the series within ``func`` is actually multiple pandas series as the
+            segments of the whole Koalas series; therefore, the length of each series
+            is not guaranteed. As an example, an aggregation against each series
+            does work as a global aggregation but an aggregation of each segment. See
+            below:
+
+            >>> def func(x) -> ks.Series[np.int32]:
+            ...     return x + sum(x)
 
         Parameters
         ----------


Test patch:
Repository: databricks/koalas

Issue:
Incompatible with pandas when floordiv with np.nan
There is an incompatible behaviour with pandas when floordiv with np.nan

```python
>>> pser = pd.Series([-100, 0, 100, None, np.nan], name="Koalas")
>>> kser = ks.from_pandas(pser)

>>> pser // np.nan
0   NaN
1   NaN
2   NaN
3   NaN
4   NaN
Name: Koalas, dtype: float64

>>> kser // np.nan                                                                                                                
0    0.0
1    0.0
2    0.0
3    NaN
4    NaN
Name: Koalas, dtype: float64
```


Discussion:

Resolving patch:
diff --git a/databricks/koalas/base.py b/databricks/koalas/base.py
--- a/databricks/koalas/base.py
+++ b/databricks/koalas/base.py
@@ -220,7 +220,7 @@ def rtruediv(left, right):
     def __floordiv__(self, other):
         def floordiv(left, right):
             return F.when(F.lit(right == 0), F.lit(np.inf).__div__(left)).otherwise(
-                F.floor(left.__div__(right))
+                F.when(F.lit(right) == np.nan, np.nan).otherwise(F.floor(left.__div__(right)))
             )
 
         return _numpy_column_op(floordiv)(self, other)
@@ -228,7 +228,7 @@ def floordiv(left, right):
     def __rfloordiv__(self, other):
         def rfloordiv(left, right):
             return F.when(F.lit(left == 0), F.lit(np.inf).__div__(right)).otherwise(
-                F.floor(F.lit(right).__div__(left))
+                F.when(F.lit(left) == np.nan, np.nan).otherwise(F.floor(F.lit(right).__div__(left)))
             )
 
         return _numpy_column_op(rfloordiv)(self, other)


Test patch:
diff --git a/databricks/koalas/tests/test_series.py b/databricks/koalas/tests/test_series.py
--- a/databricks/koalas/tests/test_series.py
+++ b/databricks/koalas/tests/test_series.py
@@ -1463,3 +1463,9 @@ def test_div_zero(self):
             result = pd.Series([np.inf, np.nan, -np.inf, np.nan, np.inf, -np.inf], name="Koalas")
             self.assert_eq(repr(kser.floordiv(0)), repr(result))
             self.assert_eq(repr(kser // 0), repr(result))
+
+    def test_floordiv_nan(self):
+        pser = pd.Series([-100, 0, 100, None, np.nan], name="Koalas")
+        kser = ks.from_pandas(pser)
+
+        self.assert_eq(repr(pser.floordiv(np.nan)), repr(kser.floordiv(np.nan)))
Repository: databricks/koalas

Issue:
get_dummies uses the prefix parameter whose type is dict return KeyError
Use get_dummies demo with prefix  and columns
![image](https://user-images.githubusercontent.com/17919002/81365829-c5638680-911b-11ea-8896-cdfb8f3cc68d.png)
pandas result like this:
![image](https://user-images.githubusercontent.com/17919002/81365863-e1672800-911b-11ea-901d-9ab3467ce93b.png)





Discussion:
cc @itholic can you take a look when you're available?
@HyukjinKwon Sure, leave it to me. I'll take it.

Resolving patch:
diff --git a/databricks/koalas/namespace.py b/databricks/koalas/namespace.py
--- a/databricks/koalas/namespace.py
+++ b/databricks/koalas/namespace.py
@@ -1576,6 +1576,8 @@ def get_dummies(
             "Length of 'prefix' ({}) did not match the length of "
             "the columns being encoded ({}).".format(len(prefix), len(column_labels))
         )
+    elif isinstance(prefix, dict):
+        prefix = [prefix[column_label[0]] for column_label in column_labels]
 
     all_values = _reduce_spark_multi(
         kdf._sdf, [F.collect_set(kdf._internal.spark_column_for(label)) for label in column_labels]


Test patch:
diff --git a/databricks/koalas/tests/test_reshape.py b/databricks/koalas/tests/test_reshape.py
--- a/databricks/koalas/tests/test_reshape.py
+++ b/databricks/koalas/tests/test_reshape.py
@@ -153,6 +153,24 @@ def test_get_dummies_prefix(self):
             almost=True,
         )
 
+        self.assert_eq(
+            ks.get_dummies(kdf, prefix={"A": "foo", "B": "bar"}),
+            pd.get_dummies(pdf, prefix={"A": "foo", "B": "bar"}),
+            almost=True,
+        )
+
+        self.assert_eq(
+            ks.get_dummies(kdf, prefix={"B": "foo", "A": "bar"}),
+            pd.get_dummies(pdf, prefix={"B": "foo", "A": "bar"}),
+            almost=True,
+        )
+
+        self.assert_eq(
+            ks.get_dummies(kdf, prefix={"A": "foo", "B": "bar"}, columns=["A", "B"]),
+            pd.get_dummies(pdf, prefix={"A": "foo", "B": "bar"}, columns=["A", "B"]),
+            almost=True,
+        )
+
         with self.assertRaisesRegex(NotImplementedError, "string types"):
             ks.get_dummies(kdf, prefix="foo")
         with self.assertRaisesRegex(ValueError, "Length of 'prefix' \\(1\\) .* \\(2\\)"):
Repository: databricks/koalas

Issue:
Column assignment doesn't support type list
In Pandas, I can assign a column of type list (code works below with df). But in Koalas, I get `TypeError: Column assignment doesn't support type list`. Could this please be supported?

```
import pandas as pd
import databricks.koalas as ks

d = {'col1': [1], 'col2': [2]}
df = pd.DataFrame(data=d)
ks_df = ks.DataFrame(df)

df['Masked_Verbatim'] = ["hello"]
```


Discussion:

Resolving patch:
diff --git a/databricks/koalas/frame.py b/databricks/koalas/frame.py
--- a/databricks/koalas/frame.py
+++ b/databricks/koalas/frame.py
@@ -10032,6 +10032,22 @@ def assign_columns(kdf, this_column_labels, that_column_labels):
                         yield (kdf._kser_for(this_label), this_label)
 
             kdf = align_diff_frames(assign_columns, self, value, fillna=False, how="left")
+        elif isinstance(value, list):
+            if len(self) != len(value):
+                raise ValueError("Length of values does not match length of index")
+
+            # TODO: avoid using default index?
+            with option_context(
+                "compute.default_index_type",
+                "distributed-sequence",
+                "compute.ops_on_diff_frames",
+                True,
+            ):
+                kdf = self.reset_index()
+                kdf[key] = ks.DataFrame(value)
+                kdf = kdf.set_index(kdf.columns[: len(self._internal.index_map)])
+                kdf.index.names = self.index.names
+
         elif isinstance(key, list):
             assert isinstance(value, DataFrame)
             # Same DataFrames.


Test patch:
diff --git a/databricks/koalas/tests/test_dataframe.py b/databricks/koalas/tests/test_dataframe.py
--- a/databricks/koalas/tests/test_dataframe.py
+++ b/databricks/koalas/tests/test_dataframe.py
@@ -92,6 +92,21 @@ def test_inplace(self):
         self.assert_eq(kdf, pdf)
         self.assert_eq(kser, pser)
 
+    def test_assign_list(self):
+        pdf, kdf = self.df_pair
+
+        pser = pdf.a
+        kser = kdf.a
+
+        pdf["x"] = [10, 20, 30, 40, 50, 60, 70, 80, 90]
+        kdf["x"] = [10, 20, 30, 40, 50, 60, 70, 80, 90]
+
+        self.assert_eq(kdf.sort_index(), pdf.sort_index())
+        self.assert_eq(kser, pser)
+
+        with self.assertRaisesRegex(ValueError, "Length of values does not match length of index"):
+            kdf["z"] = [10, 20, 30, 40, 50, 60, 70, 80]
+
     def test_dataframe_multiindex_columns(self):
         pdf = pd.DataFrame(
             {
Repository: databricks/koalas

Issue:
Docs Update: pivot_table does not have columns concatenated by “_”
As per the docs for pivot_table (v_1.1.0): https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.pivot_table.html#databricks.koalas.DataFrame.pivot_table

I was trying to get the pivoted dataframe with final columns concatenated by “\_” as the docs says in `aggfunc` parameter documentation. But the final dataframe has the multilevel columns instead of concatenated by “_”. 


This is the example:
```
>> df = ks.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo",
                         "bar", "bar", "bar", "bar"],
                   "B": ["one", "one", "one", "two", "two",
                         "one", "one", "two", "two"],
                   "C": ["small", "large", "large", "small",
                         "small", "large", "small", "small",
                         "large"],
                   "D": [1, 2, 2, 3, 3, 4, 5, 6, 7],
                   "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]},
                  columns=['A', 'B', 'C', 'D', 'E'])
>> df
     A    B      C  D  E
0  foo  one  small  1  2
1  foo  one  large  2  4
2  foo  one  large  2  5
3  foo  two  small  3  5
4  foo  two  small  3  6
5  bar  one  large  4  6
6  bar  one  small  5  8
7  bar  two  small  6  9
8  bar  two  large  7  9
```
Output after pivot:
```
table = df.pivot_table(index=['C'], columns="A", values=['D', 'E'],
                        aggfunc={'D': 'mean', 'E': 'sum'})
table.sort_index() 
         D             E
A      bar       foo bar foo
C
large  5.5  2.000000  15   9
small  5.5  2.333333  17  13
```
I think updated documentation aligned with the implementation would help!!

What I was interested in the final columns concatenated by “\_” in the pivoted dataframe, not sure if it's still supported ?? I have to manually do it after getting the pivot dataframe for now.


Discussion:

Resolving patch:
diff --git a/databricks/koalas/frame.py b/databricks/koalas/frame.py
--- a/databricks/koalas/frame.py
+++ b/databricks/koalas/frame.py
@@ -5077,9 +5077,6 @@ def pivot_table(self, values=None, index=None, columns=None, aggfunc="mean", fil
             Columns used in the pivot operation. Only one column is supported and
             it should be a string.
         aggfunc : function (string), dict, default mean
-            If dict is passed, the resulting pivot table will have
-            columns concatenated by "_" where the first part is the value
-            of columns and the second part is the column name in values
             If dict is passed, the key is column to aggregate and value
             is function or list of functions.
         fill_value : scalar, default None


Test patch:
Repository: databricks/koalas

Issue:
Key order not maintained on merge
If I am reading the[ ks.merge docs](https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.merge.html#databricks.koalas.DataFrame.merge) correctly, if I perform an inner merge, the order of the left keys should be maintained:

```
inner: use intersection of keys from both frames, similar to a SQL inner join;
preserve the order of the left keys.
```

However, this seems to not be the case: 

```
left = ks.DataFrame({'id':range(4), 'left_vals': ['left']*4})
right = ks.DataFrame({'id':range(10), 'right_vals': ['right']*10})

merge_df = left.merge(right, how="inner", on="id")
print(merge_df)

   id left_vals right_vals
0   2      left      right
1   0      left      right
2   1      left      right
3   3      left      right
```

The `id` column order does not match the original sequential order in the left dataframe. Am I doing something wrong, or are the docs incorrect?


Discussion:
I think we should fix the doc. the order is not guaranteed in general.

Resolving patch:
diff --git a/databricks/koalas/frame.py b/databricks/koalas/frame.py
--- a/databricks/koalas/frame.py
+++ b/databricks/koalas/frame.py
@@ -6527,14 +6527,14 @@ def merge(
         how: Type of merge to be performed.
             {'left', 'right', 'outer', 'inner'}, default 'inner'
 
-            left: use only keys from left frame, similar to a SQL left outer join; preserve key
-                order.
-            right: use only keys from right frame, similar to a SQL right outer join; preserve key
-                order.
+            left: use only keys from left frame, similar to a SQL left outer join; not preserve
+                key order unlike pandas.
+            right: use only keys from right frame, similar to a SQL right outer join; not preserve
+                key order unlike pandas.
             outer: use union of keys from both frames, similar to a SQL full outer join; sort keys
                 lexicographically.
             inner: use intersection of keys from both frames, similar to a SQL inner join;
-                preserve the order of the left keys.
+                not preserve the order of the left keys unlike pandas.
         on: Column or index level names to join on. These must be found in both DataFrames. If on
             is None and not merging on indexes then this defaults to the intersection of the
             columns in both DataFrames.
@@ -6903,8 +6903,8 @@ def join(
         K3   A3  None
 
         Another option to join using the key columns is to use the on parameter. DataFrame.join
-        always uses right’s index but we can use any column in df. This method preserves the
-        original DataFrame’s index in the result.
+        always uses right’s index but we can use any column in df. This method not preserve the
+        original DataFrame’s index in the result unlike pandas.
 
         >>> join_kdf = kdf1.join(kdf2.set_index('key'), on='key')
         >>> join_kdf.index.sort_values()


Test patch:
Repository: databricks/koalas

Issue:
`fillna()` `inplace` unexpected behaviour
When using `inplace=True` with `fillna()` on a `Series` that does not have any `na`s to fill, the entire `DataFrame` to which the `Series` belongs is replaced by the `Series`. Let me demonstrate with an example:


## Koalas example
```python
import databricks.koalas as ks

data = ks.DataFrame({'a':[1, 2, None], 'b':[1, 2, 3]})

# Expected behaviour
data['a'].fillna('Missing', inplace=True)
print(data)
#          a  b
# 0      1.0  1
# 1      2.0  2
# 2  Missing  3

# Unexpected behaviour
data['b'].fillna('Missing', inplace=True)
print(data)
#    b
# 0  1
# 1  2
# 2  3
```

As you can see, `data` has been replaced with `data['b']`. This is does not happen with `pandas` as shown below. Again this only happens with `inplace=True`.

## Pandas example
```python
import pandas as pd

data = pd.DataFrame({'a':[1, 2, None], 'b':[1, 2, 3]})

# Expected behaviour
data['a'].fillna('Missing', inplace=True)
print(data)
#          a  b
# 0        1  1
# 1        2  2
# 2  Missing  3

# Expected behaviour
data['b'].fillna('Missing', inplace=True)
print(data)
#          a  b
# 0        1  1
# 1        2  2
# 2  Missing  3
```

There also appears to be a separate issue of `koalas` converting the integers in `df.a` to floating point numbers which `pandas` doesn't do. However I'm not sure about this. Is this expected? Thanks

Ubuntu 18.04
python 3.7.6
koalas==1.2.0
pandas==1.0.5


Discussion:
Thanks for the report! I'll handle it.

Resolving patch:
diff --git a/databricks/koalas/series.py b/databricks/koalas/series.py
--- a/databricks/koalas/series.py
+++ b/databricks/koalas/series.py
@@ -3826,7 +3826,7 @@ def copy(self, deep=None) -> "Series":
         b    2
         dtype: int64
         """
-        return first_series(DataFrame(self._internal))
+        return self._kdf.copy()._kser_for(self._column_label)
 
     def mode(self, dropna=True) -> "Series":
         """


Test patch:
diff --git a/databricks/koalas/tests/test_series.py b/databricks/koalas/tests/test_series.py
--- a/databricks/koalas/tests/test_series.py
+++ b/databricks/koalas/tests/test_series.py
@@ -290,6 +290,21 @@ def test_fillna(self):
         self.assert_eq(kser.fillna(method="ffill"), pser.fillna(method="ffill"))
         self.assert_eq(kser.fillna(method="bfill"), pser.fillna(method="bfill"))
 
+        # inplace fillna on non-nullable column
+        pdf = pd.DataFrame({"a": [1, 2, None], "b": [1, 2, 3]})
+        kdf = ks.from_pandas(pdf)
+
+        pser = pdf.b
+        kser = kdf.b
+
+        self.assert_eq(kser.fillna(0), pser.fillna(0))
+        self.assert_eq(kser.fillna(np.nan).fillna(0), pser.fillna(np.nan).fillna(0))
+
+        kser.fillna(0, inplace=True)
+        pser.fillna(0, inplace=True)
+        self.assert_eq(kser, pser)
+        self.assert_eq(kdf, pdf)
+
     def test_dropna(self):
         pdf = pd.DataFrame({"x": [np.nan, 2, 3, 4, np.nan, 6]})
         kdf = ks.from_pandas(pdf)
Repository: databricks/koalas

Issue:
`astype()` handling of `None`
`koalas` appears to ignore `None` values when converting the type of a `DataFrame` or `Series` using `.astype()` instead of converting them which is the `pandas` behaviour. For instance when converting to `str`, in `pandas` `None` becomes `'None'` but this does not happen in `koalas`. Example below shown by using `sorted()` which can't handle `None`:

## Example
```python
import databricks.koalas as ks
import pandas as pd

data = pd.Series(['a', 'b', 'c', None])
sorted(data.astype(str).tolist())
# => ['None', 'a', 'b', 'c']

data = ks.Series(['a', 'b', 'c', None])
sorted(data.astype(str).tolist())
# ---------------------------------------------------------------------------
# TypeError                                 Traceback (most recent call last)
# <ipython-input-11-493f99e0fb6f> in <module>
#       1 data = ks.Series(['a', 'b', 'c', None])
# ----> 2 sorted(data.astype(str).tolist())
# 
# TypeError: '<' not supported between instances of 'NoneType' and 'str'
```

Is there a reason for this or can we bring this in line with `pandas`? Thanks

Ubuntu 18.04
python 3.7.6
koalas==1.2.0
pandas==1.0.5


Discussion:
@itholic can you take a look when you find some time?

Resolving patch:
diff --git a/databricks/koalas/series.py b/databricks/koalas/series.py
--- a/databricks/koalas/series.py
+++ b/databricks/koalas/series.py
@@ -1027,6 +1027,10 @@ def astype(self, dtype) -> "Series":
                 scol = F.when(self.spark.column.isNull(), F.lit(False)).otherwise(
                     self.spark.column.cast(spark_type)
                 )
+        elif isinstance(spark_type, StringType):
+            scol = F.when(self.spark.column.isNull(), str(None)).otherwise(
+                self.spark.column.cast(spark_type)
+            )
         else:
             scol = self.spark.column.cast(spark_type)
         return self._with_new_scol(scol)


Test patch:
diff --git a/databricks/koalas/tests/test_series.py b/databricks/koalas/tests/test_series.py
--- a/databricks/koalas/tests/test_series.py
+++ b/databricks/koalas/tests/test_series.py
@@ -1123,6 +1123,9 @@ def test_astype(self):
         kser = ks.Series(pser)
 
         self.assert_eq(kser.astype(bool), pser.astype(bool))
+        # TODO: restore after pandas 1.1.4 is released.
+        # self.assert_eq(kser.astype(str).tolist(), pser.astype(str).tolist())
+        self.assert_eq(kser.astype(str).tolist(), ["hi", "hi ", " ", " \t", "", "None"])
         self.assert_eq(kser.str.strip().astype(bool), pser.str.strip().astype(bool))
 
         pser = pd.Series([True, False, None], name="x")
Repository: databricks/koalas

Issue:
plot does not work in Ubuntu
Plotting with `matplotlib` 3.3.0+ causes an exception because of Pillow dependency added at https://github.com/matplotlib/matplotlib/commit/370e9a2d5d9e637abc90b3270d368642c69f66c6#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7 on Ubuntu:

```python
import databricks.koalas as ks
df = ks.DataFrame({'x': [1, 2], 'y': [3, 4], 'z': [5, 6]})
df.plot...
```

```
ImportError: /.../lib/python3.8/site-packages/PIL/_imaging.cpython-38-x86_64-linux-gnu.so: ELF load command address/offset not properly aligned
```


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -57,7 +57,7 @@
         'pandas>=0.23.2',
         'pyarrow>=0.10',
         'numpy>=1.14',
-        'matplotlib>=3.0.0',
+        'matplotlib>=3.0.0,<3.3.0',
     ],
     author="Databricks",
     author_email="koalas@databricks.com",


Test patch: