Repository: databricks/koalas

Issue:
concat(axis=1) currently is not supported
Concatenating along the columns is a common use case but currently isn't supported in koalas. We can achieve this by joining the dataframes by their index.

```
>>> df4 = pd.DataFrame([['bird', 'polly'], ['monkey', 'george']],
...                    columns=['animal', 'name'])
>>> pd.concat([df1, df4], axis=1)
  letter  number  animal    name
0      a       1    bird   polly
1      b       2  monkey  george
```
Add axis=1 for concat
#625


Discussion:
I think this can be done with `join(index_column)` Are you interested in submitting a PR?
Yes, I'll have a PR submitted in a few days!
That's very nice!
Hello, any news on this?

@hsubbaraj, are you working on this?
I want to learn koalas with databricks
please go ahead.
Hi
Thanks
is concat(axis=1) supported in koalas yet?
You can work around as below:

```python
>>> import databricks.koalas as ks
>>> df1 = ks.DataFrame([['bird', 'polly'], ['monkey', 'george']],
...                    columns=['animal', 'name'])
>>> df2 = ks.DataFrame([[1, 2], [3, 4]],
...                    columns=['num', 'int'])
>>> ks.set_option('compute.ops_on_diff_frames', True)
>>> df1['num'] = df2.num
>>> df1['int'] = df2.int
>>> df1
   animal    name  num  int
0    bird   polly    1    2
1  monkey  george    3    4
```

In fact, are you interested in creating a PR like the above? cc @itholic, @charlesdong1991, @harupy 
Nice! Thanks for working on this!
haven't come up with a nice and robust solution for allowing `ingore_index` to be `True`, any suggestions? @ueshin 
# [Codecov](https://codecov.io/gh/databricks/koalas/pull/1009?src=pr&el=h1) Report
> Merging [#1009](https://codecov.io/gh/databricks/koalas/pull/1009?src=pr&el=desc) into [master](https://codecov.io/gh/databricks/koalas/commit/e7170cb260156073fd3712b4bc921e14a9fb2221?src=pr&el=desc) will **decrease** coverage by `0.1%`.
> The diff coverage is `96.15%`.

[![Impacted file tree graph](https://codecov.io/gh/databricks/koalas/pull/1009/graphs/tree.svg?width=650&token=EtnrpE8dVx&height=150&src=pr)](https://codecov.io/gh/databricks/koalas/pull/1009?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #1009      +/-   ##
=========================================
- Coverage    95.2%   95.1%   -0.11%     
=========================================
  Files          34      34              
  Lines        6883    6779     -104     
=========================================
- Hits         6553    6447     -106     
- Misses        330     332       +2
```


| [Impacted Files](https://codecov.io/gh/databricks/koalas/pull/1009?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [databricks/koalas/namespace.py](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvbmFtZXNwYWNlLnB5) | `87.79% <96.15%> (+0.44%)` | :arrow_up: |
| [databricks/koalas/indexing.py](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvaW5kZXhpbmcucHk=) | `94.38% <0%> (-0.51%)` | :arrow_down: |
| [databricks/koalas/base.py](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvYmFzZS5weQ==) | `94.9% <0%> (-0.42%)` | :arrow_down: |
| [databricks/koalas/frame.py](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvZnJhbWUucHk=) | `96.49% <0%> (-0.26%)` | :arrow_down: |
| [databricks/koalas/series.py](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvc2VyaWVzLnB5) | `96.41% <0%> (-0.13%)` | :arrow_down: |
| [databricks/koalas/utils.py](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvdXRpbHMucHk=) | `98.12% <0%> (-0.05%)` | :arrow_down: |
| [databricks/koalas/missing/frame.py](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvbWlzc2luZy9mcmFtZS5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [databricks/koalas/missing/indexes.py](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvbWlzc2luZy9pbmRleGVzLnB5) | `100% <0%> (ø)` | :arrow_up: |
| [databricks/koalas/missing/window.py](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree#diff-ZGF0YWJyaWNrcy9rb2FsYXMvbWlzc2luZy93aW5kb3cucHk=) | `100% <0%> (ø)` | :arrow_up: |
| ... and [4 more](https://codecov.io/gh/databricks/koalas/pull/1009/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/databricks/koalas/pull/1009?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/databricks/koalas/pull/1009?src=pr&el=footer). Last update [e7170cb...f672805](https://codecov.io/gh/databricks/koalas/pull/1009?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Many thanks for your comments @HyukjinKwon 

I am very sorry that I have been very busy recently and don't have time for my existing PRs. I will try my best to update the PR following your comments this weekend!

Thanks again for your understanding and patience!
Hey, @charlesdong1991 no worries at all. Thanks for trying this change out.
### Softagram Impact Report for pull/1009 (head commit: f672805b2ca0)


### :star: Change Overview


![Showing the changed files, dependency changes and the impact - click for full size](https://opensource.softagram.com/cdn/impact/dd6f973b-a9da-477d-8f74-1a29916573a5_koalas_1009_impact_change_graph_0v61tQ1Ou741y7fSQyBwNnkJr54aiP.png)
*(Open in [Softagram Desktop](https://softagram.com/desktop) for full details)*






### :page_facing_up: Full report

* Permalink: [Full report for pull/1009](https://opensource.softagram.com/dashboard/teams/7ff2e81c-61e0-4605-b4ac-d496bc4c37c7/projects/dd6f973b-a9da-477d-8f74-1a29916573a5/impact-reports/f1070b54-70dc-4379-9d67-747f212fdc39)

#
[Impact Report explained.](http://help.softagram.com/en/articles/2847546-pull-merge-request-report-explained) Give feedback on this report to support@softagram.com


Resolving patch:
diff --git a/databricks/koalas/frame.py b/databricks/koalas/frame.py
--- a/databricks/koalas/frame.py
+++ b/databricks/koalas/frame.py
@@ -9655,34 +9655,8 @@ def __setitem__(self, key, value):
             isinstance(value, DataFrame) and value is not self
         ):
             # Different Series or DataFrames
-            if isinstance(value, Series):
-                value = value.to_frame()
-            else:
-                assert isinstance(value, DataFrame), type(value)
-                value = value.copy()
-            level = self._internal.column_labels_level
-
-            value.columns = pd.MultiIndex.from_tuples(
-                [
-                    tuple([name_like_string(label)] + ([""] * (level - 1)))
-                    for label in value._internal.column_labels
-                ]
-            )
-
-            if isinstance(key, str):
-                key = [(key,)]
-            elif isinstance(key, tuple):
-                key = [key]
-            else:
-                key = [k if isinstance(k, tuple) else (k,) for k in key]
-
-            if any(len(label) > level for label in key):
-                raise KeyError(
-                    "Key length ({}) exceeds index depth ({})".format(
-                        max(len(label) for label in key), level
-                    )
-                )
-            key = [tuple(list(label) + ([""] * (level - len(label)))) for label in key]
+            key = self._index_normalized_label(key)
+            value = self._index_normalized_frame(value)
 
             def assign_columns(kdf, this_column_labels, that_column_labels):
                 assert len(key) == len(that_column_labels)
@@ -9707,6 +9681,54 @@ def assign_columns(kdf, this_column_labels, that_column_labels):
 
         self._internal = kdf._internal
 
+    def _index_normalized_label(self, labels):
+        """
+        Returns a label that is normalized against the current column index level.
+        For example, the key "abc" can be ("abc", "", "") if the current Frame has
+        a multi-index for its column
+        """
+        level = self._internal.column_labels_level
+
+        if isinstance(labels, str):
+            labels = [(labels,)]
+        elif isinstance(labels, tuple):
+            labels = [labels]
+        else:
+            labels = [k if isinstance(k, tuple) else (k,) for k in labels]
+
+        if any(len(label) > level for label in labels):
+            raise KeyError(
+                "Key length ({}) exceeds index depth ({})".format(
+                    max(len(label) for label in labels), level
+                )
+            )
+        return [tuple(list(label) + ([""] * (level - len(label)))) for label in labels]
+
+    def _index_normalized_frame(self, kser_or_kdf):
+        """
+        Returns a frame that is normalized against the current column index level.
+        For example, the name in `pd.Series([...], name="abc")` can be can be
+        ("abc", "", "") if the current DataFrame has a multi-index for its column
+        """
+
+        from databricks.koalas.series import Series
+
+        level = self._internal.column_labels_level
+        if isinstance(kser_or_kdf, Series):
+            kdf = kser_or_kdf.to_frame()
+        else:
+            assert isinstance(kser_or_kdf, DataFrame), type(kser_or_kdf)
+            kdf = kser_or_kdf.copy()
+
+        kdf.columns = pd.MultiIndex.from_tuples(
+            [
+                tuple([name_like_string(label)] + ([""] * (level - 1)))
+                for label in kdf._internal.column_labels
+            ]
+        )
+
+        return kdf
+
     def __getattr__(self, key: str) -> Any:
         if key.startswith("__"):
             raise AttributeError(key)
diff --git a/databricks/koalas/namespace.py b/databricks/koalas/namespace.py
--- a/databricks/koalas/namespace.py
+++ b/databricks/koalas/namespace.py
@@ -45,7 +45,13 @@
 
 from databricks import koalas as ks  # For running doctests and reference resolution in PyCharm.
 from databricks.koalas.base import IndexOpsMixin
-from databricks.koalas.utils import default_session, name_like_string, scol_for, validate_axis
+from databricks.koalas.utils import (
+    default_session,
+    name_like_string,
+    scol_for,
+    validate_axis,
+    align_diff_frames,
+)
 from databricks.koalas.frame import DataFrame, _reduce_spark_multi
 from databricks.koalas.internal import _InternalFrame
 from databricks.koalas.typedef import pandas_wraps
@@ -107,6 +113,9 @@ def from_pandas(pobj: Union["pd.DataFrame", "pd.Series"]) -> Union["Series", "Da
         raise ValueError("Unknown data type: {}".format(type(pobj)))
 
 
+_range = range  # built-in range
+
+
 def range(
     start: int, end: Optional[int] = None, step: int = 1, num_partitions: Optional[int] = None
 ) -> DataFrame:
@@ -1539,11 +1548,11 @@ def concat(objs, axis=0, join="outer", ignore_index=False):
     objs : a sequence of Series or DataFrame
         Any None objects will be dropped silently unless
         they are all None in which case a ValueError will be raised
-    axis : {0/'index'}, default 0
+    axis : {0/'index', 1/'columns'}, default 0
         The axis to concatenate along.
     join : {'inner', 'outer'}, default 'outer'
-        How to handle indexes on other axis(es)
-    ignore_index : boolean, default False
+        How to handle indexes on other axis (or axes).
+    ignore_index : bool, default False
         If True, do not use the index values along the concatenation axis. The
         resulting axis will be labeled 0, ..., n - 1. This is useful if you are
         concatenating objects where the concatenation axis does not have
@@ -1552,14 +1561,17 @@ def concat(objs, axis=0, join="outer", ignore_index=False):
 
     Returns
     -------
-    concatenated : object, type of objs
+    object, type of objs
         When concatenating all ``Series`` along the index (axis=0), a
         ``Series`` is returned. When ``objs`` contains at least one
-        ``DataFrame``, a ``DataFrame`` is returned.
+        ``DataFrame``, a ``DataFrame`` is returned. When concatenating along
+        the columns (axis=1), a ``DataFrame`` is returned.
 
     See Also
     --------
-    DataFrame.merge
+    Series.append : Concatenate Series.
+    DataFrame.join : Join DataFrames using indexes.
+    DataFrame.merge : Merge DataFrames by indexes or columns.
 
     Examples
     --------
@@ -1645,6 +1657,17 @@ def concat(objs, axis=0, join="outer", ignore_index=False):
     1      b       2
     0      c       3
     1      d       4
+
+    >>> df4 = ks.DataFrame([['bird', 'polly'], ['monkey', 'george']],
+    ...                    columns=['animal', 'name'])
+
+    Combine with column axis.
+
+    >>> ks.concat([df1, df4], axis=1)
+      letter  number  animal    name
+    0      a       1    bird   polly
+    1      b       2  monkey  george
+
     """
     if isinstance(objs, (DataFrame, IndexOpsMixin)) or not isinstance(
         objs, Iterable
@@ -1655,10 +1678,6 @@ def concat(objs, axis=0, join="outer", ignore_index=False):
             '"{name}"'.format(name=type(objs).__name__)
         )
 
-    axis = validate_axis(axis)
-    if axis != 0:
-        raise NotImplementedError('axis should be either 0 or "index" currently.')
-
     if len(objs) == 0:
         raise ValueError("No objects to concatenate")
     objs = list(filter(lambda obj: obj is not None, objs))
@@ -1674,6 +1693,63 @@ def concat(objs, axis=0, join="outer", ignore_index=False):
                 "and ks.DataFrame are valid".format(name=type(objs).__name__)
             )
 
+    axis = validate_axis(axis)
+    if axis == 1:
+        if isinstance(objs[0], ks.Series):
+            concat_kdf = objs[0].to_frame()
+        else:
+            concat_kdf = objs[0]
+
+        with ks.option_context("compute.ops_on_diff_frames", True):
+
+            def resolve_func(kdf, this_column_labels, that_column_labels):
+                duplicated_names = set(
+                    this_column_label[1:] for this_column_label in this_column_labels
+                ).intersection(
+                    set(that_column_label[1:] for that_column_label in that_column_labels)
+                )
+                assert (
+                    len(duplicated_names) > 0
+                ), "inner or full join type does not include non-common columns"
+                pretty_names = [name_like_string(column_label) for column_label in duplicated_names]
+                raise ValueError(
+                    "Labels have to be unique; however, got " "duplicated labels %s." % pretty_names
+                )
+
+            for kser_or_kdf in objs[1:]:
+                if isinstance(kser_or_kdf, Series):
+                    # TODO: there is a corner case to optimize - when the series are from
+                    #   the same DataFrame.
+                    that_kdf = kser_or_kdf.to_frame()
+                else:
+                    that_kdf = kser_or_kdf
+
+                this_index_level = concat_kdf._internal.column_labels_level
+                that_index_level = that_kdf._internal.column_labels_level
+
+                if this_index_level > that_index_level:
+                    concat_kdf = that_kdf._index_normalized_frame(concat_kdf)
+                if this_index_level < that_index_level:
+                    that_kdf = concat_kdf._index_normalized_frame(that_kdf)
+
+                if join == "inner":
+                    concat_kdf = align_diff_frames(
+                        resolve_func, concat_kdf, that_kdf, fillna=False, how="inner",
+                    )
+                elif join == "outer":
+                    concat_kdf = align_diff_frames(
+                        resolve_func, concat_kdf, that_kdf, fillna=False, how="full",
+                    )
+                else:
+                    raise ValueError(
+                        "Only can inner (intersect) or outer (union) join the other axis."
+                    )
+
+        if ignore_index:
+            concat_kdf.columns = list(map(str, _range(len(concat_kdf.columns))))
+
+        return concat_kdf
+
     # Series, Series ...
     # We should return Series if objects are all Series.
     should_return_series = all(map(lambda obj: isinstance(obj, Series), objs))


Test patch:
diff --git a/databricks/koalas/tests/test_namespace.py b/databricks/koalas/tests/test_namespace.py
--- a/databricks/koalas/tests/test_namespace.py
+++ b/databricks/koalas/tests/test_namespace.py
@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+import itertools
 
 import pandas as pd
 
@@ -92,10 +93,6 @@ def test_concat(self):
 
         self.assertRaisesRegex(ValueError, "All objects passed", lambda: ks.concat([None, None]))
 
-        self.assertRaisesRegex(
-            NotImplementedError, "axis should be either 0 or", lambda: ks.concat([kdf, kdf], axis=1)
-        )
-
         pdf3 = pdf.copy()
         kdf3 = kdf.copy()
 
@@ -128,3 +125,82 @@ def test_concat(self):
             r"Only can inner \(intersect\) or outer \(union\) join the other axis.",
             lambda: ks.concat([kdf, kdf4], join=""),
         )
+
+        self.assertRaisesRegex(
+            ValueError,
+            r"Only can inner \(intersect\) or outer \(union\) join the other axis.",
+            lambda: ks.concat([kdf, kdf4], join="", axis=1),
+        )
+
+        self.assertRaisesRegex(
+            ValueError,
+            r"Only can inner \(intersect\) or outer \(union\) join the other axis.",
+            lambda: ks.concat([kdf.A, kdf4.B], join="", axis=1),
+        )
+
+        self.assertRaisesRegex(
+            ValueError,
+            r"Labels have to be unique; however, got duplicated labels \['A'\].",
+            lambda: ks.concat([kdf.A, kdf4.A], join="inner", axis=1),
+        )
+
+    def test_concat_column_axis(self):
+        pdf1 = pd.DataFrame({"A": [0, 2, 4], "B": [1, 3, 5]}, index=[1, 2, 3])
+        pdf2 = pd.DataFrame({"C": [1, 2, 3], "D": [4, 5, 6]}, index=[1, 3, 5])
+        kdf1 = ks.from_pandas(pdf1)
+        kdf2 = ks.from_pandas(pdf2)
+
+        kdf3 = kdf1.copy()
+        kdf4 = kdf2.copy()
+        pdf3 = pdf1.copy()
+        pdf4 = pdf2.copy()
+
+        columns = pd.MultiIndex.from_tuples([("X", "A"), ("X", "B")])
+        pdf3.columns = columns
+        kdf3.columns = columns
+
+        columns = pd.MultiIndex.from_tuples([("X", "C"), ("X", "D")])
+        pdf4.columns = columns
+        kdf4.columns = columns
+
+        pdf5 = pd.DataFrame({"A": [0, 2, 4], "B": [1, 3, 5]}, index=[1, 2, 3])
+        pdf6 = pd.DataFrame({"C": [1, 2, 3]}, index=[1, 3, 5])
+        kdf5 = ks.from_pandas(pdf5)
+        kdf6 = ks.from_pandas(pdf6)
+
+        ignore_indexes = [True, False]
+        joins = ["inner", "outer"]
+
+        objs = [
+            ([kdf1.A, kdf2.C], [pdf1.A, pdf2.C]),
+            ([kdf1, kdf2.C], [pdf1, pdf2.C]),
+            ([kdf1.A, kdf2], [pdf1.A, pdf2]),
+            ([kdf1.A, kdf2.C], [pdf1.A, pdf2.C]),
+            ([kdf1.A, kdf1.A.rename("B")], [pdf1.A, pdf1.A.rename("B")]),
+            ([kdf3[("X", "A")], kdf4[("X", "C")]], [pdf3[("X", "A")], pdf4[("X", "C")]]),
+            ([kdf3, kdf4[("X", "C")]], [pdf3, pdf4[("X", "C")]]),
+            ([kdf3[("X", "A")], kdf4], [pdf3[("X", "A")], pdf4]),
+            ([kdf3, kdf4], [pdf3, pdf4]),
+            ([kdf3[("X", "A")], kdf3[("X", "B")]], [pdf3[("X", "A")], pdf3[("X", "B")]],),
+            (
+                [kdf3[("X", "A")], kdf3[("X", "B")].rename("ABC")],
+                [pdf3[("X", "A")], pdf3[("X", "B")].rename("ABC")],
+            ),
+            (
+                [kdf3[("X", "A")].rename("ABC"), kdf3[("X", "B")]],
+                [pdf3[("X", "A")].rename("ABC"), pdf3[("X", "B")]],
+            ),
+            ([kdf5, kdf6], [pdf5, pdf6]),
+            ([kdf6, kdf5], [pdf6, pdf5]),
+        ]
+
+        for ignore_index, join in itertools.product(ignore_indexes, joins):
+            for obj in objs:
+                kdfs, pdfs = obj
+                with self.subTest(ignore_index=ignore_index, join=join, objs=obj):
+                    actual = ks.concat(kdfs, axis=1, ignore_index=ignore_index, join=join)
+                    expected = pd.concat(pdfs, axis=1, ignore_index=ignore_index, join=join)
+                    self.assert_eq(
+                        repr(actual.sort_values(list(actual.columns)).reset_index(drop=True)),
+                        repr(expected.sort_values(list(expected.columns)).reset_index(drop=True)),
+                    )