Repository: pandas-dev/pandas
Pull request: BUG: concat not copying index and columns when copy=True
- [x] closes #29879
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
pd.concat with copy=True doesn't copy columns index
#### Code Sample

```python
import pandas as pd
print(pd.__version__)  # '0.25.3'

# simple frame with a name attribute for columns
df = pd.DataFrame({'a': [1.1], 'b': [2.2]})
df.columns.name = 'x'
print(df)
# x    a    b
# 0  1.1  2.2

# tile rows to a new frame, apparently using the copy of the first frame
df2 = pd.concat([df] * 2, copy=True)
print(df2)
# x    a    b
# 0  1.1  2.2
# 0  1.1  2.2

# clear the column name for the first frame
df.columns.name = None

# and observe the name for the second
assert df2.columns.name is None
print(df2)
#      a    b
# 0  1.1  2.2
# 0  1.1  2.2

# aha, it's the same object
assert df.columns is df2.columns
```

#### Problem description

It is expected that `copy=True` should make a copy of the `columns` index. However, the demonstration shows it's the same object. Modifications, such as the `name` property, are made to the source frame and the output frame created by `pd.concat`.

#### Expected Output

It is expected that `df2.columns` is a copy of the original `df.columns`. This would prevent (e.g.) changing the `name` property for both frames.

A workaround is to manually copy this property after `pd.concat`:
```python
df2 = pd.concat([df] * 2, copy=True)
df2.columns = df2.columns.copy()
```

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.5.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 63 Stepping 2, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 0.25.3
numpy            : 1.17.3
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.3.1
setuptools       : 42.0.1.post20191125
Cython           : 0.29.14
pytest           : 5.3.0
hypothesis       : None
sphinx           : 2.2.1
blosc            : None
feather          : None
xlsxwriter       : 1.2.6
lxml.etree       : 4.4.1
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : 7.9.0
pandas_datareader: None
bs4              : 4.8.1
bottleneck       : 1.3.1
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.1
matplotlib       : 3.1.1
numexpr          : 2.7.0
odfpy            : None
openpyxl         : 3.0.1
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : 1.3.11
tables           : 3.6.1
xarray           : None
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.2.6
</details>

Hints from issue discussion:
Another example of this that hits a slightly different code path:
```python
In [1]: import pandas as pd; pd.__version__
Out[1]: '0.26.0.dev0+958.g545d17529'

In [2]: df = pd.Series(list('abc'), name='foo').to_frame()

In [3]: df2 = pd.Series(list('xyz'), name='foo').to_frame()

In [4]: df.columns is df2.columns  # verifying not the same
Out[4]: False

In [5]: df3 = pd.concat([df, df2], copy=True)

In [6]: df.columns is df3.columns  # shouldn't be the same but are
Out[6]: True

In [7]: df2.columns is df3.columns  # this is okay
Out[7]: False
```
this analysis is not correct. Indexes are immutable, thus they are shared, concat ``copy=True`` does not guarantee a copy of an Index as its not necessary.

I would expect this
```
In [6]: df.columns is df3.columns  # shouldn't be the same but are
Out[6]: True
```
Further testing shows that this issue is more generic, not just `columns`. For instance with the other axis:
```python
df = pd.DataFrame({'a': [1.1], 'b': [2.2]})
df3 = pd.concat([df] * 2, copy=True, axis=1)
print(df3)
#      a    b    a    b
# 0  1.1  2.2  1.1  2.2
print(df3.index is df.index)  # True
print(df3.columns is df.columns)  # False
```
> Indexes are immutable, thus they are shared,

We should share the underlying data, but not the actual Index object? (so make a new Index object from the same data) 
As it's only the data that is considered immutable, not the full object (eg the name can change).

Commit message: BUG: updated details in a comment (GH29879)

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index 1cd325dad9f07..aaba575a31fa6 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -144,7 +144,7 @@ Reshaping
 - Bug in :meth:`DataFrame.pivot_table` when only MultiIndexed columns is set (:issue:`17038`)
 - Fix incorrect error message in :meth:`DataFrame.pivot` when ``columns`` is set to ``None``. (:issue:`30924`)
 - Bug in :func:`crosstab` when inputs are two Series and have tuple names, the output will keep dummy MultiIndex as columns. (:issue:`18321`)
-
+- Bug in :func:`concat` index and columns not copied when ``copy=True`` (:issue:`29879`)
 
 Sparse
 ^^^^^^
diff --git a/pandas/core/indexes/api.py b/pandas/core/indexes/api.py
index 4072d06b9427c..c252bdbb7ca15 100644
--- a/pandas/core/indexes/api.py
+++ b/pandas/core/indexes/api.py
@@ -63,7 +63,7 @@
 
 
 def get_objs_combined_axis(
-    objs, intersect: bool = False, axis=0, sort: bool = True
+    objs, intersect: bool = False, axis=0, sort: bool = True, copy: bool = False
 ) -> Index:
     """
     Extract combined index: return intersection or union (depending on the
@@ -81,13 +81,15 @@ def get_objs_combined_axis(
         The axis to extract indexes from.
     sort : bool, default True
         Whether the result index should come out sorted or not.
+    copy : bool, default False
+        If True, return a copy of the combined index.
 
     Returns
     -------
     Index
     """
     obs_idxes = [obj._get_axis(axis) for obj in objs]
-    return _get_combined_index(obs_idxes, intersect=intersect, sort=sort)
+    return _get_combined_index(obs_idxes, intersect=intersect, sort=sort, copy=copy)
 
 
 def _get_distinct_objs(objs: List[Index]) -> List[Index]:
@@ -105,7 +107,10 @@ def _get_distinct_objs(objs: List[Index]) -> List[Index]:
 
 
 def _get_combined_index(
-    indexes: List[Index], intersect: bool = False, sort: bool = False
+    indexes: List[Index],
+    intersect: bool = False,
+    sort: bool = False,
+    copy: bool = False,
 ) -> Index:
     """
     Return the union or intersection of indexes.
@@ -119,6 +124,8 @@ def _get_combined_index(
         calculate the union.
     sort : bool, default False
         Whether the result index should come out sorted or not.
+    copy : bool, default False
+        If True, return a copy of the combined index.
 
     Returns
     -------
@@ -143,6 +150,13 @@ def _get_combined_index(
             index = index.sort_values()
         except TypeError:
             pass
+
+    # GH 29879
+    if copy:
+        # No need for deep copy. The underlying data of Indexes are
+        # considered immutable so they can be shared
+        index = index.copy(deep=False)
+
     return index
 
 
diff --git a/pandas/core/reshape/concat.py b/pandas/core/reshape/concat.py
index 9528de36a3664..b42497b507e1f 100644
--- a/pandas/core/reshape/concat.py
+++ b/pandas/core/reshape/concat.py
@@ -517,7 +517,11 @@ def _get_new_axes(self) -> List[Index]:
     def _get_comb_axis(self, i: int) -> Index:
         data_axis = self.objs[0]._get_block_manager_axis(i)
         return get_objs_combined_axis(
-            self.objs, axis=data_axis, intersect=self.intersect, sort=self.sort
+            self.objs,
+            axis=data_axis,
+            intersect=self.intersect,
+            sort=self.sort,
+            copy=self.copy,
         )
 
     def _get_concat_axis(self) -> Index:
diff --git a/pandas/tests/reshape/test_concat.py b/pandas/tests/reshape/test_concat.py
index b3b2c5a05c6ad..5811f3bc196a1 100644
--- a/pandas/tests/reshape/test_concat.py
+++ b/pandas/tests/reshape/test_concat.py
@@ -2750,3 +2750,17 @@ def test_concat_sparse():
     )
     result = pd.concat([a, a], axis=1)
     tm.assert_frame_equal(result, expected)
+
+
+@pytest.mark.parametrize("test_series", [True, False])
+def test_concat_copy_index(test_series, axis):
+    # GH 29879
+    if test_series:
+        ser = Series([1, 2])
+        comb = concat([ser, ser], axis=axis, copy=True)
+        assert comb.index is not ser.index
+    else:
+        df = DataFrame([[1, 2], [3, 4]], columns=["a", "b"])
+        comb = concat([df, df], axis=axis, copy=True)
+        assert comb.index is not df.index
+        assert comb.columns is not df.columns


Code review comments:
diff_hunk: @@ -143,6 +150,13 @@ def _get_combined_index(
             index = index.sort_values()
         except TypeError:
             pass
+
+    # GH 29879
+    if copy:
+        # No need for deep copy. The underlying data of Indexes are
+        # considered immutable so they can be shared
+        index = index.copy(deep=False)
path: pandas/core/indexes/api.py
text: @user1:
you don't need the comment or deep=False which is the default.

@author:
ok then. I wanted to be a bit more explicit, but sure if you think it's unnecessary.

Have cleaned it up.