Repository: pandas-dev/pandas
Pull request: TST: Added iloc tests for casting from object to numeric dtypes for duplicate columns
- [x] closes #22035
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Problem statement:
Unable to change dtype of a column after concat and duplicate column names
#### Code Sample, a copy-pastable example if possible

```python
import pandas

a = pandas.DataFrame({'a': ['0'], 'b': ['str']})

print(a.dtypes)

b = pandas.concat([a, pandas.DataFrame({'b': ['str2']})], axis=1)

print(b.dtypes)

b.iloc[:, 0] = [int(v) for v in a.iloc[:, 0]]

print(b.dtypes)
```

#### Problem description

The issue seems to be that if there is a DataFrame with duplicate column names, I cannot modify an (unrelated) column and convert its dtype from object to int.

#### Expected Output

The final print should be:

```
a     int64
b    object
b    object
dtype: object
```

And not:

```
a    object
b    object
b    object
dtype: object
```

It is interesting that if I change concat line to (see renaming of column `b` to `c`):

```
b = pandas.concat([a, pandas.DataFrame({'c': ['str2']})], axis=1)
```

Then the output is correctly:

```
a     int64
b    object
c    object
dtype: object
```

So it seems it has something with the fact that there are duplicate column names on other two columns, not the one being changed.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-46-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.3
pytest: None
pip: 18.0
setuptools: 40.0.0
Cython: None
numpy: 1.15.0
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
I'm working on this and found (I think) a version of this bug:
```
import pandas
df = pandas.DataFrame([['0', '0', 1]], columns=['A', 'B', 'B'])
df.iloc[:, 0] = df.iloc[:, 0]
```
And stacktrace:
```
../../core/indexing.py:188: in __setitem__
    self._setitem_with_indexer(indexer, value)
../../core/indexing.py:599: in _setitem_with_indexer
    setter(labels[0], value)
../../core/indexing.py:537: in setter
    self.obj[item] = s
../../core/frame.py:3261: in __setitem__
    self._set_item(key, value)
../../core/frame.py:3337: in _set_item
    NDFrame._set_item(self, key, value)
../../core/generic.py:3177: in _set_item
    self._data.set(key, value)
../../core/internals/managers.py:1021: in set
    group=True):
pandas/_libs/internals.pyx:463: in get_blkno_placements
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   ValueError: Buffer has wrong number of dimensions (expected 1, got 0)

pandas/_libs/internals.pyx:361: ValueError
```
I think this is caused by pandas putting columns with the same name (but different dtype) into the same block, so here we're assigning a single column into a 2 column block?
This works now. Needs tests
Hey! It's my first issue. Can I work on it?
Hey,

yes, for sure.
Hoping to contribute if this is still open. I was reading in https://github.com/pandas-dev/pandas/blob/master/pandas/tests/indexing/test_iloc.py#L1182, does this test already cover what's been described in this issue? If not, I am happy to write one that does.
Hm not really, I think we would like something casting from object to numeric dtypes

would be great if you could provide a pr:)
@phofl opened a PR where a test is added in `test_iloc.py`.

Commit message: fix spacing

Patch to review:
diff --git a/pandas/tests/indexing/test_iloc.py b/pandas/tests/indexing/test_iloc.py
index 853c7079a3c1b..ff74705ad09a6 100644
--- a/pandas/tests/indexing/test_iloc.py
+++ b/pandas/tests/indexing/test_iloc.py
@@ -1186,6 +1186,17 @@ def test_iloc_setitem_series_duplicate_columns(self):
         df.iloc[:, 0] = df.iloc[:, 0].astype(np.float64)
         assert df.dtypes.iloc[2] == np.int64
 
+    def test_iloc_setitem_dtypes_duplicate_columns(self):
+        # GH#22035
+        df = DataFrame([["0", "str", "1.2"]], columns=["a", "b", "b"])
+        df.iloc[:, 0] = df.iloc[:, 0].astype("int64")
+        expected_int = DataFrame([[0, "str", "1.2"]], columns=["a", "b", "b"])
+        tm.assert_frame_equal(df, expected_int)
+        
+        df.iloc[:, 2] = df.iloc[:, 2].astype("float")
+        expected_float = DataFrame([[0, "str", 1.2]], columns=["a", "b", "b"])
+        tm.assert_frame_equal(df, expected_float)
+
 
 class TestILocCallable:
     def test_frame_iloc_getitem_callable(self):


Code review comments:
diff_hunk: @@ -1186,6 +1186,17 @@ def test_iloc_setitem_series_duplicate_columns(self):
         df.iloc[:, 0] = df.iloc[:, 0].astype(np.float64)
         assert df.dtypes.iloc[2] == np.int64
 
+    def test_iloc_setitem_dtypes_duplicate_columns(self):
+        # GH#22035
+        df = DataFrame([["0", "str", "1.2"]], columns=["a", "b", "b"])
path: pandas/tests/indexing/test_iloc.py
text: @user1:
please paramterize over dtypes

@author:
My apologies for missing the parametrization earlier