Repository: pandas-dev/pandas
Pull request: TST: GH32822 data types in frame manip
- [ ] closes #32822
- [ ] [1 Test added and passed]
- [ ] All [code checks passed]


Problem statement:
dtype change when doing operations on dataframes
#### Code Sample

```python
    In [1]: import pandas as pd
    In [2]: import numpy as np
    In [3]: s = pd.Series([1,2,np.nan], dtype='Int64')
    In [4]: t = pd.Series([1,2,3], dtype='Int64')
    In [5]: s-t
    Out[5]:
    0       0
    1       0
    2    <NA>
    dtype: Int64
    In [6]: (s-t).dtype
    Out[6]: Int64Dtype()
    In [7]: s.to_frame() - t.to_frame()
    Out[7]:
          0
    0     0
    1     0
    2  <NA>
    In [8]: (s.to_frame() - t.to_frame())[0].dtype
    Out[8]: dtype('O')
```
#### Problem description

The Int64 dtype is changed when doing operations on two dataframes whereas it is kept (expected behavior) on series operations.

#### Expected Output
```
    # In [8]: (s.to_frame() - t.to_frame())[0].dtype
    # Out[8]: dtype('Int64Dtype()')
```
#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.4.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 18.7.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : en_GB.UTF-8

pandas           : 1.0.3
numpy            : 1.18.2
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 46.0.0
Cython           : None
pytest           : 5.4.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.13.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.16.0
pytables         : None
pytest           : 5.4.1
pyxlsb           : None
s3fs             : 0.4.0
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : 0.8.6
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None
</details>

Hints from issue discussion:
This is because `sub` function on DataFrames is executed differently when DataFrames have just one column. For DataFrames with more columns result is calculated sequentially like on sequence of Series.
````
>>> s = pd.DataFrame({'a': [1, 2, 3],'b':[1, 2, 3]}, dtype='Int64')
>>> t = pd.DataFrame({'a': [1, 2, np.nan],'b':[1, 2, np.nan]}, dtype='Int64')
>>> (s-t)['a'].dtype
Int64Dtype()
````
When DataFrames contain only one column, `sub` is calculated on numpy arrays and hence result array dtype is object (casting int and NA).

Function `ops.should_series_dispatch` is responsible for deciding whether function would be evaluated sequentially on each column or not. I tried adding binary operations like `add`, `sub`, `mul` etc. to the list of operations that are executed sequentially on DataFrame. Those binary operations work on Series (Series._binop), so it seems like they should work on DataFrames. However it results with multiple tests not passing. I'm not sure how to fix it or how to approach this differently.
I have few more observations. Series uses `pd.core.construction.extract_array` for binary operations, which casts Series of ints to IntegerArray. So that dtype is maintained. DataFrame casts values arrays to np.array and dtype is determined based on values (hence problems with `nan`).  
IMO best option would be forcing DataFrame to execute binary operations sequentially on its series, but I'm open to discussion on that. 

This looks to work on master now. Could use a test

```
In [11]:     In [1]: import pandas as pd
    ...:     In [2]: import numpy as np
    ...:     In [3]: s = pd.Series([1,2,np.nan], dtype='Int64')
    ...:     In [4]: t = pd.Series([1,2,3], dtype='Int64')
    ...:     In [5]: s-t
Out[11]:
0       0
1       0
2    <NA>
dtype: Int64

In [12]: In [6]: (s-t).dtype
Out[12]: Int64Dtype()

In [13]: In [7]: s.to_frame() - t.to_frame()
Out[13]:
      0
0     0
1     0
2  <NA>

In [14]: In [8]: (s.to_frame() - t.to_frame())[0].dtype
Out[14]: Int64Dtype()
```
I can fix it, can I make a pull request?
Sure go for it @daksh-intwala 
It works fine on 1.2.4 version
is issue still pending @mroeschke ? I can see that in the newer version we are not seeing this issue
We would need a unit test (or confirmation of an existing/similar one) to close out this issue to prevent regressions.
take

Commit message: TST: GH32822 data types in frame manip

Patch to review:
diff --git a/pandas/tests/frame/test_arithmetic.py b/pandas/tests/frame/test_arithmetic.py
index 7c33242192d2e..7ac1976f6c098 100644
--- a/pandas/tests/frame/test_arithmetic.py
+++ b/pandas/tests/frame/test_arithmetic.py
@@ -2040,3 +2040,13 @@ def _constructor_sliced(self):
 
     result = sdf + sdf
     tm.assert_frame_equal(result, expected)
+
+
+def test_frame_sub_dtype():
+    # GH 32822
+    s = Series([1, 2, np.nan], dtype='Int64')
+    t = Series([1, 2, 3], dtype='Int64')
+
+    expected = (s - t).dtype
+    result = (s.to_frame() - t.to_frame())[0].dtype
+    assert expected is result


Code review comments:
diff_hunk: @@ -2040,3 +2040,13 @@ def _constructor_sliced(self):
 
     result = sdf + sdf
     tm.assert_frame_equal(result, expected)
+
+
+def test_frame_sub_dtype():
+    # GH 32822
+    s = Series([1, 2, np.nan], dtype='Int64')
+    t = Series([1, 2, 3], dtype='Int64')
+
+    expected = (s - t).dtype
path: pandas/tests/frame/test_arithmetic.py
text: @user1:
Could you please validate the whole objects with tm.assert_series_equals and please use multiletter variables