BUG: upcasting on reshaping ops #13247
State: closed

- [x] closes #13247 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@jreback: nice catch, I did some manual testing of that before, and it passed... but extended testing yields different answers for these cases.  What is the "right" thing to do here?

Current behavior:

##### Series/DataFrame:

`int32 + float 16 (no nans) => float64`

##### Panel/Panel4D:

(no nans)
`int32 + float16 (no nans) => float16`

I think the right choice would be `float64`, but open to other ideas.  WDYT?

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13337?src=pr) is **84.23%**

> Merging [#13337](https://codecov.io/gh/pydata/pandas/pull/13337?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #13337   diff @@
==========================================
  Files           138        138          
  Lines         50712      50720     +8   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42714      42720     +6   
- Misses         7998       8000     +2   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [2e3c82e...4eb00e2](https://codecov.io/gh/pydata/pandas/compare/2e3c82e81bf00af157268a842a270a6181fcb168...4eb00e298d7f088360d7cbf7de909c52d357b748)

I think _any_ int/float casting can simply go to float64. Yes you can get creative, e.g. I think
int16/int8 & float32 -> float32 but prob not worth it. (and float16 is barely used / supported)

I updated the PR... but see below:

OK, so for the panel case, it seems like we go into mixed types (`x._is_mixed_type == True`).  So there are multiple block managers, each item of the panel is of different type.

So in order to really fix it, one has to look at `as_matrix` function.  And in the end it comes down to this:

The function here: https://github.com/pydata/pandas/blob/4de83d25d751d8ca102867b2d46a5547c01d7248/pandas/core/internals.py#L4390-L4449 , takes in float and int blocks, and then falls through to the last line:

`return _lcd_dtype(counts[FloatBlock] + counts[SparseBlock])`

The above lead to this change in the PR:

```
elif have_int and have_float and not have_complex:
        return np.dtype('float64')
```

I was also able to get rid of multiple ifs (the ones you wanted to move to a function), so the code is a bit more cleaner.  The side effect is that an empty DataFrame's dtype is object, instead of float64 (in some cases).  I am guessing that we should be consistent and perhaps always have a type `O` DataFrame for empty DataFrames (instead of `float64`).  Thoughts?

```
In [1]: import pandas as pd

In [2]: df = pd.DataFrame(columns=['A', 'B'])

In [3]: df.values.dtype
Out[3]: dtype('O')

In [4]: df2 = pd.DataFrame()

In [5]: df2.values.dtype
Out[5]: dtype('float64')
```

@jennolsen84 `[5]` is a bit undefined. It does currently do this on master as well. Not averse to changing this to `object` . This might break some tests though.

can you rebase / update?

I have hit this bug through Dask DataFrame repartition, so I would love to see this issue fixed.

Updated: I have tried to install pandas from the fork and it worked for me! Thank you!

@jennolsen84 thanks for this. too many open PR's and haven't gone thru in a while. if you can update we can merge soon.

@jennolsen84 can you rebase
@jennolsen84 can you rebase
Hi, I rebased this code to latest commit on master, but there were merge conflicts. So, I rewrote them and I want to do a pr on this page.
How do I do this or do I need to create a new pr?
@jaehoonhwang what you do is branch off of current master. Then pull in these commits, fixing any conflicts. You then push to your branch and treat like any other fix. The only trick here is to cherry-pick the original commits off of this PR.
@jaehoonhwang You cannot replace others PR with yours, so just create a new PR and reference this PR in it.
closing in favor of #15594 (rebase of this PR).