BUG: fix DataFrame.apply returning wrong result when dealing with dtype (#28773)
State: closed

- [x] closes #28773
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The DataFrame.apply was sometimes returning wrong result when we passed
function, that was dealing with dtypes. It was caused by retrieving
the DataFrame.values of whole DataFrame, and applying the function
to it: values are represented by NumPy array, which has one
type for all data inside. It sometimes caused treating objects
in DataFrame as if they had one common type. What's worth mentioning,
the problem only existed, when we were applying function on columns.

The implemented solution "cuts" the DataFrame by columns and applies
function to each part, as it was whole DataFrame. After that, all
results are concatenated into final result on whole DataFrame.
The "cuts" are done in following way: the first column is taken, and
then we iterate through next columns and take them into first cut
while their dtype is identical as in the first column. The process
is then repeated for the rest of DataFrame.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Hello @Reksbril! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/apply.py`](https://github.com/pandas-dev/pandas/blob/1f81b77705495e4fb4ebf4cc0dacce2312cc3270/pandas/core/apply.py):

> [Line 290:89](https://github.com/pandas-dev/pandas/blob/1f81b77705495e4fb4ebf4cc0dacce2312cc3270/pandas/core/apply.py#L290): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (101 > 88 characters)





##### Comment last updated at 2020-04-17 17:15:35 UTC
>  The rest of changes are result of black pandas reformatting.

I guess you're using a different version of black than our CI is?  Regardless, can you revert non-relevant changes so we can focus on the important parts

@jbrockmendel done.
This is a lot of new code.  A couple things:

- Assuming this is the correct solution to this problem, the new code would belong in core.apply
- the "cuts" are a lot like the `DataFrame._data.blocks` that already exist.  Usually we discourage people from bothering with blocks but it is an option
- The hard part of the problem is identifying when you should be operating column-wise (i.e. not call .values).  Finding an elegant-ish way of doing that will be key here.
@jreback 
I've resolved a minor problem which caused most of tests to fail.

>     * the "cuts" are a lot like the `DataFrame._data.blocks` that already exist.  Usually we discourage people from bothering with blocks but it is an option
What's the benefit from using `DataFrame._data.blocks` instead of current solution? I'm just using DataFrame.iloc to get only part of DF and it seems to be very "clean" approach.
>     * The hard part of the problem is identifying when you should be operating column-wise (i.e. not call .values).  Finding an elegant-ish way of doing that will be key here.
I'm not sure if we can determine that at all. The obvious thing we can check is if all dtypes are the same (I think it's already done in current code and some specific apply variant is used in that case). As the functions given in input can be very complicated, the only solution to this problem I see, is to give some experimental input to the function, to determine if it relies on dtypes. However, that would be neither elegant nor easy to do (if even possible).


@Reksbril you're going to have to go through `frame_apply` and try to adapt the existing machinery rather than pile a new layer on top of it.  Within FrameApply there is a `apply_series_generator` that exists precisely to have a function operate column-by-column.  So a good solution to this problem would be to edit FrameApply.apply_standard so that this case goes through the apply_series_generator route.
@jreback ping
@jreback ping
@Reksbril needs rebase, ill take a look
@jreback ping
@Reksbril looks like a merge conflict and CI failure - can you fix those up and someone will take a look
@Reksbril can you rebase and address comments
Yeah, I will. I'm sorry for long delay.
I won't be able to finish this for a few more months. The current version is the best I could do right now.
> I won't be able to finish this for a few more months. The current version is the best I could do right now.

Thanks @Reksbril for working on this. closing to clear queue for now. ping when you want to continue and will reopen.