BUG: Allow 'apply' to be used with non-numpy-dtype DataFrames
State: closed

Addresses issue in #12244 in which a non-numpy-dtype for DataFrame.values causes a `TypeError` to be thrown in the `reduce == True` case for `DataFrame.apply`.  Resolved by first passing `DataFrame.values` through `Series` initialization and taking its `values` attribute, which is an `ndarray` and hence will have a valid `dtype`.  Note that the output of `apply` will still have the original `dtype`.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timezones
description: Timezone data dtype

Comments:
In light of #12291, I think there could be a <i>better</i> way to handle the `Series` dummy creation that will also address the issue in #12244 that I am testing right now.

see my very first comment
there is a way to do this in _sanitize_array but needs to be exposed as a private function

See my response to that comment of yours.  `_sanitize_array`, at least when I tried implementing it, did not do what I needed it to do, which is return an object with a valid `numpy` dtype.

Also, there is a possibility that the `numpy` array creation is not even necessary if we can just create an empty `Series` object right from the get go, rendering any of this discussion about array sanitization (which in some ways implies that the array is 'dirty' when it really isn't in the context of `scipy`) moot.

you don't need to make this use numpy at sll

Yes, exactly.  That's what I am testing right now.  Removing that `np.empty` dependency would be even better than what I had initially done.

Test failures due to the fact that `Series` initialization without any data for numerical types all default to `np.float64`.  Reverting changes.  See also my comment in #12291.

you can do something like: `getattr(self, 'dtype', self.values.dtype)`

I'm confused as to why I would need to do this.  Isn't that just more complicated?  Also, using the code provided in the issue, it just defaults to `self.values.dtype`, which is what I don't want.

I think the issue that the code in `reduce.pyx` cannot deal with the non-numpy dtypes. In theory they should but they are heavily numpy based currently. I think https://github.com/pydata/pandas/issues/11970 will eventually have to deal with this. Not sure exactly how we want to track a test/fix like this. IOW where it is makes sense, but want to remember to deal with this issue.

@wesm ?

`lib.reduce` can handle non-numpy dtypes AFAICT because I attempted to simplify the code by removing my original check for an `ExtensionDType` and changing the dummy `Series` initialization by writing `dummy = Series(index=index, dtype=values.dtype)`, and the problematic code brought up in the issue behaved just fine.  It's that casting to `np.float64` for integer dtypes that I mentioned earlier that prevented me from keeping it in this PR.

Given that tests are passing and the fact that I've already dealt with several other alternative designs that couldn't quite pass through the testing phase, I think this PR should be good to merge unless there are other suggestions or input?

pls add a whatsnew note, ping when green.

also I think add a test where we actually do something in the apply. (this only makes sense with a single tz-aware dtype, but that's ok), e.g. 
`df.apply(lambda x: x + pd.Timedelta('1day'))`

just as a confirming test

Changes have been made, though it looks like it will be some time before my build gets under way.  On a separate note, how do I subscribe to the `pandas` developer list (is it the Google group or the Python email list)?  Also, is there a way for me to get updates when PR's and commits have been made to the `master` branch?

@jreback : Finally!  Travis is happy, and changes have all been made.  Ready to merge.

lgtm. I'll rebase when merge. For future reference, I put random blank lines in Bug Fixes. If you put the whatsnew note there it won't have conflicts with other people.

@jreback : Okay, good to know!  Will rebase.  Btw, is there anyway for me to get notifications when commits (or merges) are made to `master`?

yes issue and PR will be closed and GitHub notifies 

Sorry clarification: I am talking about other people's merged PR's and commits into `master`.  I didn't realize I had a merge conflict until I saw the GitHub notification with your comment.

if u look at the prs it will you if it's clean to merge

u can also watch the repo to get notifications 
I don't thinks very granular just on or off or mentions only 

@jreback : Rebased and Travis gives the green light.

@gfyoung to respond to your other question about the mailing list: there are indeed two, the pydata google groups (https://groups.google.com/forum/?fromgroups#!forum/pydata) is a more general mailing list also with user questions/for the wider ecosystem, and the pandas-dev mailing list is more focused for pandas development (https://mail.python.org/mailman/listinfo/pandas-dev). You can for both subscribe on the web interface.

thanks!