ENH: Add numba engine to df.apply
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

~~Performance is a mixed bag, it is sometimes faster and sometimes slower than Python.
This is because there is a huge cost (esp. in boxing the numba representations back into Series/Index since the constructors are slow). 
Right now, in the no-op case(return self), we are around 5-10x slower - I'm hoping to bring that down to around 2x.
(There is an issue with the Index being repeatedly unboxed unnecessarily)~~

EDIT: I think I've closed the gap, the difference isn't much now. 

We are also ~10x faster in a normalization (subtract mean, divide by std. dev) test I did.

Labels:
name: Apply
description: Apply, Aggregate, Transform, Map
name: numba
description: numba-accelerated operations

Comments:
@mroeschke 

This should be ready for a first pass now.
I know this is a lot of code, as of right now, this PR implements a pretty minimal subset of the pandas features, which are:

1. Numba versions of DataFrame, Series, Index
2. Common Series methods that have a numpy equivalent
   a. Think sum, var, mean, and co.
3. Basic indexing support
   a. It mimics our khash stuff, just with a numba dictionary instead. We might look into re-using the khash code, it would require some Cython code to expose that to numba though.

As of right now, there isn't support for things such as the Arrow dtypes, and non-numeric Indexes, but those shouldn't be hard to add in the future - I just didn't think the maintenance burden would be worth it as of right now.

(If you want, I can try to split up the PR, but because 1 is still a lot of lines, its still gonna be pretty big to review unfortunately).

Performance is decent as of now.
We are generally anywhere between 2-10x faster when going row-wise, but can be slower in cases where there are few rows/columns.

Most of the time is spent boxing/unboxing (converting to/from) the numba representation, and the pandas representation of the DataFrame. In the future, we could optimize this to potentially get like 100x speedups.

This should be fairly fixable with some effort, when just need to re-implement the concat/results wrapping that apply does in numba. I've held off on this because I've written a lot of code so far, and like before, I'd like to see people use this feature before I add to the maintenance burden of the numba code.

Sadly, multithreading doesn't yet work - numba has no thread safe structures, but this won't be relevant until the unboxing/boxing overhead can be dealt with.
gentle ping @mroeschke 
@mroeschke 
Can you take another look at this?

I've simplified the hacky parts a bit so we no longer clobber the ``_data`` attribute of an Index, and added some more tests.

There's still some work to be done on catching and testing all the unsupported cases(e.g. EAs, Arrow arrays, object dtype, etc.), but I'll put that in a followup PR to keep this one small.
Final question. What error do we currently get when trying to use apply with numba with an unsupported type (like `datetime64[ns]`?
> Final question. What error do we currently get when trying to use apply with numba with an unsupported type (like `datetime64[ns]`?

I added an error message and a test.

It's basically a ValueError saying the dtype in this column is unsupported.
Great work on this @lithomas1. In a follow up, could you add a whatsnew note in 2.2?
Thanks for the reviews - I think there should be a whatnsew from when I added support with raw apply.

This definetely deserves a bigger blurb, though. I'll add that sometime next week.