Complex Dtype Support for Hashmap Algos
State: closed

- [x] fixes #17927 #26475
- [x] tests added / passed
- [x] passes black pandas
- [x] passes git diff upstream/master -u -- "*.py" | flake8 --diff

Ref: https://github.com/pandas-dev/pandas/issues/18009

Based on https://github.com/pandas-dev/pandas/pull/27599

First set of tests for complex number handling + sensible results from functions that rely on hash tables.

Use generic object hashing for now.

@jbrockmendel you interested in reviewing?

Labels:
name: Complex
description: Complex Numbers

Comments:
do we have a list of what algorithms this is used for?  if they are all unique/factorize-like, we might be able to do a `.view(floatlike)` to avoid an object cast
This seems OK to me to start - could probably optimize later with things like `.view(floatlike)` though there may be some drawbacks to doing that as well
I'm finding in a mostly-unrelated branch that having actual support for complex (in particular complex128) would be tremendously helpful.  Just gentle encouragement.
Sure will take a look @jreback 
> I'm finding in a mostly-unrelated branch that having actual support for complex (in particular complex128) would be tremendously helpful. Just gentle encouragement.

Sure which branch was this? This doesn’t offer complex128 support - but makes a lot of basic functional work. But I can perhaps look into that in a follow up @jbrockmendel 
fixing this up today.
cc @realead
@alimcmaster1 IIUC #38179 would make object-casting unnecessary.  is that correct?
@alimcmaster1 can you merge master. complex support was merged for hashtables
> @alimcmaster1 IIUC #38179 would make object-casting unnecessary. is that correct?

Yes correct - super helpful change. I've updated to incorporate that. Mind taking another look? 
> @alimcmaster1 can you merge master. complex support was merged for hashtables

updated - mind taking a look @jreback 
There are no test cases with not-a-number values, like `np.nan`, `1j*np.nan`, `np.nan+1j`, `1j+np.nan` and `np.nan+1j*np.nan` (best all at once). These are often tricky and are easily overlooked - thus always a good idea to have some unit tests with them.
@alimcmaster1 if you can merge master
@alimcmaster1 I think this can be simplified in large part bc one one of the arguments to ensure_data has now been removed
> @alimcmaster1 I think this can be simplified in large part bc one one of the arguments to ensure_data has now been removed

Yes good point - will pick this up in the next few days
Going to close due to this PR going stale. Let us know if you'd like to continue after merging master and addressing the comments.
> There are no test cases with not-a-number values, like `np.nan`, `1j*np.nan`, `np.nan+1j`, `1j+np.nan` and `np.nan+1j*np.nan` (best all at once). These are often tricky and are easily overlooked - thus always a good idea to have some unit tests with them.

Good idea added some testing around this in `test_duplicates.py`

Added testing for complex64 and complex128 in `test_duplicates.py` , `test_reductions.py` and `test_value_counts.py`

I think as a follow up we can address the inferred_type issue in complex indexes:
```
pd.Series([3, 2, 1], index=pd.Index([3j, 1 + 1j, 1])).index.inferred_type
Out[23]: 'mixed-integer'
(Would expect this to be "complex")

In [24]: pd.Series([3, 2, 1], index=pd.Index([3j, 1 + 1j, 1], dtype=np.complex128)).index.inferred_type
Out[24]: 'complex'
```

cc @realead if you can look
restarting CI test failures unrelated.

```
Could not find conda environment: pandas-dev
You can list all discoverable environments with `conda info --envs`.

Error: Process completed with exit code 1.
```

All comments addressed this should be good.
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
> LGTM

Appreciated the review :)