PERF: set multiindex labels with a coerced dtype (GH8456)
State: closed

closes #8456 


Labels:
name: Bug
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
cc @shoyer
cc @JanSchulz
cc @jorisvandenbossche 

@shoyer both done

looks pretty good to me!

note: not really sure why pandas has `_ensure_int8` functions (and the like)... I would think `np.asarray(x, 'int8')` would be succinct enough!

@jreback 
please consider reverting this change; casting to smaller integer types comes with the risk of overflow/wraparound, plus a performance cost anywhere there is a call to `com._ensure_int64` or when numpy internally has to up-cast. as an example indexing into sorted multi-index has been impacted by this commit, because numpy has to up-cast before doing binary search:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_xs_mi_ix                               |   8.5303 |   0.6206 |  13.7452 |
series_xs_mi_ix                              |   8.0659 |   0.5600 |  14.4023 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [c11e75c] : PERF: set multiindex labels with coerced dtype (GH8456)
Base   [6bbb39e] : Merge pull request #8675 from pydata/setitem
```

so it will change an `O(log(n))` performance to `O(n)`. (down-casting the scalar value is _not_ an option here because of the risk of overflow, so an entire array has to be up-cast).

there are also many places in the code that explicitly call into `com._ensure_int64`, and this commit will impact their performance as well.

also _practically_, down casting has a _negative_ impact on the memory usage, because many operations generate a temporary version of the array with upcasted types (even when viable, down-casting the other operand is risky or costly to do it safely), so then there will be two copies of the same values in the memory instead of only one, and that definitely increases memory usage.

@behzadnouri these are only for indexers. The risk of overflow is 0. Yet you gain enormous memory savings. 

If you see a perf impact then it would be much easier to NOT upcast when doing the indexing.

I would say that their are some negative perf implications of this, but I suspect they can be fixed and/or outweiged by holding a smaller dtype of indexer.

pls create a new issue for this, with a specific example of the problem. keep in mind that the benchmarks you are showing are for a scalar indexing operation which is not very useful in and of itself. (that said if it can be fixed great). 

@jreback as in the last paragraph of my comment this will cause _more_ memory usage.
memory profile results added in the https://github.com/pydata/pandas/issues/9073

you will have to prove it on several use cases. E.g. need more than a single example. I agree that their is some casting issues, but I suspect those can be addressed.

@jreback below is the proof; any time the code hits the lines below with smaller integer type, it creates a copy of the same data and that costs both memory and performance:

```
$ find -maxdepth 4 -iname '*.py' -exec grep -IrHn -i 'ensure_int64.*index' '{}' \;
./pandas/core/groupby.py:3692:        sorter, _ = _algos.groupsort_indexer(com._ensure_int64(group_index),
./pandas/core/groupby.py:3708:    group_index = com._ensure_int64(group_index)
./pandas/core/generic.py:1828:                indexer = com._ensure_int64(indexer)
./pandas/core/common.py:729:        indexer = _ensure_int64(indexer)
./pandas/core/common.py:768:        indexer = _ensure_int64(indexer)
./pandas/core/common.py:822:    indexer = _ensure_int64(indexer)
./pandas/core/common.py:971:    return _ensure_int64(indexer)
./pandas/core/common.py:980:    return _ensure_int64(indexer)
./pandas/core/frame.py:4125:        labels = com._ensure_int64(frame.index.labels[level])
./pandas/core/series.py:1123:            labels = com._ensure_int64(self.index.labels[level])
./pandas/core/index.py:1896:            left_lev_indexer = com._ensure_int64(left_lev_indexer)
./pandas/tseries/index.py:464:                index = tslib.tz_localize_to_utc(com._ensure_int64(index), tz,
```

you are missing the point. These should in theory be removable. This bizness of conversions between int64 is just a distraction. These _can_ be indexed via a smaller indexer.