PERF: Using Numpy C-API arange
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

This PR was opened as @jbrockmendel suggested (ref https://github.com/pandas-dev/pandas/pull/32177#discussion_r382923663)

---

### Benchmarks:

#### Master:

```
In [1]: import pandas._libs.internals as internals

In [2]: %timeit internals.BlockPlacement(slice(1_000_000)).as_array
1.55 ms ± 143 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

---

#### PR:

```
In [1]: import pandas._libs.internals as internals

In [2]: %timeit internals.BlockPlacement(slice(1_000_000)).as_array
1.46 ms ± 3.55 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```
 

Labels:
name: Performance
description: Memory or execution speed performance
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Since the perf numbers are pretty similar, can you report a few rounds of each
> Since the perf numbers are pretty similar, can you report a few rounds of each

Sure!

---

```
In [1]: import pandas._libs.internals as internals                                                            

In [2]: %timeit internals.BlockPlacement(slice(1)).as_array                                                   
1.46 µs ± 45.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # master
586 ns ± 35.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # PR

In [3]: %timeit internals.BlockPlacement(slice(10)).as_array                                                  
1.65 µs ± 21.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # master
832 ns ± 19.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # PR

In [4]: %timeit internals.BlockPlacement(slice(100)).as_array                                                 
1.74 µs ± 14.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # master
879 ns ± 12 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # PR

In [5]: %timeit internals.BlockPlacement(slice(1_000)).as_array                                               
2.94 µs ± 105 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) # master
2.1 µs ± 77.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # PR

In [6]: %timeit internals.BlockPlacement(slice(10_000)).as_array                                              
10.1 µs ± 198 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) # master
9.39 µs ± 170 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) # PR

In [7]: %timeit internals.BlockPlacement(slice(100_000)).as_array                                             
81 µs ± 1.75 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) # master
83 µs ± 1.36 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) # PR

In [8]: %timeit internals.BlockPlacement(slice(1_000_000)).as_array                                           
1.72 ms ± 165 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # master
1.56 ms ± 13.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # PR
```
thats a nice improvement for the small cases, thanks

lgtm, thanks for accomodating my nitpicking
@jbrockmendel I have a list of your nitpicks and requests, I try to address them when I have the time :)
Why does this generate anything different from Cython? Shouldn't it generate the same result?


> Why does this generate anything different from Cython? Shouldn't it generate the same result?

The generated C code has to call np.arange in python-space (and lookup np.int64, and ...)
So doesn’t work like the built in range call then huh? Was expecting it would automatically convert to C without the Python lookup

Sent from my iPhone

> On Mar 13, 2020, at 3:34 PM, jbrockmendel <notifications@github.com> wrote:
> 
> ﻿
> Why does this generate anything different from Cython? Shouldn't it generate the same result?
> 
> The generated C code has to call np.arange in python-space (and lookup np.int64, and ...)
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or unsubscribe.

thanks I agree we should not just wholesale convert calls to use the c-api, but in specific cases where we are actually returning a numpy array (as opposed to iteration), ok