Quick Frame Shift Implementation
State: closed

Fix for issue #5609


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
ok....when you run the tests...lots of failure....pls investigate....

I see that the indexer in pandas/core/internal is unused. Removing this argument will mean we can take out `_shift_indexer`. Should I remove the indexer argument in pandas/core/internal ?

yep indexer can go 

Taking indexer out clashes with the shift in the SparseBlock. Should the same roll logic work here ? Any idea what are the other shift calls that may need to be changed here?

So I would take the indexer out of the Block.shift method as you don't need it their. Take the argument out of the SparseBlock.shfit as well, and compute the indexer their (move the _shift_indexer code into that function).

The indexer is obtained as:
`indexer = com._shift_indexer(len(self._get_axis(axis)), periods)`

The first argument to `_shift_indexer` requires the length of the axis using `_get_axis` method defined in NDFrame. Is there an equivalent call inside `Block`?

Is it the same as saying `len(self.values)` or `len(self.values.T)`?

just take the guts of `_shift_indexer` (and remove it from `common.py`). The length it is looking for is `len(self.values)` i believe.

can you post the results of the vbench?

If you can rebase/squash would be great

Are there any docs on how to setup and run vbench? I couldn't get it running! I tested the old and new using timeit.

go to `vb_suite`; run `./test_perf.sh -b master -t HEAD`

I put some notes I had written up for statsmodels at https://github.com/pydata/pandas/wiki/Performance-Testing

@jreback hopefully everything is accurate. I'll add more later.

@TomAugspurger awesome just what was needed!

@jreback I couldn't run the test like you said. Perhaps it has to do with the fact that I use a Windows system. Can some one run this? I do  have the squashed commit here. 

ok just post the results them of the benchmark before and after the change

## Test code

```
import pandas as pd
import numpy as np
import timeit

xdim = 10000
ydim = 500
repeats = 10
df = pd.DataFrame(np.random.rand(xdim,ydim))

# axis 0 shift
s = timeit.default_timer()
for i in range(repeats):
    df1 = df.shift(1,axis=0)
e = timeit.default_timer()
print "Axis 0 shift",e-s

# axis 1 shift
s = timeit.default_timer()
for i in range(repeats):
    df1 = df.shift(1,axis=1)
e = timeit.default_timer()
print "Axis 1 shift",e-s
```

## Results

```
V0.13.1

Axis 0 shift 0.958936203491
Axis 1 shift: IndexError: index 500 is out of bounds for size 500

----------------------------------------------------------------------------------
V0.13.1-330-gb562eb2

Axis 0 shift 0.1622
Axis 1 shift 0.3466 

```

@jreback I am not sure if the squashed set is supposed to have all these commits. Also I notice that after squashing, the shift has the axis in reverse. Not sure what happened here!!!!!

try

```
git  rebase upstream/master
git push yourforkname yourbranchname -f 
```

@jreback The rebase was a little confusing. I am getting a lot of conflicts. Not sure why! Sorry I am a little new to git. My squashed commits are in b562eb2 + ff18ac9 shown above. Any chance we can work with this. As of the last commit, I checked that all the tests were passing.

ok...here's what you need to do:

```
git checkout -b new_branch --track origin/master
git cherry-pick <commit1> thisbranch
git cherry-pick <commit2> thisbranch
git push myfork thisbranch
```

where <commit1> and 2 are your commits from above

this takes the commits out and puts them in a new branch

push this branch up and do a new PR, closing this one.

Please refer to #6672 for the new pull request.