BUG: preserve dtypes in interpolate
State: closed

Closes https://github.com/pydata/pandas/issues/6290

@jreback is there a good way to update **part** of a BlockManager inplace? I have a new BlockManager from `interpolate` and the original BlockManager (or DataFrame) which contains some "all good" columns that didn't get interpolate on, and some bad columns that did get interpolated on.

For the non inplace version I'm calling `self._constructor` on the BlockManager returned from interpolating and concating that to the all good columns.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@TomAugspurger no....this is where it gets tricky

you basically split the block's values then create and return 2 blocks from the Block routine (the apply at the BlockManager level willcombine them properly).

downcast, putmask, eval, convert all do this; they do it item by item (which you don't want to do)

give a shot and I can help 

you want to do this check at a much lower level, in Block.interpolate; e.g. instead of passing every array to the interpolate routines, you just selectively pass, then reassemble 

@jreback two things
1. We raise a `ValueError` if you interpolate with `method='time'` on an object that doesn't have a time series index. With this PR if you don't have any NaNs in the Series/Frame you just get it back, no `ValueError.` If there are any NaNs it still raises. I think this is actually an improvement, but I could be persuaded otherwise.
2. I'm just doing this on the `self._interpolate` method in core/internals (the part I added). Would you like to do a similar check for the `self._interpolate_with_fill` side also?

@TomAugspurger you could do this on `_interpolate_with_fill` as well. keeping in mind things:
- this may break some tests (which expect the float to come back always)
- this may introduce a performance issue (isnull is not free; though reasonably cheap)...maybe just give an overall vbench to see if anything pops up

you could try to a-prior split blocks (or post-process the changed ones)....can do later PR 

this looks good...you punted the splitting blocks?

The perf isn't looking good (assuming I'm running it correctly). There's an 80x slowdown on one of my new vbenches. I'll look at it closer tomorrow and ping you.

On Feb 13, 2014, at 9:35 PM, "jreback" <notifications@github.com<mailto:notifications@github.com>> wrote:

this looks good...you punted the splitting blocks?

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/6291#issuecomment-35052784.

Am I doing this correctly? I'm running `python test_perf.py -b cab2a93 -H`,

`cab2a93` is just a merge commit from the hexbin plots earlier today.

The script runs and here are the interpolate related ones:

```
$cat ../vb_suite.log | grep interpolate
frame_interpolate_some_good                  |   1.8273 |
frame_interpolate_some_good_infer            |   3.2444 |
frame_interpolate                            |  79.5443 |
```

But when I checkout those commits manually, I get:

```
cab2a93

In [11]: %timeit df.interpolate()
10 loops, best of 3: 110 ms per loop

master  # should be the same as cab2a93, and it is.

In [5]: %timeit df.interpolate()
10 loops, best of 3: 110 ms per loop

my branch

In [5]: %timeit df.interpolate()
10 loops, best of 3: 78.4 ms per loop
```

So it's faster when I actually check it out, not 80x slower.

`df` is the same as the one from the vbench:

```
df = DataFrame(randn(10000, 100))
df.values[::2] = np.nan
```

no...those are the actual running times in ms, not the ratio...

`./test_perf.sh -b <1 prior commit to yours> -t <last commit of yours> -r interpolate`

I make sure its checked in and pushed because it pulls from the remote repo

Thanks. This makes more sense.

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_interpolate_some_good                  |   1.8317 |   3.9976 |   0.4582 |
frame_interpolate                            |  79.5573 | 109.9803 |   0.7234 |
frame_interpolate_some_good_infer            |   3.3453 |   3.9697 |   0.8427 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

Here's the perf for the fillna parts:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_fillna_inplace                         |   9.4923 |   9.4840 |   1.0009 |
replace_fillna                               |   1.5057 |   1.4993 |   1.0042 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
```

I fixed that typo on vbench/merge_join also.

@TomAugspurger looks like github finally works again...pls rebase and can get this in (and if you want to change to `is_bool.....etc` cool

github emailed me back - they had a bug that was preventing things from working today 

fixed now

Closing without merging in favor of https://github.com/pydata/pandas/pull/6378.