Issue #10174. Add 'interpolation' keyword in DataFrame.quantile and Series.quantile
State: closed

Closes #10174.
I have added the new argument to the doc too.


Labels:
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
This need some tests.

I'll add some tests too.
Meanwhile I am trying to fix the build fail due to a stray \xe2 character.

The interpolation argument was added to np.percentile() only in numpy version 1.9.0. The tests don't pass where numpy version < 1.9.0. How to handle this?

For the tests to handle the numpy version, you can do something like this:

```
if _np_version_under1p9:
    raise nose.SkipTest('numpy >= 1.9 required')
```

What should happen when numpy version is < 1.9.0? 
1. Should quantile() still work and ignore the `interpolation` argument? OR
2. should it fail saying that pd > 1.9 is required?

if 1. , then I can put the `if _np_version_under1p9:` check inside `quantile()` and handle the different numpy versions transparently. We can throw a warning if the interpolation keyword is specified, but not being used (np < 1.9).

The default interpolation argument should work on all supported versions of NumPy.

For numpy < 1.9.0, it's OK not to support other interpolation options. In these cases, we should raise a `ValueError` with an informative error message. We should certainly not ignore the interpolation argument or merely raise a warning -- that could lead to unanticipated bugs.

Now the default interpolation argument works on all supported versions of NumPy.
And raised `ValueError` if an interpolation method other than `linear` is specified when numpy versions <1.9.

Will add tests soon.

I added tests for Dataframe.quantile . This is my first time writing tests, so please verify if it is done right.
If someone verifies these tests,  I'll write the tests for Series.quantile too.

Tests look good! Most important things to be tested is that the keyword is passed to percentile correctly, that it generate an error for older numpy versions, and that the default is not changed. And those are included, so OK!

I added the explicit test that the result with interpolation='linear' and without specifying it is the same.
Also added the github issue number.

this is getting to be lots of duplicated code. I would prefer that this all be moved to `core/generic.py`, see #10207 

can you rebase?

@jreback rebase done!

not really sure what you did. you should have 1 commit.

pls see contributing docs: http://pandas.pydata.org/pandas-docs/stable/contributing.html

I screwed up a lot. I fetched upstream and then rebased on that or something.  
It's better I close this and create a new pull request following the contributing guide.
 @jreback Is there any better way?

@mayankasthana No, there is no need to close this and create a new PR.

Normally, doing this:

```
git fetch upstream
git rebase upstream/master
git push -f origin interpolationQuantile
```

should be all that is needed to clean this up

@jreback Thanks for the help with git.  Here is the expected 1 commit. 

looks pretty good. pls add a release note.

@jreback @jorisvandenbossche Where should I add the feature description in `v0.17.0.txt`?
Under `Other API Changes`  or `Other enhancements`? And does it require a usage example?

You can put it under 'Other enhancements'. A one line sentence explaining it is enough I think.

@jorisvandenbossche @jreback Added release note, rebased to get a single commit. Please review.

@mayankasthana can you update

@jreback How can the signature be same for Dataframe.quantile and Series.quantile?
For `Dataframe` it  is `def quantile(self, q=0.5, axis=0, numeric_only=True, interpolation='linear')`
and for `Series` it is `def quantile(self, q=0.5, interpolation='linear'):`
So, should the `Series.quantile` be `def quantile(self, q=0.5, axis=0, numeric_only=True, interpolation='linear')` and in the code to `assert`  `axis=0` and `numeric_only=True`?
How would the usage for `Series.quantile` be? For using interpolation, only named parameter will have to be used.
As for the update, I'll finish this up by tonight.

It is true that the signatures are quite different (Series.quantile does not accept axis at the moment), so I think it is OK to leave that for another PR (combining them into generic.py).

@mayankasthana I don't think its a big deal to make these have the same signature. just add to `Series` `axis/numeric_only`.

@jreback I didn't mean the method signatures. I meant that the doc strings are too different to be made generic with replacements.
For example, the return value for `DataFrame.quantile` is `Series or DataFrame`, and for `Series.quantile` it is `float or Series`. 
The examples I saw of replacements are like this `@Appender(_shared_docs['boxplot'] % _shared_doc_kwargs)` with the `_shared_doc_kwargs` having the `DataFrame` or `Series` specific args, which are very simple replacements (Eg. `dict(axes='index, columns', klass='DataFrame',
                          axes_single_arg="{0, 1, 'index', 'columns'}")`. 
But these simple replacements from a generic `_shared_doc` won't work here because the return values, the parameters, examples are very different.
Am I understanding what you want correctly?

note that you will need to rebase as https://github.com/pydata/pandas/pull/10881 was merged

pls rebase and address comments

@jreback Rebased. Test is green.

@jreback Rebased. Test is green.