Repository: numpy/numpy

Issue:
BUG: numpy.percentile output is not sorted
The output of `numpy.percentile` is not always sorted

### Reproducing code example:


```python
import numpy as np
q = np.arange(0, 1, 0.01) * 100
percentile = np.percentile(np.array([0, 1, 1, 2, 2, 3, 3 , 4, 5, 5, 1, 1, 9, 9 ,9, 8, 8, 7]) * 0.1, q)
equals_sorted = np.sort(percentile) == percentile
print(equals_sorted)
assert equals_sorted.all()
```

### Error message:
[ True  True  True  True  True  True  True  True  True  True  True  True
  True  True  True  True  True  True  True  True  True  True  True  True
  True  True  True  True  True  True  True  True  True  True  True  True
  True  True  True  True  True  True  True  True  True  True  True  True
  True  True  True  True  True  True  True  True  True  True  True  True
  True  True  True  True  True  True  True  True  True  True  True  True
  True  True  True  True  True  True  True  True  True  True  True  True
  True  True  True  True  True False False  True  True  True  True False
  True  True  True False]
AssertionError                            Traceback (most recent call last)
<ipython-input-63-2850fe4a4ce3> in <module>
      1 q = np.percentile(np.array([0, 1, 1, 2, 2, 3, 3 , 4, 5, 5, 1, 1, 9, 9 ,9, 8, 8, 7]) * 0.1, np.arange(0, 1, 0.01) * 100)
      2 equals_sorted = np.sort(q) == q
----> 3 assert equals_sorted.all()

AssertionError: 

### Numpy/Python version information:

1.17.2 3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]




Discussion:
Why would you expect it to be sorted? Percentile is elementwise - the outputs are in the order of the inputs.
Hi !
Indeed, percentile is elmenet-wise - when considering `q`, which in our case is
`np.arange(0, 1, 0.01) * 100`.
I expect the output to be sorted because `q` is sorted.

There are some numerical errors within a single ULP, that differ for different inputs with the same output value. I doubt there is anything to be done about that.
A slightly reduced failing case:
```
In [40]: np.percentile(np.array([0, 1, 1, 2, 2, 3, 3 , 4, 5, 5, 1, 1, 9, 9 ,9, 8, 8, 7]) * 0.1, [89, 90, 95, 96, 98, 99])
Out[40]: array([0.9, 0.9, 0.9, 0.9, 0.9, 0.9])

In [41]: np.diff(_)
Out[41]:
array([-1.11022302e-16,  2.22044605e-16, -1.11022302e-16,  1.11022302e-16,
       -1.11022302e-16])
```
here showing non-sorted-ness via the diff.

I think there probably is something we can do about this. I think this comes down to the stability of these lines, which perform a `lerp` operation (essentially `add(v_below*weights_below, v_above*weights_above)`):

https://github.com/numpy/numpy/blob/b9fa88eec62e34e906689408096beb2450830d9a/numpy/lib/function_base.py#L3907-L3908

https://github.com/numpy/numpy/blob/b9fa88eec62e34e906689408096beb2450830d9a/numpy/lib/function_base.py#L3928-L3929

https://github.com/numpy/numpy/blob/b9fa88eec62e34e906689408096beb2450830d9a/numpy/lib/function_base.py#L3939-L3942

There are a bunch of tradeoffs to be made when linearly interpolating floating point values, but I suspect that there's a "correct" choice here, and we just haven't made it.

Some more background here: https://math.stackexchange.com/questions/907327/accurate-floating-point-linear-interpolation

Yeah, I agree, +1 on reorganizing the operations so that it is strictly monotonic (numerically). Would be good if it is also no worse, or at least almost identical precision wise. I am sure we really do not have to worry about a few extra operations/speed here.

**EDIT: Marked as good first issue. This is _only_ a good first issue if you are willing to dive into the intricacies of IEEE floating point numbers. But after that, this is probably a fairly straight forward reorganization within python code.**
I would be interested in taking on this issue. I was looking at some of the failing cases and noticed that they all involved linearly interpolating between the same number. i.e. in Eric's example all of the percentiles he listed listed are located in between two 9s. Therefore I think the linear interpolation between them must be 9 exactly correct? fixing the problem of linearly interpolating between two number that are the same seems like it would deal with the issues presented in this bug and not cause a noticeable hit in performance. If however we want to ensure that the linear interpolation will be monotonic always, we can do that but It will require a piecewise function that I would think would decrease performance.
@ngonzo95 there should be a way to spell the arithmetic of the interpolation differently to achieve this, i.e. change/rearrange the formula that is used for the calculation (so that it is mathematically identical, but numerically guarantees monotonicity). No piecewise calculation should be necessary. 
> No piecewise calculation should be necessary.

It depends what your requirements on `lerp` are. Some that we may or may not care about:
* monotonic (`(lerp(a, b, t1) - lerp(a, b, t0)) * (b - a) * (t1 - t0) >= 0`)
* bounded (`a <= lerp(a, b, t) <= b`)
* symmetric (`lerp(a, b, t) == lerp(b, a, 1-t)`)

(`0 <= t <= 1`)
Oh OK, I did not expect piecewise to be necessary, but do not know the intrinsicaties of this well enough I guess.
looking into it more I discovered that the function a + (b-a)*t has the property of being both monotonic (definition noted above) and consistent (lerp (a, a, t) = a). I believe this should be sufficient for the functions requirements. It seems one of the main draw backs of this function is that lerp(a, b, 1) !=b. However I think the way we are calculating weights ensures that 0<=t<1.
> It seems one of the main draw backs of this function is that lerp(a, b, 1) !=b. However I think the way we are calculating weights ensures that 0<=t<1.

Note that unfortunately `lerp(a, b. 1-eps) > b)` is possible with that formulation.
New to the open source.
Wanted to solve this as my good first issue. How can i contribute? Are there any prerequisites?
> I was looking at some of the failing cases and noticed that they all involved linearly interpolating between the same number

In scikit-learn, we recently stumbled in this issue: https://github.com/scikit-learn/scikit-learn/issues/15733

Since we expect `q` to be strictly increasing, we can apply `np.maximum.accumulate` reorder the array. However, if we could solve the issue in NumPy directly, this would be great. Is there anywhere that we can dig in to have a good fix?
@glemaitre: All of the relevant lines in numpy are linked in my comment above, https://github.com/numpy/numpy/issues/14685#issuecomment-541467915
Hey, there seems to have been an update to one of the stackexchange answers provided by @eric-wieser with a good alternative interpolation.
The thread includes a proof of monotonicity, and the proposed fix appears to address all of the issues mentioned.
If this would make sense for the issue, I would be willing to implement this as a first commit, or someone else could try it.
![20191209_020250](https://user-images.githubusercontent.com/58437698/70414473-2cbfc200-1a28-11ea-8f1e-a244db56b6e7.jpg)

Note that there is another issue with lerp in `quantile()`: inf values are not handled correctly, see #12282.

Resolving patch:
diff --git a/numpy/lib/function_base.py b/numpy/lib/function_base.py
--- a/numpy/lib/function_base.py
+++ b/numpy/lib/function_base.py
@@ -3875,7 +3875,13 @@ def _quantile_is_valid(q):
 
 def _lerp(a, b, t, out=None):
     """ Linearly interpolate from a to b by a factor of t """
-    return add(a*(1 - t), b*t, out=out)
+    diff_b_a = subtract(b, a)
+    # asanyarray is a stop-gap until gh-13105
+    lerp_interpolation = asanyarray(add(a, diff_b_a*t, out=out))
+    subtract(b, diff_b_a * (1 - t), out=lerp_interpolation, where=t>=0.5)
+    if lerp_interpolation.ndim == 0 and out is None:
+        lerp_interpolation = lerp_interpolation[()]  # unpack 0d arrays
+    return lerp_interpolation
 
 
 def _quantile_ureduce_func(a, q, axis=None, out=None, overwrite_input=False,


Test patch:
diff --git a/numpy/lib/tests/test_function_base.py b/numpy/lib/tests/test_function_base.py
--- a/numpy/lib/tests/test_function_base.py
+++ b/numpy/lib/tests/test_function_base.py
@@ -5,6 +5,10 @@
 from fractions import Fraction
 import math
 import pytest
+import hypothesis
+from hypothesis.extra.numpy import arrays
+import hypothesis.strategies as st
+
 
 import numpy as np
 from numpy import ma
@@ -3104,6 +3108,73 @@ def test_no_p_overwrite(self):
         np.quantile(np.arange(100.), p, interpolation="midpoint")
         assert_array_equal(p, p0)
 
+    def test_quantile_monotonic(self):
+        # GH 14685
+        # test that the return value of quantile is monotonic if p0 is ordered
+        p0 = np.arange(0, 1, 0.01)
+        quantile = np.quantile(np.array([0, 1, 1, 2, 2, 3, 3, 4, 5, 5, 1, 1, 9, 9, 9,
+                                         8, 8, 7]) * 0.1, p0)
+        assert_equal(np.sort(quantile), quantile)
+
+    @hypothesis.given(arr=arrays(dtype=np.float, shape=st.integers(min_value=3,
+                                                                   max_value=1000),
+                                 elements=st.floats(allow_infinity=False,
+                                                    allow_nan=False,
+                                                    min_value=-1e300,
+                                                    max_value=1e300)))
+    def test_quantile_monotonic_hypo(self, arr):
+        p0 = np.arange(0, 1, 0.01)
+        quantile = np.quantile(arr, p0)
+        assert_equal(np.sort(quantile), quantile)
+
+
+class TestLerp:
+    @hypothesis.given(t0=st.floats(allow_nan=False, allow_infinity=False,
+                                   min_value=0, max_value=1),
+                      t1=st.floats(allow_nan=False, allow_infinity=False,
+                                   min_value=0, max_value=1),
+                      a = st.floats(allow_nan=False, allow_infinity=False,
+                                    min_value=-1e300, max_value=1e300),
+                      b = st.floats(allow_nan=False, allow_infinity=False,
+                                    min_value=-1e300, max_value=1e300))
+    def test_lerp_monotonic(self, t0, t1, a, b):
+        l0 = np.lib.function_base._lerp(a, b, t0)
+        l1 = np.lib.function_base._lerp(a, b, t1)
+        if t0 == t1 or a == b:
+            assert l0 == l1  # uninteresting
+        elif (t0 < t1) == (a < b):
+            assert l0 <= l1
+        else:
+            assert l0 >= l1
+
+    @hypothesis.given(t=st.floats(allow_nan=False, allow_infinity=False,
+                                  min_value=0, max_value=1),
+                      a=st.floats(allow_nan=False, allow_infinity=False,
+                                  min_value=-1e300, max_value=1e300),
+                      b=st.floats(allow_nan=False, allow_infinity=False,
+                                  min_value=-1e300, max_value=1e300))
+    def test_lerp_bounded(self, t, a, b):
+        if a <= b:
+            assert a <= np.lib.function_base._lerp(a, b, t) <= b
+        else:
+            assert b <= np.lib.function_base._lerp(a, b, t) <= a
+
+    @hypothesis.given(t=st.floats(allow_nan=False, allow_infinity=False,
+                                  min_value=0, max_value=1),
+                      a=st.floats(allow_nan=False, allow_infinity=False,
+                                  min_value=-1e300, max_value=1e300),
+                      b=st.floats(allow_nan=False, allow_infinity=False,
+                                  min_value=-1e300, max_value=1e300))
+    def test_lerp_symmetric(self, t, a, b):
+        # double subtraction is needed to remove the extra precision that t < 0.5 has
+        assert np.lib.function_base._lerp(a, b, 1 - (1 - t)) == np.lib.function_base._lerp(b, a, 1 - t)
+
+    def test_lerp_0d_inputs(self):
+        a = np.array(2)
+        b = np.array(5)
+        t = np.array(0.2)
+        assert np.lib.function_base._lerp(a, b, t) == 2.6
+
 
 class TestMedian: