Repository: cupy/cupy

Issue:
Read past end of array in percentile kernel can result in incorrect 100th quantile value
* Description:
The [percentile_weightnening kernel](https://github.com/cupy/cupy/blob/master/cupy/_statistics/order.py#L242-L252) in `cupy.percentile` will read one element past the end of the input array if it is asked to calculate the 100th quantile. If a NaN happens to be stored at that device memory location, the reported 100th quantile value will be NaN rather than the largest value in the array. Because that element is multiplied by zero (`weight_above`), this issue _only_ manifests when that value past the end of the array is a NaN.

* Conditions
  - CuPy version: 8.1.0 (installed via conda)
  - OS/Platform: Ubuntu 20.04
  - CUDA version: 10.1
Reviewing the source code, it seems clear that this should not be system-dependent, and I have encountered this bug across a range of OSes and CUDA versions in CI

* Code to reproduce (SEE COMMENT BELOW FOR BETTER REPRODUCER)
This is somewhat difficult to consistently reproduce without modifying cuPy code. One has to simply be unlucky about the value that happens to be at the memory address just past the end of the input array. If we add the following lines to order.py at [line 241](https://github.com/cupy/cupy/blob/master/cupy/_statistics/order.py#L241) just before the invocation of percentile_weightnening, however, we can force the issue:

```python
# Allocate new array with spot for NaN at the end
ap_new = cupy.zeros((ap.shape[0] + 1,), dtype=ap.dtype)
# Put NaN at end of array
ap_new[-1] = cupy.nan
# Set rest of array to the original values
ap_new[:-1] = ap
# Set the input array to percentile_weightnening to the original values (without NaN)
ap = ap_new[:-1]
```
Note that we are still passing the same values into the kernel; we have merely ensured that the memory just past the end of the input array contains a NaN.

With this modification in place, the following will consistently reproduce the issue:
```python
import cupy
quantiles = cupy.array([0, 50, 100], dtype='float32')
arr = cupy.array([0, 1, 2, 3, 4], dtype='float32')
print(cupy.percentile(arr, quantiles))
# Expected output: [0, 2, 4]
# Actual output: [0, 2, nan]
```

Without modifying the cupy code, we have to be unlucky to encounter this bug. I first discovered this issue due to an intermittent failure in a cuML unit test as described [here](https://github.com/rapidsai/cuml/issues/2933#issuecomment-733285578).


* Error messages, stack traces, or logs
Running cuda-memcheck shows that an invalid read is taking place in `percentile_weightnening`.
```
========= Invalid __global__ read of size 4                                                                                                                                                                     
=========     at 0x00000190 in percentile_weightnening                                                                                                                                                          
=========     by thread (5,0,0) in block (0,0,0)                                                                                                                                                                
=========     Address 0x7faa0480abd0 is out of bounds                                                                                                                                                           
=========     Device Frame:percentile_weightnening (percentile_weightnening : 0x190)  
```
We can also confirm that the array index in [this line](https://github.com/cupy/cupy/blob/master/cupy/_statistics/order.py#L250) can reach a value equal to the size of the array by adding a `printf` inside the kernel.


Discussion:
I've just managed to find a consistent reproducer that does not require any modification of cupy code. By flooding the available GPU memory with NaNs first, we can ensure that a NaN will be found immediately after the input array. I don't know if this will reproduce the issue 100% of the time, and you may need to adjust the size of the NaN array based on your available GPU memory, but the following has reproduced the issue every time I've run it on a Quadro RTX 8000:

```python
import cupy
import gc

nans = cupy.full((int(1e10),), cupy.nan, dtype='float32')
del nans
gc.collect()

quantiles = cupy.array([0, 50, 100], dtype='float32')
arr = cupy.array([0, 1, 2, 3, 4], dtype='float32')
print(cupy.percentile(arr, quantiles))
# Expected output: [0, 2, 4]
# Actual output: [0, 2, nan]
```
I'm working on a fix for this and will post a PR tomorrow.

Resolving patch:
diff --git a/cupy/_statistics/order.py b/cupy/_statistics/order.py
--- a/cupy/_statistics/order.py
+++ b/cupy/_statistics/order.py
@@ -240,17 +240,18 @@ def _quantile_unchecked(a, q, axis=None, out=None, interpolation='linear',
             ret = cupy.rollaxis(out, 0, out.ndim)
 
         cupy.ElementwiseKernel(
-            'S idx, raw T a, raw int32 offset', 'U ret',
+            'S idx, raw T a, raw int32 offset, raw int32 size', 'U ret',
             '''
             ptrdiff_t idx_below = floor(idx);
             U weight_above = idx - idx_below;
 
             ptrdiff_t offset_i = _ind.get()[0] * offset;
+            ptrdiff_t max_idx = size - 1;
             ret = a[offset_i + idx_below] * (1.0 - weight_above)
-              + a[offset_i + idx_below + 1] * weight_above;
+              + a[min(offset_i + idx_below + 1, max_idx)] * weight_above;
             ''',
             'percentile_weightnening'
-        )(indices, ap, ap.shape[-1] if ap.ndim > 1 else 0, ret)
+        )(indices, ap, ap.shape[-1] if ap.ndim > 1 else 0, ap.size, ret)
         ret = cupy.rollaxis(ret, -1)  # Roll q dimension back to first axis
 
     if zerod:


Test patch:
diff --git a/tests/cupy_tests/statistics_tests/test_order.py b/tests/cupy_tests/statistics_tests/test_order.py
--- a/tests/cupy_tests/statistics_tests/test_order.py
+++ b/tests/cupy_tests/statistics_tests/test_order.py
@@ -6,6 +6,7 @@
 
 import cupy
 import cupy.core._accelerator as _acc
+from cupy import cuda
 from cupy import testing
 
 
@@ -117,6 +118,36 @@ def test_percentile_unexpected_interpolation(self, dtype):
             with pytest.raises(ValueError):
                 xp.percentile(a, q, axis=-1, interpolation='deadbeef')
 
+    # See gh-4453
+    @testing.for_float_dtypes()
+    def test_percentile_memory_access(self, dtype):
+        # Create an allocator that guarantees array allocated in
+        # cupy.percentile call will be followed by a NaN
+        original_allocator = cuda.get_allocator()
+
+        def controlled_allocator(size):
+            memptr = original_allocator(size)
+            base_size = memptr.mem.size
+            assert base_size % 512 == 0
+            item_size = dtype().itemsize
+            shape = (base_size // item_size,)
+            x = cupy.ndarray(memptr=memptr, shape=shape, dtype=dtype)
+            x.fill(cupy.nan)
+            return memptr
+
+        # Check that percentile still returns non-NaN results
+        a = testing.shaped_random((5,), cupy, dtype)
+        q = cupy.array((0, 100), dtype=dtype)
+
+        cuda.set_allocator(controlled_allocator)
+        try:
+            percentiles = cupy.percentile(a, q, axis=None,
+                                          interpolation='linear')
+        finally:
+            cuda.set_allocator(original_allocator)
+
+        assert not cupy.any(cupy.isnan(percentiles))
+
     @testing.for_all_dtypes()
     @for_all_interpolations()
     @testing.for_all_dtypes(no_float16=True, no_bool=True, no_complex=True)