Repository: cupy/cupy

Issue:
cupy.random.uniform(low=low, high=high) returns high
```python
import cupy as cp

cp.random.seed(seed=13)
high = 2**32 - 1
low = 2**32 - 3
size = 100000000
(cp.random.uniform(low=low, high=high, size=size) == high).sum()
```
returns `array(11)`, while `array(0)` is expected, as "Samples are drawn from a uniform distribution over the half-open interval [low, high)".

Interestingly, the documentation of standard `random.uniform` says "The end-point value b may or may not be included in the range depending on floating-point rounding in the equation `a + (b-a) * random()`".  `RandomState._scale_kernel ` seems to do the same thing, so it might be a documentation problem.

BTW, the behavior looks like it is consistent with `numpy`.

* Conditions
```
CuPy Version          : 6.0.0
CUDA Root             : /usr/local/cuda
CUDA Build Version    : 10000
CUDA Driver Version   : 10010
CUDA Runtime Version  : 10000
cuDNN Build Version   : 7301
cuDNN Version         : 7605
NCCL Build Version    : 1000
NCCL Runtime Version  : (unknown)
Linux 4.4.180-102-default 
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
Tesla P100
```


Discussion:
This also happens in numpy

```
import numpy as np

np.random.seed(seed=13)
high = 2**32 - 1
low = 2**32 - 3
size = 100000000
(np.random.uniform(low=low, high=high, size=size) == high).sum()
```
returns 12

This is probably due to floating-point precision issues in the comparison with high when the integer is cast to floating-point.
```
>>> np.array(2**32 - 1)
array(4294967295)
>>> np.array(2**32 - 1).astype(np.float64)
array(4.2949673e+09)
```

BTW, I am just being curious, but is there any reason for you to be using CuPy v6.0? we have just released v8 and you can get it with pip by doing `pip install cupy-cuda100` 🙂
@emcastillo 

First of all, thank you for pointing out an available update. 

I don't think the issue is related to precision loss due to int to float64 conversion. 
```python
high = 2**32 -1
low = 2**32 - 3
np.array(high).astype('float64').astype('int') == high # True
np.array(low).astype('float64').astype('int') == low   # True
```
However, for larger numbers (like `~2**54`) this would indeed take effect. 

Moreover, the issue still persists if we cast `high` and `low` to float beforehand. 
```python
import cupy as cp

def tofloat64(a):
    return cp.array(a).astype('float64')

cp.random.seed(seed=13)
high = tofloat64(2**32 - 1)
low = tofloat64(2**32 - 3)
size = 100000000
(cp.random.uniform(low=low, high=high, size=size) == high).sum() #array(11)
```
Just to confirm, as Numpy's documentation says, this is the effect of floating-point rounding and not a CuPy issue, right?

```python
import numpy

a = numpy.array(2**32 - 2, dtype=numpy.float64)
b = numpy.array(1.999999999999999, dtype=numpy.float64)
c = a + b

print('a = {}'.format(a))
print('b = {}'.format(b))
print('c = a + b'.format(c, a, b))
print('c = {}'.format(c))
```
a = 4294967294.0
b = 1.999999999999999
c = a + b
c = 4294967296.0

@anaruse what do you mean by "this"? Do you mean your code example? I believe, its behavior is defined by IEEE 754. 
@akopich, I simply wanted to make sure this is not a CuPy issue, do you think this is a CuPy issue?
Ah, I think I finally figured out what you're saying is the issue.
In numpy.random.uniform, "The high limit may be included in the returned array of floats due to floating-point rounding in the equation low + (high-low) * random_sample(). ".
You're saying that the problem is that there is no such an explanation in the cupy.random.uniform document?
@anaruse
I was just about to explain this.
I see two options here. Either the documentation should explicitly cover a possibility for `high` to be generated (the approach `numpy` and the standard library follow), or the code shouldn't yield `high`.

@akopich, Could you take a look at https://github.com/cupy/cupy/pull/4153?  Probably, this PR will fix the issue.

At first I didn't thinks there was an easy solution for this issue, but I realized that CUDA has intrinsics that allow you to specify a rounding mode (https://docs.nvidia.com/cuda/floating-point/index.html#rounding-modes), so if you select "round toward negative" for the rounding mode, that should solve this issue.


The specification change is trade-off.  `random.uniform(...).mean()` can be much different than before.  Without the new spec, it is easy to avoid `high` value by `x[x == high] = low`.
@toslunar I don't quite understand which change will cause a significant shift of the mean in your opinion. 

> `x[x == high] = low`

the choice of the RHS of the expression seems rather arbitrary to me. Why should we (in some cases, significantly) increase the mass assigned to `low`? We can also do `x[x == high] = (low+high)/2` to keep the mean intact.  
> shift of the mean

My opinion is that the result using round-toward-negative https://github.com/cupy/cupy/pull/4153#issuecomment-712578483 is not desirable to everyone.

The result with `x[x == high] = low` (maybe `x[x >= high] = low`) looks similar to the round-toward-negative result, but I'm not sure if it's always true.
I found the discussion in NumPy numpy/numpy#10803.  I'll read it soon.
@toslunar 
We're dealing with the expression `round(low+round(r*round(high-low)))`. Let `D` denote the largest distance between the two consecutive floats in the interval `[low, high]`. The change of the rounding policy will shift the value of `round(high-low)` by at most `D`. As long as `|r|<=1`, the value `r*round(high-low)` will be also shifted by at most `D`. Hence, `round(r*round(high-low))` shall not be shifted by more than `2D` and  `round(low+round(r*round(high-low)))` -- by `3D`. Clearly, the mean will not be shifted by more than `3D` either. And it doesn't strike me as a significant change. 

> The result with `x[x == high] = low` (maybe `x[x >= high] = low`) looks similar to the round-toward-negative...

Why, is it `low` on RHS? Why not smth like `std::nextafter(high, 0.0)`? 

Thank you for referencing the numpy issue. Looks like they found it possible to just change the rounding mode, but lest they break the others' code, they just edited the docs. 
Talked offline with @anaruse and dev members regarding this, we will keep the current implementation as the current behavior is compatible with the NumPy's documented behavior. But we should this restriction documented in the NumPy docs to CuPy docs.

Resolving patch:
diff --git a/cupy/random/_distributions.py b/cupy/random/_distributions.py
--- a/cupy/random/_distributions.py
+++ b/cupy/random/_distributions.py
@@ -834,7 +834,8 @@ def uniform(low=0.0, high=1.0, size=None, dtype=float):
     """Returns an array of uniformly-distributed samples over an interval.
 
     Samples are drawn from a uniform distribution over the half-open interval
-    ``[low, high)``.
+    ``[low, high)``. The samples may contain the ``high`` limit due to
+    floating-point rounding.
 
     Args:
         low (float): Lower end of the interval.


Test patch: