Issue 1:
🐛 Bug

While looking at issue #26971, I wondered if torch's uniform number generators was generating too many zeros since it generates in average more than 5 zeros per 10^8 generations according to the following:

sum((torch.rand(100000000) <= 0.).sum().float() for i in range(20)) / 20

On GPU, it generates in average around 3 zeros per 10^8 generators:

sum((torch.rand(100000000, device=torch.device('cuda')) <= 0.).sum().float() for i in range(20)) / 20

If you do the same test with NumPy, you will see that it never generates zeros:

sum((np.random.rand(100000000).astype('float32') <= 0.).sum() for i in range(20)) / 20

(if you try with 'float16' instead of 'float32', you will see that it generates some zeros)

Thus, I think that PyTorch's generator might be biased towards zero and this may hide other biases that I'm not aware.

To Reproduce

See above.

Expected behavior

In my opinion, the behavior should be similar to NumPy's unless NumPy has a bug instead.

Environment

Collecting environment information...

PyTorch version: 1.2.0

Is debug build: No

CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.3 LTS

GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0

CMake version: Could not collect

Python version: 3.6

Is CUDA available: Yes

CUDA runtime version: 9.1.85

GPU models and configuration:

GPU 0: GeForce GTX 1080 Ti

GPU 1: GeForce GTX 1080 Ti

GPU 2: GeForce GT 1030

Nvidia driver version: 430.26

cuDNN version: Could not collect

Versions of relevant libraries:

[pip3] msgpack-numpy==0.4.3.1

[pip3] numpy==1.16.2

[pip3] numpydoc==0.8.0

[pip3] torch==1.2.0

[pip3] torchvision==0.4.0a0+6b959ee

[conda] blas 1.0 mkl

[conda] mkl 2019.1 144

[conda] mkl-service 1.1.2 py36he904b0f_5

[conda] mkl_fft 1.0.10 py36ha843d7b_0

[conda] mkl_random 1.0.2 py36hd81dba3_0

[conda] pytorch 1.2.0 py3.6_cuda10.0.130_cudnn7.6.2_0 pytorch

[conda] torch 1.1.0 pypi_0 pypi

[conda] torchvision 0.2.1 pypi_0 pypi

Issue 2:
🐛 Bug

When using uniformly-distributed random number generation, especially using float32, over a range including 0, the sampling code ignores the extra precision of floating-point around 0, leading to very bad sampling.

The generated floats also have at most 24 (float32) or 53 (float64) bits of entropy.

Consequence 1: Multiplying any PyTorch float32 uniform_(0,1) by a large integer like 2**33 is guaranteed to give an integer.

Consequence 2: It is actually possible to generate an exact 0.0 using the float32 uniform_(0,1), with probability 2**-24, whereas that should actually have been 2**-149.

Consequence 3: It is not possible to generate a float32 strictly between 0.0 and 2**-24 with uniform_(0,1), whereas it should have been possible with probability 2**-24.

To Reproduce

Test 1: If the following is != 0, the RNG sucks: torch.FloatTensor(int(1e8)).uniform_().eq(0).sum()

Test 2: If the following is == -inf, the RNG sucks: torch.FloatTensor(int(1e8)).uniform_().log().sum()

Expected behavior

Test 1: A return of 0.

Test 2: A finite, negative value.

Environment

Not relevant.

PyTorch Version 1.1.0a0+c865d46, but the same problem dates back to the beginning of the history of this code, which is over a year.

Additional context

This has serious knock-on effects on many other distributions that use uniform_() as a basis. In particular, log(uniform_()) has a 2**-24 chance of producing -inf, and log2(uniform_()) cannot produce a number < -24 (except -inf).

cc @pbelevich