Repository: Theano/Theano

Issue:
MRG_RandomStreams on GPU produces NaN when nstreams is large
For one of my models I need to generate a random matrix of size 10000x784. I had the impression I needed to set `nstreams` to the number of elements in my matrix. Doing so, produces NaN on the GPU (actually not for the whole matrix, only for the last couple of rows of the matrix).

Here is a [gist](https://gist.github.com/MarcCote/1f95703c6a3e599d6229) reproducing the problem on a GeForce GTX 660 (960 CUDA cores, 1.5 Go RAM).

Now I know better, I've read the [comment](https://github.com/Theano/Theano/blob/master/theano/sandbox/rng_mrg.py#L1018) in `sandbox/rng_mrg.py:1018`, so I set `nstreams=960` instead. Still, I thought I should report this unexpected behavior when used on GPU since on CPU it works fine.



Discussion:
We very recently disabled the warning about n_streams since it mostly confuses people.  I guess that you saw it and felt the you had to set the value.

We use some form of heuristic for 280 and 480, but then again I don't think anybody did extensive research on what was the most efficient configuration so maybe it doesn't change much, or actually hurts to change it.

I would strongly advise to just leave it at the default (and ignore the warning) unless you notice a performance problem.

That being said, I think we should error out rather than fail when it is set to something very large, I'm just not sure how to do it (maybe just a maximum value?).

It is possible that the computation is not done correctly when n_streams is too large, we should investigate that, and fail with a meaningful message.

I'm not able to reproduce this problem. I remember there was some problems in the past on the last number in the sequences. Which version of Theano do you have? Can you update to the dev version and test again?

Just tested with the dev version (and also before I updated to the latest commit), the NaN are gone but still have problem. There are some sort of int (or uninitialized memory values) at the end of the array instead!

Using gpu device 0: GeForce GTX 660
[ 394.48202515  388.72000122  399.98104858 ...,  388.24841309  403.51867676  386.09875488]
WARNING: unused streams above 1048576 (Tune GPU_mrg get_n_streams)
[ 394.69119263  398.94842529  394.30621338 ...,   70.           82.          119.        ]

Probably the number I used depends on the number of CUDA cpus and GPU RAM.
In [3]: theano.__version__
Out[3]: '0.6.0.dev-5b2a9c5cf4466e92cddd9da98dff15430b568e24'

I tried your gist. For some runs, it seem to work correctly. Then later
runs started to show nan. Mayby it is just memory not initialized?

On Fri, Sep 5, 2014 at 2:51 PM, Marc-Alexandre Côté <
notifications@github.com> wrote:

> Just tested with the dev version, the NaN are gone but still have problem.
> There are some sort of int at the end of the array instead!
> 
> Using gpu device 0: GeForce GTX 660
> [ 394.48202515 388.72000122 399.98104858 ..., 388.24841309 403.51867676
> 386.09875488]
> WARNING: unused streams above 1048576 (Tune GPU_mrg get_n_streams)
> [ 394.69119263 398.94842529 394.30621338 ..., 70. 82. 119. ]
> 
> Probably the number I used depends on the number of CUDA cpus and GPU RAM.
> In [3]: theano.**version**
> Out[3]: '0.6.0.dev-5b2a9c5cf4466e92cddd9da98dff15430b568e24'
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/2076#issuecomment-54665968.

Does cuda-memcheck complain?

@lamblin Nope. I used default options for `cuda-memcheck`.

It looks like when nstreams is bigger than 256 \* 4096, which is the max number of blocks times the max number of streams per block, we do not actually compute the numbers, but let the memory un-initialized.
@sebastien-j: can you try changing line https://github.com/Theano/Theano/blob/master/theano/sandbox/rng_mrg.py#L589 from `if (idx < Nstreams_used)` to something like `for (; idx < Nstreams_used; idx += numThreads)`?
You may have to declare idx in the for loop instead, or declare it non-const.

Actually I would rather we place a limit on n_streams than extend the kernel.  But I guess that working code is better than what we have, whatever the solution.

Agreed. I think too that putting a limit on n_streams is good. Adding an
extra loop could slow things down. We don't want that.

On Tue, Sep 30, 2014 at 8:09 PM, abergeron notifications@github.com wrote:

> Actually I would rather we place a limit on n_streams than extend the
> kernel. But I guess that working code is better than what we have, whatever
> the solution.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/2076#issuecomment-57402117.

Why would the extra loop slow anything down if it is only executed once?
In general, I think it would be better if we support more cases, but in that case I don't really see a use case for more than 2**20 parallel streams, so I'm OK with a limit and a meaningful error message.

Extra loop mean more register. The number of register per threads is
limited. So it could slow things down. But that code wasn't optimized for
register usage, so we don't have information on how it could affect it or
not.

On Wed, Oct 1, 2014 at 3:01 PM, Pascal Lamblin notifications@github.com
wrote:

> Why would the extra loop slow anything down if it is only executed once?
> In general, I think it would be better if we support more cases, but in
> that case I don't really see a use case for more than 2**20 parallel
> streams, so I'm OK with a limit and a meaningful error message.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/2076#issuecomment-57517557.


Resolving patch:
diff --git a/theano/sandbox/rng_mrg.py b/theano/sandbox/rng_mrg.py
--- a/theano/sandbox/rng_mrg.py
+++ b/theano/sandbox/rng_mrg.py
@@ -734,6 +734,13 @@ def c_code(self, node, nodename, inp, out, sub):
             unsigned int threads_per_block = std::min((unsigned int)n_streams_used_in_this_call, (unsigned int)NUM_VECTOR_OP_THREADS_PER_BLOCK);
             unsigned int n_blocks = std::min(ceil_intdiv((unsigned int)n_streams_used_in_this_call, threads_per_block), (unsigned int)NUM_VECTOR_OP_BLOCKS);
 
+            if (n_streams > (unsigned int)NUM_VECTOR_OP_THREADS_PER_BLOCK * (unsigned int)NUM_VECTOR_OP_BLOCKS)
+            {
+                PyErr_Format(PyExc_ValueError, "On GPU, n_streams should be at most %%u",
+                    (unsigned int)NUM_VECTOR_OP_THREADS_PER_BLOCK * (unsigned int)NUM_VECTOR_OP_BLOCKS);
+                %(fail)s;
+            }
+
             if (threads_per_block * n_blocks < n_streams)
             {
                 if (! %(nodename)s_printed_warning)
@@ -761,7 +768,7 @@ def c_code(self, node, nodename, inp, out, sub):
         """ % locals()
 
     def c_code_cache_version(self):
-        return (8,)
+        return (9,)
 
 
 class GPUA_mrg_uniform(GpuKernelBase, mrg_uniform_base):


Test patch:
diff --git a/theano/sandbox/test_rng_mrg.py b/theano/sandbox/test_rng_mrg.py
--- a/theano/sandbox/test_rng_mrg.py
+++ b/theano/sandbox/test_rng_mrg.py
@@ -17,6 +17,7 @@
 from theano.tests import unittest_tools as utt
 from nose.plugins.skip import SkipTest
 from nose.plugins.attrib import attr
+from nose.tools import assert_raises
 
 #TODO: test gpu
 # Done in test_consistency_GPU_{serial,parallel}
@@ -305,6 +306,22 @@ def test_consistency_GPU_parallel():
     samples = numpy.array(samples).flatten()
     assert(numpy.allclose(samples, java_samples))
 
+def test_GPU_nstreams_limit():
+    """Verify that a ValueError is raised when n_streams
+    is greater than 2**20 on GPU. This is the value of
+    (NUM_VECTOR_OP_THREADS_PER_BLOCK * NUM_VECTOR_OP_BLOCKS).
+    """
+    if not cuda_available:
+        raise SkipTest('Optional package cuda not available')
+
+    seed = 12345
+    R = MRG_RandomStreams(seed=seed, use_cuda=True)
+    
+    def eval_uniform(size, nstreams):
+        return R.uniform(size=size, nstreams=nstreams, dtype='float32').eval()
+    
+    eval_uniform((10,), 2**20)
+    assert_raises(ValueError, eval_uniform, (10,), 2**20 + 1)
 
 def test_consistency_GPUA_serial():
     '''Verify that the random numbers generated by GPUA_mrg_uniform, serially,