Repository: lisa-lab/pylearn2

Issue:
speed up NanGuardMode on GPU and move to Theano.
This can be done as in gh-1054. Do the reduction on the GPU, then this will transfer much less data.

The CudaNdarray object do not support many reduction, but we can compile a Theano function that take a gpu object, do the reduction and return the result on the CPU to inspect it.



Discussion:
Hi 

I think this ticket is to change something like "np.any(np.isnan(X))" to "np.isnan(np.min(X))", thus to speed up right?

After tracing the method that NanGuardMode uses, I find that it is already using the faster way accordingly, in file /pylearn2/utils/general.py . 

Actually, this ticket is about not performing NumPy operations (such as `np.min` and `np.isnan`) when `X` is on GPU (is a `CudaNdarray`, not an ndarray), because in that case, the whole array is first copied on CPU memory as an ndarray, and then `np.min` and `np.isnan` are called.
What we want is that at least the `min` operation runs directly on GPU, and then we can transfer only one number to the CPU.

@hantek Any update on this?

Hi David

Can we find some codes using the NanGuardMode class? I am still not sure how to use that mode in a sample computational graph. It seems that the constructor doesn't need information about the nodes, but in the **init**() function it needs "node" and "fn" to be the parameter for nan_check().

And, what is a "thunk" in theano?

http://deeplearning.net/software/theano/extending/pipeline.html#step-3-execute-linker-to-obtain-a-thunk

You can try this out by compiling a theano function with
`mode=NanGuardMode()`. Just create a function that intentionally gets NaNs
somehow.

On Wed, Apr 8, 2015 at 1:05 AM, Zhouhan LIN notifications@github.com
wrote:

> Hi David
> 
> Can we find some codes using the NanGuardMode class? It seems that the
> constructor doesn't need information about the nodes, but in the _init_()
> function it needs "node" and "fn" to be the parameter for nan_check().
> 
> And, what is a "thunk" in theano?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/lisa-lab/pylearn2/issues/1172#issuecomment-90802731.

OK. Thanks a lot!

Could you tell me which part of code transfers the data on GPU onto CPU? I looked at the inputs and outputs property of fn, but they are already numpy.ndarray objects. and from other properties of fn I didn't find one with CudaNdarray format. Where can I find the CudaNdarray?

I have found one with a more complex computation graph.

in the nan_check() function, node.out is a CudaNdarraySharedVariable, which can be used to compute a theano function, and elements in fn.inputs (or fn.outputs) can be either numpy.ndarray or CudaNdarray. 

If I use node.out to compile the theano function, then the compiled function is requiring inputs, which would actually cause data transfer between CPU and GPU. So, is there a way to utilize elements in fn.outputs directly? I find that it has values already and it is stored on GPU.

Hi
Just to note that I kind have know how to do it after talking with Pascal, so there will be a PR shortly.


Resolving patch:
diff --git a/pylearn2/devtools/nan_guard.py b/pylearn2/devtools/nan_guard.py
--- a/pylearn2/devtools/nan_guard.py
+++ b/pylearn2/devtools/nan_guard.py
@@ -11,6 +11,8 @@
 import logging
 from theano.compile import Mode
 import theano
+import theano.tensor as T
+import theano.sandbox.cuda as cuda
 import numpy as np
 from pylearn2.models.dbm import flatten
 from pylearn2.utils import contains_nan, contains_inf
@@ -36,6 +38,23 @@ class NanGuardMode(Mode):
         If True, raise an error when a value greater than 1e10 is encountered.
     """
     def __init__(self, nan_is_error, inf_is_error, big_is_error=True):
+        if cuda.cuda_available:
+            self.guard_input = cuda.fvector('nan_guard')
+            if nan_is_error or inf_is_error:
+                self.gpumin = theano.function(
+                    [self.guard_input], T.min(self.guard_input),
+                    mode='FAST_RUN'
+                )
+            if inf_is_error:
+                self.gpumax = theano.function(
+                    [self.guard_input], T.max(self.guard_input),
+                    mode='FAST_RUN'
+                )
+            if big_is_error:
+                self.gpuabsmax = theano.function(
+                    [self.guard_input], T.max(T.abs_(self.guard_input)),
+                    mode='FAST_RUN'
+                )
         def do_check_on(var, nd, f, is_input):
             """
             Checks `var` for NaNs / Infs. If detected, raises an exception
@@ -56,15 +75,31 @@ def do_check_on(var, nd, f, is_input):
             """
             error = False
             if nan_is_error:
-                if contains_nan(var):
+                err = False
+                if cuda.cuda_available and isinstance(var, cuda.CudaNdarray):
+                    err = np.isnan(self.gpumin(var.reshape(var.size)))
+                else:
+                    err = contains_nan(var)
+                if err:
                     logger.error('NaN detected')
                     error = True
             if inf_is_error:
-                if contains_inf(var):
+                err = False
+                if cuda.cuda_available and isinstance(var, cuda.CudaNdarray):
+                    err = (np.isinf(self.gpumin(var.reshape(var.size))) or \
+                           np.isinf(self.gpumax(var.reshape(var.size))))
+                else:
+                    err = contains_inf(var)
+                if err:
                     logger.error('Inf detected')
                     error = True
             if big_is_error:
-                if np.abs(var).max() > 1e10:
+                err = False
+                if cuda.cuda_available and isinstance(var, cuda.CudaNdarray):
+                    err = (self.gpuabsmax(var.reshape(var.size)) > 1e10)
+                else:
+                    err = (np.abs(var).max() > 1e10)
+                if err:
                     logger.error('Big value detected')
                     error = True
             if error:


Test patch:
diff --git a/pylearn2/devtools/tests/test_nan_guard.py b/pylearn2/devtools/tests/test_nan_guard.py
new file mode 100644
--- /dev/null
+++ b/pylearn2/devtools/tests/test_nan_guard.py
@@ -0,0 +1,49 @@
+"""
+This test is for testing the NanGuardMode.
+"""
+from pylearn2.devtools.nan_guard import NanGuardMode
+import numpy
+import theano
+import theano.tensor as T
+
+
+def test_NanGuardMode():
+    """
+    Tests if NanGuardMode is working by feeding in numpy.inf and numpy.nans
+    intentionally. A working implementation should be able to capture all
+    the abnormalties.
+    """
+    x = T.matrix()
+    w = theano.shared(numpy.random.randn(5, 7).astype(theano.config.floatX))
+    y = T.dot(x, w)
+
+    fun = theano.function(
+        [x], y,
+        mode=NanGuardMode(nan_is_error=True, inf_is_error=True)
+    )
+    a = numpy.random.randn(3, 5).astype(theano.config.floatX)
+    infa = numpy.tile(
+        (numpy.asarray(100.) ** 1000000).astype(theano.config.floatX), (3, 5))
+    nana = numpy.tile(
+        numpy.asarray(numpy.nan).astype(theano.config.floatX), (3, 5))
+    biga = numpy.tile(
+        numpy.asarray(1e20).astype(theano.config.floatX), (3, 5))
+
+    work = [False, False, False]
+
+    fun(a)  # normal values
+    try:
+        fun(infa)  # INFs
+    except AssertionError:
+        work[0] = True
+    try:
+        fun(nana)  # NANs
+    except AssertionError:
+        work[1] = True
+    try:
+        fun(biga)  # big values
+    except AssertionError:
+        work[2] = True
+
+    if not (work[0] and work[1] and work[2]):
+        raise AssertionError("NanGuardMode not working.")