Repository: Theano/Theano

Issue:
test_conv3d_bwd failing 
I am getting 15-20 failing tests for bleeding edge w/ Ubuntu 16.04, Cuda 8.0.44, cuDNN 5105, and theano 0.9.0dev3.dev-e6a3b009bc4ce68fb5ba711d775689ff41d8a587. 

All of the failing tests are WrongValue errors in test_conv_3d_bwd. The number of failing tests varies from run to run. The mean abs differences are not small. 

Tests run with the command: `THEANO_FLAGS=device=cpu nosetests -sdv theano.sandbox.cuda.tests.test_dnn > test-errors.log 2>&1`

I have fast_math on in my theanorc but did not make a difference when I turned it off. GPU is a GTX 1080. GCC is 5.4. Happy to give you any additional information needed to diagnose.

Full text of errors attached: 

[test-errors.txt](https://github.com/Theano/Theano/files/553424/test-errors.txt)



Discussion:
What are your other theano flags? Do you have some cudnn conv flags like
time_once or something else?

I agree the relative error seem high.

On Wed, Oct 26, 2016 at 10:16 AM, Jonathan Strong notifications@github.com
wrote:

> I am getting 15-20 failing tests for bleeding edge w/ Ubuntu 16.04, Cuda
> 8.0.44, cuDNN 5105, and theano 0.9.0dev3.dev-
> e6a3b009bc4ce68fb5ba711d775689ff41d8a587.
> 
> All of the failing tests are WrongValue errors in test_conv_3d_bwd. The
> number of failing tests varies from run to run. The mean abs differences
> are not small.
> 
> Tests run with the command: THEANO_FLAGS=device=cpu nosetests -sdv
> theano.sandbox.cuda.tests.test_dnn > test-errors.log 2>&1
> 
> I have fast_math on in my theanorc but did not make a difference when I
> turned it off. GPU is a GTX 1080. GCC is 5.4. Happy to give you any
> additional information needed to diagnose.
> 
> Full text of errors attached:
> 
> test-errors.txt
> https://github.com/Theano/Theano/files/553424/test-errors.txt
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/Theano/Theano/issues/5143, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AALC-48VUffPaPO_mxDh5eoy9krwq4RTks5q32DBgaJpZM4KhOW3
> .

yes - I did have time_on_shape_change for algo_fwd, algo_bwd_data and algo_bwd_filter. I also had cnmem on. Running without those I got no errors. 

cnmem should not affect the tests.

On Wed, Oct 26, 2016 at 5:37 PM, Jonathan Strong notifications@github.com
wrote:

> yes - I did have time_on_shape_change for algo_fwd, algo_bwd_data and
> algo_bwd_filter. I also had cnmem on. Running without those I got no
> errors.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/Theano/Theano/issues/5143#issuecomment-256484625, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AALC-7EWOAfLyD2zKqfukMNO1FQ0hcSDks5q38gBgaJpZM4KhOW3
> .

I ran again w/ cnmem on, but time_on_shape_change off and also passed. That was the problem. Slightly concerning, however, that that setting would have caused that. It didn't occur to me that it would impact the calculations so drastically. 

Yes, this is worrying that the results are so far off (relative error consistently around 0.5).
I'm not sure how this is possible.
I'll try to reproduce, in the mean time, could you try each of the possible values for algo_bwd_filter, so we can try to pinpoint if a specific one has an issue?
Hello !

To work around this issue, I have temporarly created a theano branch with modified printed infos, and I have run the tests for all `algo_bwd_filter` values (even for the values not allowed with 3D convolutions) with cnmem on and set to 1024 Mbytes. For cnmem I use this value instead of 5000 (as  @jonathanstrong did) because it seems it is better to let the maximum free available memory to cuDNN when it has to choose an algorithm.

Tests are run in 2 cases:
* with a GeForce 840M gpu, on my computer (2 Gb of gpu memory). cuDNN version is 5105 (as in the issue).
* with a GTX 1080 card, on MILA machines via SLURM (8 Gb of gpu memory). cuDNN version is 5103 (pre-installed, not the same as in the issue, but I don't know if it has some effect).

Here is the theano branch used: https://github.com/notoraptor/theano/tree/issue-5143
Here are the results: https://github.com/notoraptor/theano-issue-5143
Here is the general format of the command line used for the tests:
```
THEANO_FLAGS=device=cpu,floatX=float32,exception_verbosity=high,lib.cnmem=1024,dnn.conv.algo_bwd_filter=<algo> nosetests -sdv theano.sandbox.cuda.tests.test_dnn:test_conv3d_bwd > test-only-bwd-filter-<algo>.log 2>&1
```
The complete test script is here: https://github.com/notoraptor/theano-issue-5143/blob/master/tests-only-bwd-filter.sh

Then, I have no more WrongValue errors on the GTX 1080 while I have still these errors in the GeForce 840M. Some RuntimeErrors (`RuntimeError: GpuDnnConvGradW: error getting worksize: CUDNN_STATUS_NOT_SUPPORTED`) occur in both cases when `algo_bwd_filter` is set to `small`. This error makes all tests fail on the GTX, while I have sometimes RuntimeError and sometimes WrongValue error with GeForce.

So it seems it is mainly a problem related to the amount of free memory available on the GPU card. For the moment I don't know if there is a way to warn the user, for example if he tries to let cuDNN choose an algo when the memory is not big enough. I am looking for that.

About the `small` algorithm, it is listed in the "good algorithms" in `GpuDnnConv3dGradW` constructor ( https://github.com/Theano/Theano/blob/master/theano/sandbox/cuda/dnn.py#L762 ) but the theano doc says this algorithm is not available for 3D convolutions ( http://deeplearning.net/software/theano_versions/dev/library/config.html?highlight=algo_bwd_filter#config.config.dnn.conv.algo_bwd_filter ), so I plan to make a PR to remove it from this "good algorithms" list.

What do you think about it, @lamblin @nouiz ?!
Hello!

After further investigations, it appears that the problem does not come from memory (as I said before) but rather from the last cuDNN version.

I have updated Theano to a recent version (up to commit https://github.com/Theano/Theano/commit/c072d6691ff3a98cde74cbae5ca45cb9ba25b504 ) and I have installed the latest version of cuDNN (version 5105, the same as in this issue) on the computer with a GTX 1080 card, and then the problems occur.

I have also run the equivalent tests in the new backends (`theano.gpuarray.tests.test_dnn:test_conv3d_bwd`, with `gpuarray.preallocate=1024` in place of `lib.cnmem=1024`), and the same errors occur: some "WrongValue" errors (+ some "RuntimeError" when algo is `small`) in both GPU cards and in both backends.

NB: new backend raises a ValueError (`ValueError: ("convolution algo %s can't be used for 3d convolutions", ('fft',))`) when algo is `deterministic` or `fft`.

In the cuDNN PDF documentation, we can find this text (also related in theano dev doc: http://deeplearning.net/software/theano_versions/dev/library/gpuarray/dnn.html ):

> 2.5. Reproducibility (determinism)
> By design, most of cuDNN's routines from a given version generate the same bit-wise
> results across runs when executed on GPUs with the same architecture and the same
> number of SMs. However, bit-wise reproducibility is not guaranteed across versions,
> as the implementation of a given routine may change. With the current release, the
> following routines do not guarantee reproducibility because they use atomic operations:
> * cudnnConvolutionBackwardFilter when CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0 or CUDNN_CONVOLUTION_BWD_FILTER_ALGO_3 is used
> * cudnnConvolutionBackwardData when CUDNN_CONVOLUTION_BWD_DATA_ALGO_0 is used
> * cudnnPoolingBackward when CUDNN_POOLING_MAX is used
> * cudnnSpatialTfSamplerBackward

Maybe the problem comes from that (that is, the implementation of `CUDNN_CONVOLUTION_BWD_FILTER_ALGO_3` may have changed from cuDNN version 5103 to version 5105?).

All test results are here: https://github.com/notoraptor/theano-issue-5143

@nouiz @lamblin 

Resolving patch:
diff --git a/theano/gpuarray/dnn.py b/theano/gpuarray/dnn.py
--- a/theano/gpuarray/dnn.py
+++ b/theano/gpuarray/dnn.py
@@ -720,6 +720,11 @@ def get_op_params(self):
         return defs
 
     def make_node(self, img, topgrad, output, desc, alpha=None, beta=None):
+        if img.type.ndim == 5 and self.algo != 'none' and desc.owner.op.subsample != (1, 1, 1):
+            warnings.warn('cuDNN backward filter operation for 3D convolutions may produce bad results '
+                          'with certain cuDNN algorithms depending on the compute capability of your GPU '
+                          'if subsample is not (1, 1, 1). If you encounter problems, consider '
+                          'setting the theano flag "dnn.conv.algo_bwd_filter" to "none".')
         ctx_name = infer_context_name(img, topgrad, output)
         img = as_gpuarray_variable(img, ctx_name)
         topgrad = as_gpuarray_variable(topgrad, ctx_name)
diff --git a/theano/sandbox/cuda/dnn.py b/theano/sandbox/cuda/dnn.py
--- a/theano/sandbox/cuda/dnn.py
+++ b/theano/sandbox/cuda/dnn.py
@@ -789,6 +789,11 @@ def grad(self, inp, grads):
                 DisconnectedType()(), d_alpha, d_beta)
 
     def make_node(self, img, topgrad, output, desc, alpha=None, beta=None):
+        if self.algo != 'none' and desc.owner.op.subsample != (1, 1, 1):
+            warnings.warn('cuDNN backward filter operation for 3D convolutions may produce bad results '
+                          'with certain cuDNN algorithms depending on the compute capability of your GPU '
+                          'if subsample is not (1, 1, 1). If you encounter problems, consider '
+                          'setting the theano flag "dnn.conv.algo_bwd_filter" to "none".')
         img = as_cuda_ndarray_variable(img)
         topgrad = as_cuda_ndarray_variable(topgrad)
         output = as_cuda_ndarray_variable(output)


Test patch:
diff --git a/theano/gpuarray/tests/test_dnn.py b/theano/gpuarray/tests/test_dnn.py
--- a/theano/gpuarray/tests/test_dnn.py
+++ b/theano/gpuarray/tests/test_dnn.py
@@ -973,7 +973,6 @@ def get_conv3d_test_cases():
                    [(8, 4, 20, 12, 15), (5, 4, 6, 12, 4), (2, 2, 2)],
                    [(8, 1, 20, 12, 15), (5, 1, 6, 12, 4), (3, 3, 3)],
                    [(8, 1, 20, 12, 15), (5, 1, 6, 12, 4), (3, 2, 1)],
-                   [(8, 1, 20, 12, 15), (5, 1, 6, 12, 4), (3, 2, 1)],
                    # Test with 1x1x1 filters
                    [(8, 1, 10, 10, 10), (10, 1, 1, 1, 1), (1, 1, 1)],
                    # Test with dimensions larger than 1024 (thread block dim)
diff --git a/theano/sandbox/cuda/tests/test_dnn.py b/theano/sandbox/cuda/tests/test_dnn.py
--- a/theano/sandbox/cuda/tests/test_dnn.py
+++ b/theano/sandbox/cuda/tests/test_dnn.py
@@ -1554,7 +1554,6 @@ def get_conv3d_test_cases():
                    [(8, 4, 20, 12, 15), (5, 4, 6, 12, 4), (2, 2, 2)],
                    [(8, 1, 20, 12, 15), (5, 1, 6, 12, 4), (3, 3, 3)],
                    [(8, 1, 20, 12, 15), (5, 1, 6, 12, 4), (3, 2, 1)],
-                   [(8, 1, 20, 12, 15), (5, 1, 6, 12, 4), (3, 2, 1)],
                    # Test with 1x1x1 filters
                    [(8, 1, 10, 10, 10), (10, 1, 1, 1, 1), (1, 1, 1)],
                    # Test with dimensions larger than 1024 (thread block dim)