Repository: Theano/Theano

Issue:
libgpuarray / cudnn issues
I once had a working cuda/theano/libgpuarry setup working on windows 10 but since a reinstall I have not been able to get a python setup working with cuda and libgpuarray. Using both Anaconda's python and WinPython, when using the old backend, the common theano gpu test works.

```
python D:\test.py
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release.  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce GTX 1060 3GB (CNMeM is enabled with initial size: 10.0% of memory, cuDNN 5110)
[GpuElemwise{exp,no_inplace}(<CudaNdarrayType(float32, vector)>), HostFromGpu(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.402000 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu
```

But when I switch to using the libgpuarray backend cudnn can no longer be found and gpu is not used.

```
python D:\test.py
Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
c:\users\ryanhope\appdata\local\temp\try_flags_zq9bx6.c:4:19: fatal error: cudnn.h: No such file or directory
compilation terminated.

Mapped name None to device cuda: GeForce GTX 1060 3GB (0000:01:00.0)
[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.433000 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the cpu
```

FWIW, I can import pygpu no problem but pygpu.test() eventually crashes.


Discussion:
If you build the packages yourself, you must make sure that you use the same compiler for the library as the one that was used for the python binary otherwise you can encounter weird crashes.

As for why the header is not found you can try setting the Theano flags `dnn.include_path` and `dnn.library_path` to the path where you installed the files.  The difference is probably that the old backend includes the cuda folders by default and the new one doesn't.

For the anaconda python you can try the release candidate packages with `conda install -c rdonnelly pygpu`.
I cleaned up my system of any trace of libgpuarray and pygpu and installed them with conda. That at least allowed all of the pygpu tests to pass. However, things are still not quite right. With no other changes to my .theanorc I still had the issue of the cudnn headers not being found. I used your suggestion to add the dnn.include/library options to my config but on loading theano a new header could not be found. That header was back in the cuda include directory so I realised this issue must be related to the spaces in the path name. I copied the whole cuda directory to a new location without spaces and updated my .theanorc and theano was able to load cleanly.

```
Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jun 29 2016, 11:07:13) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import theano
Using cuDNN version 5110 on context None
Preallocating 307/3072 Mb (0.100000) on cuda
Mapped name None to device cuda: GeForce GTX 1060 3GB (0000:01:00.0)
>>>
```
While better this does not look perfect. I have "device = cuda" in my .theanorc so why is libgpuarray say None? At this point I decided to run the cpu/gpu test script and got this error:

```
Exception: ('The following error happened while compiling the node', GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>), '\n', 'Compilation failed (return status=1): C:\\Users\\RyanHope\\AppData\\Local\\Theano\\compiledir_Windows-10-10.0.14393-Intel64_Family_6_Model_42_Stepping_7_GenuineIntel-2.7.12-64\\tmpnbs0d4\\mod.cpp:4:28: fatal error: gpuarray/array.h: No such file or directory\r. compilation terminated.\r. ', '[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>)]')
```
I fixed this by copying the gpuarray include folder ino the pygpy directory in site-packages. This then led to the gpuarray lib not being found:

```
Exception: ('The following error happened while compiling the node', GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>), '\n', 'Compilation failed (return status=1): D:/mingw-w64/x86_64-5.4.0-posix-seh-rt_v5-rev0/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/5.4.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lgpuarray\r. collect2.exe: error: ld returned 1 exit status\r. ', '[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>)]')
```
Since none of its installed locations were on any of the linker paths I copied the gpuarray.lib to the base Anaconda2 folder. That allowed the test script to run without any errors but it did not run on the gpu.

There is also another issue regarding libgpuarray backend + cuDNN.

I had CuDNN installed under `/opt/CUDNN/cudnn-v5.1` and CUDA under `/opt/cuda-7.5`. The CUDA installations are clean CUDA SDK, without CUDNN libraries or headers copied inside.

The old backend works like a charm with the following configuration:
```
[cuda]
root = /opt/cuda-7.5

[dnn]
library_path = /opt/CUDNN/cudnn-v5.1/lib64
include_path = /opt/CUDNN/cudnn-v5.1/include
```

but the libgpuarray fails probably because it expects CUDNN libs and headers inside CUDA installation.
```
THEANO_FLAGS = device=cuda0
Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
In file included from /tmp/try_flags_IeoZNM.c:4:0:
/opt/CUDNN/cudnn-v5.1/include/cudnn.h:63:26: erreur fatale: driver_types.h : Aucun fichier ou dossier de ce type
```

I think changing backend should not necessitate moving/copying files around and the behavior should be identical.

Update: I think this may be because of the fact that old backend compiles the test code with nvcc while the new uses gcc. nvcc probably has builtin include_path tied to the SDK folder.
I run into the same problem, 
"Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
c:\users\ryanhope\appdata\local\temp\try_flags_zq9bx6.c:4:19: fatal error: cudnn.h: No such file or directory"
but is ok with gpu backend
I got that same error, worked around it by copying cudnn into the cuda directory. See [here](https://github.com/valtron/llvm-stuff/wiki/Theano,-CUDA,-CuDNN-on-Win10) for a guide.

Now I'm running into a ton of errors during compilation:

```
%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:698:16: error: 'input_node_m150592bf75eb7b96b1e07943b9cb84a9_0' does not name a type

                ^
%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:671:29: note: in definition of macro 'APPLY_SPECIFIC'
 #undef ITEMSIZE_INPUT_5
                             ^
%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:699:16: error: 'output_node_m150592bf75eb7b96b1e07943b9cb84a9_0' does not name a type
 double __DUMMY_11;
                ^
%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:671:29: note: in definition of macro 'APPLY_SPECIFIC'
 #undef ITEMSIZE_INPUT_5
                             ^
%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:700:16: error: 'kerns_node_m150592bf75eb7b96b1e07943b9cb84a9_0' does not name a type
 __label_13:
                ^

... etc ...
```

The line numbers of the errors seem to be wrong, because it points to parts of the code that have nothing to do with the error. Does anyone know how to fix this?

Resolving patch:
diff --git a/theano/gpuarray/dnn.py b/theano/gpuarray/dnn.py
--- a/theano/gpuarray/dnn.py
+++ b/theano/gpuarray/dnn.py
@@ -233,17 +233,23 @@ def c_headers(self):
                 'gpuarray_helper.h']
 
     def c_header_dirs(self):
-        return [os.path.dirname(__file__), pygpu.get_include(),
-                config.dnn.include_path]
+        dirs = [os.path.dirname(__file__), pygpu.get_include()]
+        if config.dnn.include_path:
+            dirs.append(config.dnn.include_path)
+        return dirs
 
     def c_libraries(self):
         return ['cudnn', 'gpuarray']
 
     def c_lib_dirs(self):
-        return [config.dnn.library_path]
+        if config.dnn.library_path:
+            return [config.dnn.library_path]
+        return []
 
     def c_compile_args(self):
-        return ['-Wl,-rpath,' + config.dnn.library_path]
+        if config.dnn.library_path:
+            return ['-Wl,-rpath,"' + config.dnn.library_path + '"']
+        return []
 
     def c_code_cache_version(self):
         return (super(DnnBase, self).c_code_cache_version(), version(), 1)
@@ -256,16 +262,20 @@ def c_headers(self):
         return ['cudnn.h']
 
     def c_header_dirs(self):
-        return [config.dnn.include_path]
+        return [config.dnn.include_path] if config.dnn.include_path else []
 
     def c_libraries(self):
         return ['cudnn']
 
     def c_lib_dirs(self):
-        return [config.dnn.library_path]
+        if config.dnn.library_path:
+            return [config.dnn.library_path]
+        return []
 
     def c_compile_args(self):
-        return ['-Wl,-rpath,' + config.dnn.library_path]
+        if config.dnn.library_path:
+            return ['-Wl,-rpath,"' + config.dnn.library_path + '"']
+        return []
 
     def c_support_code(self):
         return """


Test patch: