Repository: Theano/Theano

Issue:
[Mac OS 10.9.2][CUDA 5.5] "nvcc compiler not found on $PATH"
TDLR: bleeding edge theano CUDA support _seems_ broken on Mac with latest OS/CUDA versions. Quick fix is to downgrade to the pip release version.

Hello,

I can provide additional information if needed. Here is a way to reproduce it:

```
~ $ pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git
[...]
~ $ python
Python 2.7.6 (default, Mar 25 2014, 16:47:32)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
ERROR (theano.sandbox.cuda): nvcc compiler not found on $PATH. Check your nvcc installation and try again.
>>>
~ $ echo $PATH
/Developer/NVIDIA/CUDA-5.5/bin:/Users/gabrielsynnaeve/local/bin:/bin:/usr/local/sbin:/usr/local/bin:/Users/gabrielsynnaeve/.rvm/gems/ruby-2.0.0-p0/bin:/Users/gabrielsynnaeve/.rvm/gems/ruby-2.0.0-p0@global/bin:/Users/gabrielsynnaeve/.rvm/rubies/ruby-2.0.0-p0/bin:/Users/gabrielsynnaeve/.rvm/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/texbin:/Users/gabrielsynnaeve/.gem/ruby/1.8/bin:/usr/sbin:/usr/local/Cellar/ruby/1.9.3-p374/bin:/Users/gabrielsynnaeve/bin:/Users/gabrielsynnaeve/Library/Haskell/bin:/usr/local/share/npm/bin:/Users/gabrielsynnaeve/.rvm/bin:/Users/gabrielsynnaeve/postdoc/htk/HTKTools:/Users/gabrielsynnaeve/labs/pylearn2/pylearn2/scripts:/Users/gabrielsynnaeve/gocode/bin
~ $ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2013 NVIDIA Corporation
Built on Thu_Sep__5_10:17:14_PDT_2013
Cuda compilation tools, release 5.5, V5.5.0
```

I tried to debug it by doing:

```
In [1]: import theano.sandbox as ts
ERROR (theano.sandbox.cuda): nvcc compiler not found on $PATH. Check your nvcc installation and try again.

In [2]: %debug ts.cuda.nvcc_compiler.is_nvcc_available()
```

And it crashes python (external crash popup) at:

```
ipdb> > /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py(1211)_after_fork()
   1210
-> 1211 def _after_fork():
   1212     # This function is called by Python/ceval.c:PyEval_ReInitThreads which

ipdb> The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

> /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py(1228)_execute_child()
   1227                     self._child_created = True
-> 1228                     if self.pid == 0:
   1229                         # Child
```

Quick-fix is to downgrade to release version 

```
~ $ pip install --upgrade --no-deps --force-reinstall theano
[...]
~ $ python
Python 2.7.6 (default, Mar 25 2014, 16:47:32)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
Using gpu device 0: GeForce GT 650M
```

Cheers to all the theano contributors for an amazing piece of software nonetheless (no sarcasm)! :)
Gabriel



Discussion:
I can't "bisect" to tell you when, but bleeding edge was previously working for me with this CUDA version (~December, that's still the last one). Also it works fine on the pip release version both with clang (OSX default compiler) and gcc-4.8 (installed by homebrew).

The debug crash is most likely due to the inner workings of %debug in ipython (which seems to be forking improperly).

Also, have you tried just typing 'nvcc' in a terminal window?

nvcc works fine. See the previous trace + the fact that it works with the
current pip release.

Sorry about that, I missed the terminal trace.

I don't currently have access to a mac with cuda installed and I don't know when I will.

If you want to debug this I strongly suspect something is wrong in theano/sandbox/cuda/nvcc_compiler.py:is_nvcc_available().

You could try replacing the `except Exception` clauses by `except OSError` to see what kind of exception is raised.


Resolving patch:
diff --git a/theano/sandbox/cuda/nvcc_compiler.py b/theano/sandbox/cuda/nvcc_compiler.py
--- a/theano/sandbox/cuda/nvcc_compiler.py
+++ b/theano/sandbox/cuda/nvcc_compiler.py
@@ -99,7 +99,7 @@ def is_nvcc_available():
     """Return True iff the nvcc compiler is found."""
     def set_version():
         p_out = output_subprocess_Popen([nvcc_path, '--version'])
-        ver_line = decode(p_out[0].split('\n')[-1])
+        ver_line = decode(p_out[0]).strip().split('\n')[-1]
         build, version = ver_line.split(',')[1].strip().split()
 
         assert build == 'release'


Test patch: