Repository: Theano/Theano

Issue:
Compilation problem in virtualenv "cannot find -lInclude"
Lacking root permissions on Ubuntu 12.04 I compiled the recent Python 3.5 from source, setup a virtualenv and pip-ed Theano. 
When running `import theano` I get an error: ``/usr/bin/ld: cannot find -lInclude`. The flag is attached to the following compile statement:

```
Problem occurred during compilation with the command line below:
/usr/bin/g++ -shared -g -march=corei7-avx -mcx16 -msahf -mno-movbe -maes -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-tbm -mavx -msse4.2 -msse4.1 -mno-rdrnd -mno-f16c -mno-fsgsbase --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=20480 -mtune=generic -D NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -I/path/to/virtualenv/venv/lib/python3.5/site-packages/numpy/core/include -I/path/to/virtualenv/Python-3.5.0/Include -I/path/to/virtualenv/Python-3.5.0 -o /home/fgressmann/.theano/compiledir_Linux-3.2--generic-x86_64-with-debian-wheezy-sid-x86_64-3.5.0-64/lazylinker_ext/lazylinker_ext.so /home/fgressmann/.theano/compiledir_Linux-3.2--generic-x86_64-with-debian-wheezy-sid-x86_64-3.5.0-64/lazylinker_ext/mod.cpp -L/usr/local/lib -lInclude
```

Any ideas how to fix it?

The complete traceback:

```
ImportError                               Traceback (most recent call last)
/path/to/virtualenv/venv/lib/python3.5/site-packages/theano/gof/lazylinker_c.py in <module>()
     63         if version != getattr(lazylinker_ext, '_version', None):
---> 64             raise ImportError()
     65 except ImportError:

ImportError: 

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
/path/to/virtualenv/venv/lib/python3.5/site-packages/theano/gof/lazylinker_c.py in <module>()
     80             if version != getattr(lazylinker_ext, '_version', None):
---> 81                 raise ImportError()
     82         except ImportError:

ImportError: 

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
<ipython-input-1-1a165bb0dd56> in <module>()
      1 
----> 2 import theano

/path/to/virtualenv/venv/lib/python3.5/site-packages/theano/__init__.py in <module>()
     53     object2, utils
     54 
---> 55 from theano.compile import \
     56     SymbolicInput, In, \
     57     SymbolicOutput, Out, \

/path/to/virtualenv/venv/lib/python3.5/site-packages/theano/compile/__init__.py in <module>()
      7         SpecifyShape, specify_shape, register_specify_shape_c_code)
      8 
----> 9 from theano.compile.function_module import *
     10 
     11 from theano.compile.mode import *

/path/to/virtualenv/venv/lib/python3.5/site-packages/theano/compile/function_module.py in <module>()
     16 from theano import gof
     17 from theano.compat.python2x import partial
---> 18 import theano.compile.mode
     19 from theano.compile.io import (
     20     In, SymbolicInput, SymbolicInputKit, SymbolicOutput)

/path/to/virtualenv/venv/lib/python3.5/site-packages/theano/compile/mode.py in <module>()
      9 import  theano
     10 from theano import gof
---> 11 import theano.gof.vm
     12 from theano.configparser import config, AddConfigVar, StrParam
     13 from theano.compile.ops import register_view_op_c_code, _output_guard

/path/to/virtualenv/venv/lib/python3.5/site-packages/theano/gof/vm.py in <module>()
    566 
    567 try:
--> 568     from . import lazylinker_c
    569 
    570     class CVM(lazylinker_c.CLazyLinker, VM):

/path/to/virtualenv/venv/lib/python3.5/site-packages/theano/gof/lazylinker_c.py in <module>()
    114             args = cmodule.GCC_compiler.compile_args()
    115             cmodule.GCC_compiler.compile_str(dirname, code, location=loc,
--> 116                                              preargs=args)
    117             # Save version into the __init__.py file.
    118             init_py = os.path.join(loc, '__init__.py')

/path/to/virtualenv/venv/lib/python3.5/site-packages/theano/gof/cmodule.py in compile_str(module_name, src_code, location, include_dirs, lib_dirs, libs, preargs, py_module)
   2008             # difficult to read.
   2009             raise Exception('Compilation failed (return status=%s): %s' %
-> 2010                             (status, compile_stderr.replace('\n', '. ')))
   2011         elif config.cmodule.compilation_warning and compile_stderr:
   2012             # Print errors just below the command line.

Exception: Compilation failed (return status=1): /usr/bin/ld: cannot find -lInclude. collect2: ld returned 1 exit status. 
```



Discussion:
As requested in #2724 I put 

```
print libs 
print std_libs() 
print lib_dirs 
print std_lib_dirs()
```

to `theano/gof/cmodule.py:1975`. It yields:

```
['Include']
['Include']
['/usr/local/lib']
[]
```

So I have to find out where the Include comes in.

I don't have the time to look at it now, but you are in the right
direction. If you find the solution, tell us. Look in that method, you will
probably find there where this come from.

On Mon, Oct 26, 2015 at 9:58 AM, dudheit314 notifications@github.com
wrote:

> As requested in #2724 https://github.com/Theano/Theano/issues/2724 I
> put
> 
> print libs
> print std_libs()
> print lib_dirs
> print std_lib_dirs()
> 
> to theano/gof/cmodule.py:1975. It yields:
> 
> ['Include']
> ['Include']
> ['/usr/local/lib']
> []
> 
> So I have to find out where the Include comes in.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/3556#issuecomment-151141640.

The problem seems to come from `std_lib_dirs_and_libs()` in `theano/gof/cmodule.py:1567`: At line 1628 

```
libname = os.path.basename(python_inc)
```

python_inc holds `/path/to/local/compiled/Python-3.5.0/Include` and therefore adds an Include

Do this directory exist? If so, would using the full path work?

On Mon, Oct 26, 2015 at 10:52 AM, dudheit314 notifications@github.com
wrote:

> The problem seems to come from std_lib_dirs_and_libs() in
> theano/gof/cmodule.py:1567: At line 1628
> 
> libname = os.path.basename(python_inc)
> 
> python_inc holds /path/to/local/compiled/Python-3.5.0/Include and
> therefore adds an Include
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/3556#issuecomment-151163180.

Yes, the path is existing, but changing the line to `libname = python_inc` only brings up an analog error: `/usr/bin/ld: cannot find -l/path/to/local/compiled/Python-3.5.0/Include`
Even with permission 777 it does not work.

So I simply patched it with an if:

``` python
libname = os.path.basename(python_inc)
        if libname == "Include":
            std_lib_dirs_and_libs.data = [], []
        else:
            std_lib_dirs_and_libs.data = [libname], []
```

which seems to work so far.

I get this here:<

```
 theano.gof.cmodule.std_lib_dirs_and_libs()
(['python2.7'], [])
```

Can you confirm that you get:
`(['/path/to/local/compiled/Python-3.5.0/Include'], [])`

python_inc should be the name of a library, not a path.

Can you try to change this line:

```
python_inc = distutils.sysconfig.get_python_inc()
```

to this line:

```
python_inc = distutils.sysconfig.get_python_lib()
```

Before changing this in Theano, it would need test on mac and windows too.

Yes, that is what I get and the fix seems to work as well.

Can you make a PR with the fix that work with you? With that, we can ask a
few people to tests different configuration.

On Wed, Nov 4, 2015 at 2:53 AM, dudheit314 notifications@github.com wrote:

> Yes, that is what I get and the fix seems to work as well.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/3556#issuecomment-153624463.

Sorry, when preparing the pull request I recognized that I get a different output, I forget to remove previous if-statements. So the actual output of  `theano.gof.cmodule.std_lib_dirs_and_libs()` is `(['site-packages'], [])`
Sorry again for that mistake.

I'm facing the same problem with a local install of python2.7 on RHEL6 (no sudo rights, hence a local install). The variable python_inc has the value /home/users/username/python/Python-2.7.11/Include . 

When I run it distutils.sysconfig.get_python_inc() with the standard system wide python interpreter, I get /usr/include/python2.6 . 

If I understand the code correctly does the function std_lib_dirs_and_libs() return

```
 [ [ arguments to -l when compiling ], [ arguments to -L when compiling] ]. 
```

The current version of the code infers the name of the python shared object from the name of the include directory. This works with a system wide installation on Linux because the name of the system wide python library accidentally is also the last name of the python include directory but this is assumption does not hold in general.

http://stackoverflow.com/a/24115039/288875 suggests that the directory where the python shared object resides can be obtained with:

```
distutils.sysconfig.get_config_var("LIBDIR")
```

For the RHEL6 system wide installation, this gives /usr/lib64 and for my personal installation on RHEL6 I get /home/users/username/python/lib as expected.

Looking at the available variables in sysconfig, I see that to get the name of the library, one can do:

```
distutils.sysconfig.get_config_var("LDLIBRARY")
```

which gives 'libpython2.6.so' (for the system wide installation) and 'libpython2.7.so' for the installation in my home directory.

I would propose to use these two values as a return value from std_lib_dirs_and_libs() . I noticed that this would not work on OSX with my macports installation, so I would propose only to modify the part in the if statement executed for Linux . I can produce a pull request if you are interested.


Resolving patch:
diff --git a/theano/gof/cmodule.py b/theano/gof/cmodule.py
--- a/theano/gof/cmodule.py
+++ b/theano/gof/cmodule.py
@@ -1649,9 +1649,22 @@ def std_lib_dirs_and_libs():
     elif sys.platform == 'darwin':
         std_lib_dirs_and_libs.data = [], []
     else:
+        # assume Linux
         # Typical include directory: /usr/include/python2.6
-        libname = os.path.basename(python_inc)
-        std_lib_dirs_and_libs.data = [libname], []
+
+        # get the name of the python library (shared object)
+        libname = distutils.sysconfig.get_config_var("LDLIBRARY")
+
+        if libname.startswith("lib"):
+            libname = libname[3:]
+
+        # remove extension if present
+        if libname.endswith(".so"):
+            libname = libname[:-3]
+
+        libdir = distutils.sysconfig.get_config_var("LIBDIR")
+
+        std_lib_dirs_and_libs.data = [libname], [libdir]
 
     # sometimes, the linker cannot find -lpython so we need to tell it
     # explicitly where it is located this returns


Test patch: