Repository: easybuilders/easybuild-easyblocks

Issue:
Python easyblock doesn't add the correct include dir to $CPATH
The python easyblock only add $EBROOTPYTHON/include to CPATH, but the Python include files are in
$EROOTPYTHON/include/python%(pyver)s

This makes some codes fail in compilation.


Discussion:
@akesandgren Do you have an example of something that fails because of this?

It's a bit strange this has never popped up before...
Scipion (or rather Xmipp inside Scipion) fails to find Python.h.
Since that uses SCons it would normally use the system python and link stuff to its internal tree and thus bypass the problem.

I don't know of any other examples at the moment. But the current CPATH add from python.py just uses ConfigureMakes rule for CPATH additions.

Adding include/%(pyver)s may well break stuff though, for instance if something has a dep-chain that pulls in multiple python versions. But if it does I would say that the dependency chain, for whatever it is that gets broken, is then broken in itself.

I know that the preferred style is to either have explicit -Ipython-path/include/pythonx.y or have
#include <pythonx.y/something.h>
but I see lots of code that don't follow the rules (in general that is)
Another example where this fails would be LAMMPS with the PYTHON package enabled (even though #594 is not officially in yet):
```
../pair_python.cpp:18:20: fatal error: Python.h: No such file or directory
 #include <Python.h>
                    ^
compilation terminated.

```
@akesandgren I'm not sure we should risk breaking stuff by always adding `include/pythonX.Y` to `$CPATH`...

I'm more inclined to only do so for software where it's really needed, like `Xmipp` or `LAMMPS`, since there the fix is pretty straightforward. While if software starts failing to build because we have included it through the `Python` module, it may be a lot harder to resolve those issues?

@mstud Can you mention that in #594 explicitly?
Also note that `include/%(pyver)s` won't be enough in case of Python 3, there you also may need to add the abiflags to the version, e.g. `include/python3.6m`
@mstud Good point, that does probably mean we should provide a function in `Python.py` that other easyblocks can leverage to determine the (full) include path...
Well, I don't really think there is a risk to add include/%(pyver)s (or similar for P3) to CPATH.
Code that handles this themselves would either explicitly add -I.../include/python2.7, or have #include <python2.7/something.h> or similar.

The only real risk is include file name clashing, but that could happen anyway.

I've been running with that for a while (manually added) and have not seen any problems.
Although i haven't built that many packages with the toolchain where CPATH got include/python2.7 in it.
While experimenting with #594 (and associated easyconfigs) I found this disturbing (and wrong):
[j-perdue@tlogin-0502 (18:47) ~]$ python3-config --includes
-I/general/general/sw/eb/sw/Python/3.6.3-foss-2017b/include/python3.6m -I/general/general/sw/eb/sw/Python/3.6.3-foss-2017b/include/python3.6m
[j-perdue@tlogin-0502 (18:47) ~]$ pkg-config python3 --cflags
-I/sw/eb/sw/Python/3.6.3-foss-2017b/include/python3.6m 
the former stems from some of the symlinks we have for GPFS but /general/general is just wrong (one /general would work though... the pkg-config output is prettier)

Changed things to use pkg-config (will provide info in #594)





Resolving patch:
diff --git a/easybuild/easyblocks/p/python.py b/easybuild/easyblocks/p/python.py
--- a/easybuild/easyblocks/p/python.py
+++ b/easybuild/easyblocks/p/python.py
@@ -31,7 +31,6 @@
 @author: Pieter De Baets (Ghent University)
 @author: Jens Timmerman (Ghent University)
 """
-import copy
 import glob
 import os
 import re
@@ -42,8 +41,8 @@
 from easybuild.easyblocks.generic.configuremake import ConfigureMake
 from easybuild.framework.easyconfig import CUSTOM
 from easybuild.tools.build_log import EasyBuildError, print_warning
-from easybuild.tools.modules import get_software_libdir, get_software_libdir, get_software_root, get_software_version
-from easybuild.tools.filetools import remove_file, symlink
+from easybuild.tools.modules import get_software_libdir, get_software_root, get_software_version
+from easybuild.tools.filetools import symlink
 from easybuild.tools.run import run_cmd
 from easybuild.tools.systemtools import get_shared_lib_ext
 
@@ -74,6 +73,12 @@ def extra_options():
         }
         return ConfigureMake.extra_options(extra_vars)
 
+    def __init__(self, *args, **kwargs):
+        """Constructor for Python easyblock."""
+        super(EB_Python, self).__init__(*args, **kwargs)
+
+        self.pyshortver = '.'.join(self.version.split('.')[:2])
+
     def prepare_for_extensions(self):
         """
         Set default class and filter for Python packages
@@ -183,13 +188,11 @@ def install_step(self):
 
         python_binary_path = os.path.join(self.installdir, 'bin', 'python')
         if not os.path.isfile(python_binary_path):
-            pyver = '.'.join(self.version.split('.')[:2])
-            symlink(python_binary_path + pyver, python_binary_path)
+            symlink(python_binary_path + self.pyshortver, python_binary_path)
 
     def sanity_check_step(self):
         """Custom sanity check for Python."""
 
-        pyver = 'python' + '.'.join(self.version.split('.')[:2])
         shlib_ext = get_shared_lib_ext()
 
         try:
@@ -218,6 +221,7 @@ def sanity_check_step(self):
         else:
             self.log.info("No errors found in output of %s: %s", cmd, out)
 
+        pyver = 'python' + self.pyshortver
         custom_paths = {
             'files': [os.path.join('bin', pyver), os.path.join('lib', 'lib' + pyver + abiflags + '.' + shlib_ext)],
             'dirs': [os.path.join('include', pyver + abiflags), os.path.join('lib', pyver)],
@@ -250,3 +254,20 @@ def sanity_check_step(self):
                 raise EasyBuildError("Expected to find exactly one _tkinter*.so: %s", tkinter_so_hits)
 
         super(EB_Python, self).sanity_check_step(custom_paths=custom_paths, custom_commands=custom_commands)
+
+    def make_module_req_guess(self):
+        """
+        Return dictionary with additional entries for path-like environment variables;
+        also include <prefix>/include/python* to $CPATH.
+        """
+        guesses = super(EB_Python, self).make_module_req_guess()
+
+        pyincdirs = glob.glob(os.path.join(self.installdir, 'include', 'python' + self.pyshortver + '*'))
+        if len(pyincdirs) == 1:
+            guesses['CPATH'].append(os.path.join('include', os.path.basename(pyincdirs[0])))
+        elif pyincdirs:
+            # only fail if *multiple* hits were found
+            # under --module-only --force or --extended-dry-run, nothing may actually be installed (yet)
+            raise EasyBuildError("Failed to isolate subdirectory with Python header files: %s", pyincdirs)
+
+        return guesses


Test patch: