Repository: spack/spack

Issue:
cuda: g++ finds C++ std headers shipped by cuda sdk instead of its own libstdc++
While trying to build *standard C++ sources (no device code)* with Spack's own `g++` wrapper from within a package's `build-env` that depends on `cuda`, C++ standard headers (e.g.: `vector`) vendored by the `cuda` SDK are found by `g++` before its own `libstdc++`, breaking any trivial compilation that involves C++ standard headers. 

### Steps to reproduce the issue

Any package depending on `cuda` fits the purpose, I've been able to reproduce the issue with a bogus `package.py` like this:

```python
from spack import *

class Nvtest(Package):
    version('0')
    depends_on('cuda')
```

The `test.cpp` source I'm trying to compile uses just one standard header and is as simple as possible:

```cpp
#include <vector>

int main() {
    std::vector<int> v;
}
```

Actual steps:

```console
$ spack install gcc@8.3.0
$ spack compiler add $(spack location -i gcc@8.3.0)
$ spack install cuda%gcc@8.3.0
$ spack build-env nvtest%gcc@8.3.0 -- g++ test.cpp
In file included from /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4e
o6j6y/include/cuda/std/detail/libcxx/include/stdlib.h:93,
                 from /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4e
o6j6y/include/cuda/std/detail/libcxx/include/cstdlib:86,
                 from /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4e
o6j6y/include/cuda/std/detail/libcxx/include/exception:81,
                 from /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4e
o6j6y/include/cuda/std/detail/libcxx/include/typeinfo:60,
                 from /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4e
o6j6y/include/cuda/std/detail/libcxx/include/memory:653,
                 from /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4e
o6j6y/include/cuda/std/detail/libcxx/include/algorithm:643,
                 from /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4e
o6j6y/include/cuda/std/detail/libcxx/include/__bit_reference:15,
                 from /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4e
o6j6y/include/cuda/std/detail/libcxx/include/vector:274,
                 from main.cpp:1:
/spack/opt/spack/linux-centos7-power8le/gcc-4.8.5/gcc-8.3.0-kvs4zwy3ojuvmc43h4yf364ie2zy5gtk/include/c++/8.3.0/s
tdlib.h:38:12: error: 'std::abort' has not been declared
 using std::abort;
# lots of similar errors omitted
```

The same behaviour is observed for both `gcc@9.2.0` (**not supported** by `cuda@10.2.89`) and system-shipped `gcc@4.8.5` (**supported** by `cuda@10.2.89`).

I think that the issue stems from this:

```console
$ spack build-env nvtest%gcc@8.3.0 | grep SPACK_INCLUDE_DIRS
SPACK_INCLUDE_DIRS=/spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4eo6j6y/include:/spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4eo6j6y/include/cuda/std/detail/libcxx/include
```

The frontend confirms the inclusion search order:

```console
$ spack build-env nvtest%gcc@8.3.0 -- g++ -E -v -
Reading specs from /spack/opt/spack/linux-centos7-power8le/gcc-4.8.5/gcc-8.3.0-kvs4zwy3ojuvmc43h4yf364ie2zy5gtk/lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/specs
#include "..." search starts here:
#include <...> search starts here:
 /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4eo6j6y/include
 /spack/opt/spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4eo6j6y/include/cuda/std/detail/libcxx/include
 /spack/opt/spack/linux-centos7-power8le/gcc-4.8.5/gcc-8.3.0-kvs4zwy3ojuvmc43h4yf364ie2zy5gtk/lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/include
 /usr/local/include
 /spack/opt/spack/linux-centos7-power8le/gcc-4.8.5/gcc-8.3.0-kvs4zwy3ojuvmc43h4yf364ie2zy5gtk/include
 /spack/opt/spack/linux-centos7-power8le/gcc-4.8.5/gcc-8.3.0-kvs4zwy3ojuvmc43h4yf364ie2zy5gtk/lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/include-fixed
 /usr/include
End of search list.
```

While `<cuda>/include` provides SDK's own headers, `<cuda>/include/cuda/std/detail/libcxx/include` pulls in standard headers provided by a vendored implementation of the C++ standard library. The fact that a path component is called `/detail/` makes me think that it should be treated as an implementation detail (as is usually done in C++ code bases) and not explicitly injected in the inclusion search space.


Discussion:
In the meantime, `git bisect` pointed me to 9c10b80591fec2cf391c97f8f53040c3429dbdec (merged in #13819) that looks like the culprit.
Thank you for reporting. CUDA 10.2 (#13819) introduced [libcu++](https://gist.github.com/ax3l/9489132#device-side-c-standard-support) which tries to provide more and more stdlib functionality for the device over the next releases. ~~Some of this functionality is intentionally pre-injected before host-compiler includes, afaik, but~~ (*update: they are intentionally [not interfering](https://on-demand.gputechconf.com/supercomputing/2019/video/sc1942-the-cuda-c++-standard-library/).*) the CUDA-specific things are properly living in `include/cuda/std` which is the public entry point for facade headers.

As you already found correctly, the implementation details of `include/cuda/std/detail/libcxx/include` should definitely *not* be part of the public include path in `SPACK_INCLUDE_DIRS`. Only `<cuda-prefix>/include/` should be there.

I do wonder why Spack picks them up, it did not in the past. Maybe it stems from the `find_libraries(... recursive=True)` call (PR #11449). @scheibelp do you know more about this?
@ax3l thanks for looking into this. I tried to check what [this call](https://github.com/spack/spack/blob/ba87a7f53a4bc57446295833cb0f71e978ccb9fc/var/spack/repos/builtin/packages/cuda/package.py#L108-L109) scrapes out of `cuda` prefix and it looks like it is interested only in libraries:

```python
def libs(self):
        libs = find_libraries('libcuda', root=self.prefix, shared=True,
                              recursive=True)
        # libs contains:
        # /spack/linux-centos7-power8le/gcc-8.3.0/cuda-10.2.89-zamew7asat3psayj22opj5xnm4eo6j6y/targets/ppc64le-linux/lib/stubs/libcuda.so
```

It looks like there is some other component that is doing a similar thing but for headers... No clues about its existence from `cuda/package.py` tho :confused:
Ok, let me quick-fix breaking builds by keeping 10.1 the default version for now, until we find a fix.
Hm, for some reason the `pkg.spec["cuda@10.2.89%gcc@8.3.0 arch=linux-ubuntu18.04-skylake"].headers.directories` in `lib/spack/spack/build_environment.py` during `set_build_environment_variables` includes these includes...

`lib/spack/spack/spec.py` builds up this header list quite optimistically ;) in `_headers_default_handler` by finding all `headers = find_headers('*', root=spec.prefix.include, recursive=True)`
This is already including a lot of detail headers in the public headers list, e.g. from thrust and other cuda-boundled C++ libraries, and also adds the unnecessary include you see now.

We can potentially overwrite the
```py
@property
    def headers(self):
```
method of the cuda package, but I think this behavior in Spack's `build_environment.py`/`spec.py` is generally dangerous. (C/C++ facade headers et al. and detail includes are very common.)

As another quick-fix/work-around we could adjust the regex `include_regex = re.compile(r'(.*)(\binclude\b)(.*)')` in `lib/spack/llnl/util/filesystem.py` class `HeaderList` to only math the first occuring `include/` in a path. This would exclude recurring ones like the one picked up in your case `<prefix>/include/some/path/detail/include/`.
Fix: `include_regex = re.compile(r'(((?!(\binclude\b)).)*)(\binclude\b)(.*)')` #13991
> I think this behavior in Spack's `build_environment.py`/`spec.py` is generally dangerous. (C/C++ facade headers et al. and detail includes are very common.)

This is scary. This means that any random `config.h` or similar stuff arbitrarily deep in *private* include directories is going to fight for the same `#include "config.h"` directive (unless the inclusions are prefixed with some unique path component) :cry: 

@ax3l thanks for thinking about viable fixes, in my (poor) understanding of Spack internals the current situation is a recipe for breaking random builds.

> As another quick-fix/work-around we could adjust the regex include_regex = re.compile(r'(.*)(\binclude\b)(.*)') in lib/spack/llnl/util/filesystem.py class HeaderList to only math the first occuring include/ in a path.

This sounds good to me as it looks really conservative, it will probably break builds that right now are working fine only *by chance*.
That's just the quick-fix, we consider just adding `<prefix>/include/` as a breaking change and letting all packages that use non-unix default paths add more includes manually via the package's `headers` property.

Resolving patch:
diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py
--- a/lib/spack/llnl/util/filesystem.py
+++ b/lib/spack/llnl/util/filesystem.py
@@ -1156,7 +1156,9 @@ class HeaderList(FileList):
 
     # Make sure to only match complete words, otherwise path components such
     # as "xinclude" will cause false matches.
-    include_regex = re.compile(r'(.*)(\binclude\b)(.*)')
+    # Avoid matching paths such as <prefix>/include/something/detail/include,
+    # e.g. in the CUDA Toolkit which ships internal libc++ headers.
+    include_regex = re.compile(r'(.*?)(\binclude\b)(.*)')
 
     def __init__(self, files):
         super(HeaderList, self).__init__(files)
diff --git a/var/spack/repos/builtin/packages/cuda/package.py b/var/spack/repos/builtin/packages/cuda/package.py
--- a/var/spack/repos/builtin/packages/cuda/package.py
+++ b/var/spack/repos/builtin/packages/cuda/package.py
@@ -60,8 +60,7 @@ class Cuda(Package):
         key = "{0}-{1}".format(platform.system(), platform.machine())
         pkg = packages.get(key)
         if pkg:
-            version(ver, sha256=pkg[0], url=pkg[1], expand=False,
-                    preferred=(ver == "10.1.243"))  # see GH issue 13969
+            version(ver, sha256=pkg[0], url=pkg[1], expand=False)
 
     # macOS Mojave drops NVIDIA graphics card support -- official NVIDIA
     # drivers do not exist for Mojave. See


Test patch:
diff --git a/lib/spack/spack/test/build_environment.py b/lib/spack/spack/test/build_environment.py
--- a/lib/spack/spack/test/build_environment.py
+++ b/lib/spack/spack/test/build_environment.py
@@ -17,7 +17,7 @@
 from spack.util.spack_yaml import syaml_dict, syaml_str
 from spack.util.environment import EnvironmentModifications
 
-from llnl.util.filesystem import LibraryList
+from llnl.util.filesystem import LibraryList, HeaderList
 
 
 @pytest.fixture
@@ -243,6 +243,18 @@ def test_set_build_environment_variables(
     variables.
     """
 
+    # https://github.com/spack/spack/issues/13969
+    cuda_headers = HeaderList([
+        'prefix/include/cuda_runtime.h',
+        'prefix/include/cuda/atomic',
+        'prefix/include/cuda/std/detail/libcxx/include/ctype.h'])
+    cuda_include_dirs = cuda_headers.directories
+    assert(os.path.join('prefix', 'include')
+           in cuda_include_dirs)
+    assert(os.path.join('prefix', 'include', 'cuda', 'std', 'detail',
+                        'libcxx', 'include')
+           not in cuda_include_dirs)
+
     root = spack.spec.Spec('dt-diamond')
     root.concretize()