Repository: spack/spack

Issue:
clang wrapper behaves differently than clang on OSX
This is on MacOS high sierra

balay@ipro^~/spack(xsdk-0.3.0) $ sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.13
BuildVersion:	17A405


I see spack is using the wrapper lib/spack/env/clang/clang instead of /usr/bin/clang

Perhaps there is a valid reason for this. But I have 2 issues with it:

1. It behaves differently than /usr/bin/clang. Why?

```
     17    checking for gcc... /Users/balay/spack/lib/spack/env/clang/clang
     18    checking whether the C compiler works... yes
     19    checking for C compiler default output file name... a.out
     20    checking for suffix of executables...
  >> 21    checking whether we are cross compiling... configure: error: in `/private/var/folders/b2/4wzkzbgs3d91kqp1dw47wq940000gn/T/balay/spack-stage/spack-stage-4h3G7M/pkg-config-0.29.2':
  >> 22    configure: error: cannot run C compiled programs.

```

config.log for this build has:
```
conftest.c:11:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
```

However /usr/bin/clang works.

```
balay@ipro^~/spack(xsdk-0.3.0) $ cat conftest.c 
#include <stdio.h>
int main(void)
{return 0;}
balay@ipro^~/spack(xsdk-0.3.0) $ clang conftest.c 
balay@ipro^~/spack(xsdk-0.3.0) $ 
```

2. I cannot run this wrapper outside spack to reproduce issues [by manual build of packages or trying to compile test code]

```
balay@ipro^~/spack(xsdk-0.3.0) $ /Users/balay/spack/lib/spack/env/clang/clang conftest.c 
Spack compiler must be run from Spack! Input 'SPACK_PREFIX' is missing.
balay@ipro^~/spack(xsdk-0.3.0) $ export SPACK_PREFIX=$PWD
balay@ipro^~/spack(xsdk-0.3.0) $ /Users/balay/spack/lib/spack/env/clang/clang conftest.c 
Spack compiler must be run from Spack! Input 'SPACK_ENV_PATH' is missing.
balay@ipro^~/spack(xsdk-0.3.0) $ export SPACK_ENV_PATH=$PWD
balay@ipro^~/spack(xsdk-0.3.0) $ /Users/balay/spack/lib/spack/env/clang/clang conftest.c 
Spack compiler must be run from Spack! Input 'SPACK_DEBUG_LOG_DIR' is missing.
balay@ipro^~/spack(xsdk-0.3.0) $ export SPACK_DEBUG_LOG_DIR=$PWD
balay@ipro^~/spack(xsdk-0.3.0) $ /Users/balay/spack/lib/spack/env/clang/clang conftest.c 
Spack compiler must be run from Spack! Input 'SPACK_COMPILER_SPEC' is missing.
balay@ipro^~/spack(xsdk-0.3.0) $ 
```
and so on... How do I invoke this wrapper for manual verification?

 
I see Spack doc says 'xcode-select --install' is required. I guess this will resolve this error.
http://spack.readthedocs.io/en/latest/getting_started.html?highlight=xcode-select

But I'm able to install various packages without it. So Its not clear to me why this wrapper has more requirements than /usr/bin/clang. Previously I've seen this requirement for external compilers - like pgi - or clang installed by brew - but in this case - I'm using clang from XCode.

Note: the following works [when it fails via spack]:
```
balay@ipro^~/spack/var/spack/stage/pkg-config-0.29.2-zggrhmtc22dolyoug5ylkrknom2e3vda/pkg-config-0.29.2 $ CC=/usr/bin/clang ./configure --enable-shared --with-internal-glib

```




Discussion:
@balay:

Please see https://spack.readthedocs.io/en/latest/packaging_guide.html#compiler-wrappers.  The wrappers are *very* thin and shouldn't be affecting the behavior of the underlying compiler, but they're essential for the way *all* Spack builds work.

For debugging, see https://spack.readthedocs.io/en/latest/packaging_guide.html#interactive-shell-support.

If you want to see why the wrapper is failing, try:

```
$ spack stage <spec>
$ spack cd <spec>
$ spack env <spec> bash
```

That will put you in a shell with the build environment set up properly, and you should be able to run the wrapper.



Thanks! 'spack env <spec> bash is useful.

Here is the difference in behavior between spack clang and /usr/bin/clang

```
balay@ipro^~/spack(xsdk-0.3.0) $ /Users/balay/spack/lib/spack/env/clang/clang -v conftest.c 
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin17.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.13.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name conftest.c -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fno-strict-return -masm-verbose -munwind-tables -target-cpu penryn -target-linker-version 302.3 -v -dwarf-column-info -debugger-tuning=lldb -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0 -fdebug-compilation-dir /Users/balay/spack -ferror-limit 19 -fmessage-length 211 -stack-protector 1 -fblocks -fobjc-runtime=macosx-10.13.0 -fencode-extended-block-signature -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o /var/folders/b2/4wzkzbgs3d91kqp1dw47wq940000gn/T/conftest-e0ba81.o -x c conftest.c
clang -cc1 version 9.0.0 (clang-900.0.38) default target x86_64-apple-darwin17.0.0
ignoring nonexistent directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
conftest.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
balay@ipro^~/spack(xsdk-0.3.0) $ /usr/bin/clang -v conftest.c 
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin17.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.13.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name conftest.c -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fno-strict-return -masm-verbose -munwind-tables -target-cpu penryn -target-linker-version 302.3 -v -dwarf-column-info -debugger-tuning=lldb -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -I/usr/local/include -fdebug-compilation-dir /Users/balay/spack -ferror-limit 19 -fmessage-length 211 -stack-protector 1 -fblocks -fobjc-runtime=macosx-10.13.0 -fencode-extended-block-signature -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o /var/folders/b2/4wzkzbgs3d91kqp1dw47wq940000gn/T/conftest-7ccbaa.o -x c conftest.c
clang -cc1 version 9.0.0 (clang-900.0.38) default target x86_64-apple-darwin17.0.0
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/local/include"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks (framework directory)
End of search list.
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch x86_64 -macosx_version_min 10.13.0 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -o a.out /var/folders/b2/4wzkzbgs3d91kqp1dw47wq940000gn/T/conftest-7ccbaa.o -L/usr/local/lib -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0/lib/darwin/libclang_rt.osx.a
balay@ipro^~/spack(xsdk-0.3.0) $ 

```

Looks like spack clang wrapper is setting up some env variable that is preventing /usr/bin/clang from using the following options. /usr/bin/clang is using them by default.

```
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -I/usr/local/include
```
BTW: here is a potential problem with spack using wrappers that dont' work  standalone.

Packages like petsc create makefiles [and pkg-config files] for users to use later. So when these get used later by users - they will get broken build tools.

This problem does not come up currently as for MPI builds - we use mpicc - and not gcc directly [and spack appears to not wrap mpicc - but wrap gcc used by mpicc]. So when users use mpicc - the system gcc is getting used - so this issue doesn't bite..

@BarrySmith 
BTW: Just to confirm the issue:  the following works with the wrapper:

```
balay@ipro^~/spack(xsdk-0.3.0) $ /Users/balay/spack/lib/spack/env/clang/clang conftest.c  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -I/usr/local/include
balay@ipro^~/spack(xsdk-0.3.0) $ 


```
Ah - spack is using /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang - and not /usr/bin/clang

These appear to behave differently.

```
balay@ipro^~/spack(xsdk-0.3.0) $ /usr/bin/clang conftest.c 
balay@ipro^~/spack(xsdk-0.3.0) $ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang conftest.c 
conftest.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
```

How do I tell spack to use /usr/bin/clang?

I tried modifying ~/.spack/darwin/compilers.yaml - but spack is still picking up the wrong one.

```
balay@ipro^~/spack(xsdk-0.3.0) $ cat ~/.spack/darwin/compilers.yaml
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: highsierra
    paths:
      cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77: /usr/local/bin/gfortran
      fc: /usr/local/bin/gfortran
    spec: clang@9.0.0-apple
    target: x86_64
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: highsierra
    paths:
      cc: /usr/local/bin/gcc-7
      cxx: /usr/local/bin/g++-7
      f77: /usr/local/bin/gfortran-7
      fc: /usr/local/bin/gfortran-7
    spec: gcc@7.2.0
    target: x86_64
balay@ipro^~/spack(xsdk-0.3.0) $ sh -x /Users/balay/spack/lib/spack/env/clang/clang -c conftest.c  2>&1 | grep clang | grep conftest
+ exec /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -c conftest.c
```

```
balay@ipro^~/spack(xsdk-0.3.0) $ echo $SPACK_CC

balay@ipro^~/spack(xsdk-0.3.0) $ spack env pkg-config bash
balay@ipro^~/spack(xsdk-0.3.0) $ echo $SPACK_CC
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

```

```
Maybe @mathstuf or @davydden can comment on this.

We use the XCode clang because some projects (Qt) need it and won't build without it.  I haven't seen a case where they behaved differently -- but I am not an expert on this.  I'm curious whether we need to handle this differently, e.g. have packages say `needs_xcode = True` or have things optionally depend on `xcode` on a mac, and do the special stuff through an `xcode` Spack package.
the two appear to be exactly the same for me:
```
$ /usr/bin/clang --version
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin17.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang --version
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin17.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
```

@tgamblin `qt` already has `use_xcode = True`, which will use `clang` from `xcode` directly (without wrappers and alike) with some extra steps that @mathstuf knows better.
@tgamblin

/usr/bin/clang is the default xcode compiler that is available in user's path.

I have not seen any user accessing and using /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang directly

For some reason spack is deciding to use /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang eventhough I explicitly list /usr/bin/clang in ~/.spack/darwin/compilers.xml

On Tue, 31 Oct 2017, Todd Gamblin wrote:

> Maybe @mathstuf or @davydden can comment on this.
> 
> We use the XCode clang because some projects (Qt) need it and won't build without it.  I haven't seen a case where they behaved differently -- but I am not an expert on this.  I'm curious whether we need to handle this differently, e.g. have packages say `needs_xcode = True` or have things optionally depend on `xcode` on a mac, and do the special stuff through an `xcode` Spack package.
> 
> 


BTW: Both `/usr/bin/clang` and  `/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang` are from xcode - and for some reason they are behaving differently [I don't know why]

`/usr/bin/clang` is internally using the option `-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk ` so its finding `stdio.h` etc files from xcode locations.

`Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang` is not using this option - and is requiring `/usr/include/stdio.h` - which is necessitating `xcode-select --install`

And for some reason spack is preferring  `/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang` over `/usr/bin/clang`. Even if I change in `~/spack/darwin/compilers.py` - its not using this changed value
The path to the compiler is replaced [here](https://github.com/LLNL/spack/blob/7dd79094b05172cfcd2b594326eb22bf8e3c83cd/lib/spack/spack/compilers/clang.py#L161-L169). How does `clang` determine whether it is running via `/usr/bin/clang` or via the Xcode path? What adds those flags?

What version of Xcode is this? It may be new behavior (I see you're using 10.13).
So should we just always provide `-isysroot` on Mac OS X?
For an Xcode `clang`, yes. I don't know the best thing to do for a FOSS `clang` build.
@davydden, @mathstuf:
> qt already has use_xcode = True, which will use clang from xcode directly (without wrappers and alike) with some extra steps that @mathstuf knows better.

Is Qt the only package that needs to run the version of clang installed by `xcode-select`?  If that is the case, could we just have the XCode-specific stuff handled at the package level, and not bother having the clang compiler know about XCode?
balay@ipro^~/spack(xsdk-0.3.0) $ xcodebuild -version
Xcode 9.0.1
Build version 9A1004

Ok - checking  `Xcode 8.2.1` on `10.11.6` - both - `/usr/bin/clang` and  `xcrun -f clang` are behaving the same way - so perhaps its xcode-9 regression/bug?

Note: - on this second machine - `xcode-select --install` is done - so I don't know if that makes a different.

BTW: Shouldn't that 'detection' happen only when doing 'spack compiler find' - and not when user wants to use it?

```
diff --git a/lib/spack/spack/compilers/clang.py b/lib/spack/spack/compilers/clang.py
index 5a88897b6..fb0ff1965 100644
--- a/lib/spack/spack/compilers/clang.py
+++ b/lib/spack/spack/compilers/clang.py
@@ -160,13 +160,7 @@ class Clang(Compiler):
 
     def _find_full_path(self, path):
         basename = os.path.basename(path)
-
-        if not self.is_apple or basename not in ('clang', 'clang++'):
-            return super(Clang, self)._find_full_path(path)
-
-        xcrun = Executable('xcrun')
-        full_path = xcrun('-f', basename, output=str)
-        return full_path.strip()
+        return super(Clang, self)._find_full_path(path)
 
     @classmethod
     def fc_version(cls, fc):
```

with this hack - `spack install pkg-config` completes without errors. Now to check xsdk...

Qt is the only package so far. I wouldn't be surprised if others show up over time.
> Ok - checking Xcode 8.2.1 on 10.11.6 - both - /usr/bin/clang and xcrun -f clang are behaving the same way - so perhaps its xcode-9 regression/bug?

It could also be an intended behavior change we just need to cope with.
> Qt is the only package so far. I wouldn't be surprised if others show up over time.

If Qt is the only one, and if it already handles things in the package, why do we need to modify the compiler path used by the clang wrapper?  Can't the Qt package just do that?

The Spack wrappers provide rpath and include path support. How do we tell Qt to use Spack's `zlib`? All Qt's configure does is take a `-system-zlib` option. IIRC, there's no way to provide "use *this* zlib" option other than sticking it in the compiler's default paths. In order to make Spack's wrapper work as Xcode (which Qt requires), an entire Xcode mockup needs to be provided. That kind of logic doesn't belong in a package, IMO.
@mathstuf: Agreed -- I think the wrappers are needed for that.

I think the compiler class should do something slightly different, though.  Currently, `clang.py` hacks around the path thing by overriding `_find_full_path`.  I think it should instead run `xcrun` in `setup_custom_environment()`, then set `SPACK_CC` and `SPACK_CXX` based on the result.  This will only change the compiler path for packages that set `use_xcode = True`.

Thoughts?
The `xcrun` dereference could also be done inside of the `use_xcode` block to replace the paths. Not sure if that messes with other bits though.

Resolving patch:
diff --git a/lib/spack/spack/compiler.py b/lib/spack/spack/compiler.py
--- a/lib/spack/spack/compiler.py
+++ b/lib/spack/spack/compiler.py
@@ -125,7 +125,6 @@ def __init__(self, cspec, operating_system, target,
         def check(exe):
             if exe is None:
                 return None
-            exe = self._find_full_path(exe)
             _verify_executables(exe)
             return exe
 
@@ -286,17 +285,6 @@ def check(key):
         successful.reverse()
         return dict(((v, p, s), path) for v, p, s, path in successful)
 
-    def _find_full_path(self, path):
-        """Return the actual path for a tool.
-
-        Some toolchains use forwarding executables (particularly Xcode-based
-        toolchains) which can be manipulated by external environment variables.
-        This method should be used to extract the actual path used for a tool
-        by finding out the end executable the forwarding executables end up
-        running.
-        """
-        return path
-
     def setup_custom_environment(self, pkg, env):
         """Set any environment variables necessary to use the compiler."""
         pass
diff --git a/lib/spack/spack/compilers/clang.py b/lib/spack/spack/compilers/clang.py
--- a/lib/spack/spack/compilers/clang.py
+++ b/lib/spack/spack/compilers/clang.py
@@ -158,16 +158,6 @@ def default_version(cls, comp):
 
         return _version_cache[comp]
 
-    def _find_full_path(self, path):
-        basename = os.path.basename(path)
-
-        if not self.is_apple or basename not in ('clang', 'clang++'):
-            return super(Clang, self)._find_full_path(path)
-
-        xcrun = Executable('xcrun')
-        full_path = xcrun('-f', basename, output=str)
-        return full_path.strip()
-
     @classmethod
     def fc_version(cls, fc):
         # We could map from gcc/gfortran version to clang version, but on macOS
@@ -202,6 +192,14 @@ def setup_custom_environment(self, pkg, env):
             # consequently render MPI non-functional outside of Spack.
             return
 
+        # Use special XCode versions of compiler wrappers when using XCode
+        # Overwrites build_environment's setting of SPACK_CC and SPACK_CXX
+        xcrun = Executable('xcrun')
+        xcode_clang = xcrun('-f', 'clang', output=str).strip()
+        xcode_clangpp = xcrun('-f', 'clang++', output=str).strip()
+        env.set('SPACK_CC', xcode_clang, force=True)
+        env.set('SPACK_CXX', xcode_clangpp, force=True)
+
         xcode_select = Executable('xcode-select')
 
         # Get the path of the active developer directory
diff --git a/lib/spack/spack/environment.py b/lib/spack/spack/environment.py
--- a/lib/spack/spack/environment.py
+++ b/lib/spack/spack/environment.py
@@ -460,7 +460,9 @@ def set_or_unset_not_first(variable, changes, errstream):
     modifications have already been requested.
     """
     indexes = [ii for ii, item in enumerate(changes)
-               if ii != 0 and type(item) in [SetEnv, UnsetEnv]]
+               if ii != 0 and
+               not item.args.get('force', False) and
+               type(item) in [SetEnv, UnsetEnv]]
     if indexes:
         good = '\t    \t{context} at {filename}:{lineno}'
         nogood = '\t--->\t{context} at {filename}:{lineno}'


Test patch: