Repository: easybuilders/easybuild-framework

Issue:
RPath compiler wrapper breaking precompiled header builds
This issue was found when building OpenCV-4.0.1-foss-2018b-Python-3.6.6.eb with EB 4.1.1. The build has multiple errors `undefined reference to main`. This is caused by the rpath wrappers adding `-Wl,-rpath=...` options to the g++ call, which turns otherwise compile-only commands to generate precompiled headers (`-x c++-header`) into commands that link. 

See [easybuild-OpenCV-4.0.1-20200623.143513.mcEaZ.log.gz](https://github.com/easybuilders/easybuild-framework/files/4819395/easybuild-OpenCV-4.0.1-20200623.143513.mcEaZ.log.gz) for the full logs. But it can also be shown using only the generated rpath wrapper to compile a simple file:

```
paulm@tcn180 14:43 ~$ m list
Currently Loaded Modulefiles:
 1) surfsara   2) 2019   3) EasyBuild/4.1.1   4) compilerwrappers   5) eb/4.1.1(default)   6) GCCcore/8.2.0   7) zlib/1.2.11-GCCcore-8.2.0   8) binutils/2.31.1-GCCcore-8.2.0   9) GCC/8.2.0-2.31.1  

paulm@tcn180:~$ cat t.cc
#include <cstdio>

# Straight g++ call, no wrappers, succeeds
paulm@tcn180 14:45 ~$ /sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/GCCcore/8.2.0/bin/g++ -x c++-header t.cc
paulm@tcn180 14:45 ~$ 

# Wrapped call, fails
paulm@tcn180 14:45 ~$ /scratch-shared/paulm/eb-3OiLzK/tmpvR7vdm/rpath_wrappers/gxx_wrapper/g++ -x c++-header t.cc 
/lib/../lib64/crt1.o(.text+0x20): error: undefined reference to 'main'
collect2: error: ld returned 1 exit status

# Straight g++ call with -Wl,-rpath addition, no wrappers, fails
paulm@tcn180 14:46 ~$ /sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/GCCcore/8.2.0/bin/g++ -x c++-header -Wl,-rpath=/tmp/doh t.cc
/lib/../lib64/crt1.o(.text+0x20): error: undefined reference to 'main'
collect2: error: ld returned 1 exit status
```

We (me and @casparvl) suggest that the rpath wrappers check for the `-x c++-header` (and similar) options to make sure an otherwise correct call of g++ is not turned into one that fails. Note that there is a bit of a mist as to the expected behaviour of GCC with various options, as the docs aren't entirely clear:

* For example, calling `g++ t.cc` fails if `t.cc` does not contain a main function as it does linking of the generated object code. But calling `g++ -x c++-header t.cc` apparently turns on compile-only mode and succeeds (but the [gcc docs](https://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html) only say: "To create a precompiled header file, simply compile it as you would any other file, if necessary using the -x option to make the driver treat it as a C or C++ header file")
* Adding `-Wl,-rpath=...` makes GCC link (although the docs do not specify this), unless `-c` is used, in which case linking is never done.
* The [examples in the GCC docs of using precompiled headers](https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_headers.html) do not use `-c` when using `-x c++-header`. But upstream package could be more explicit by using `-c -x c++-header`, which would still work when the RPATH wrappers would add `-Wl,-rpath=...`, but it can't really be expected of those packages to handle that case of downstream building.


Discussion:
Ugh, what a mess, thanks for the detailed analysis @paulmelis!

I agree that letting the rpath wrapper check for `-x c++header` and similar options makes sense...

Resolving patch:
diff --git a/easybuild/scripts/rpath_args.py b/easybuild/scripts/rpath_args.py
--- a/easybuild/scripts/rpath_args.py
+++ b/easybuild/scripts/rpath_args.py
@@ -41,7 +41,7 @@
 rpath_include = sys.argv[3]
 args = sys.argv[4:]
 
-# wheter or not to use -Wl to pass options to the linker
+# determine whether or not to use -Wl to pass options to the linker based on name of command
 if cmd in ['ld', 'ld.gold', 'ld.bfd']:
     flag_prefix = ''
 else:
@@ -58,7 +58,7 @@
 else:
     rpath_include = []
 
-version_mode = False
+add_rpath_args = True
 cmd_args, cmd_args_rpath = [], []
 
 # process list of original command line arguments
@@ -69,7 +69,16 @@
 
     # if command is run in 'version check' mode, make sure we don't include *any* -rpath arguments
     if arg in ['-v', '-V', '--version', '-dumpversion']:
-        version_mode = True
+        add_rpath_args = False
+        cmd_args.append(arg)
+
+    # compiler options like "-x c++header" imply no linking is done (similar to -c),
+    # so then we must not inject -Wl,-rpath option since they *enable* linking;
+    # see https://github.com/easybuilders/easybuild-framework/issues/3371
+    elif arg == '-x':
+        idx_next = idx + 1
+        if idx_next < len(args) and args[idx_next] in ['c-header', 'c++-header']:
+            add_rpath_args = False
         cmd_args.append(arg)
 
     # FIXME: also consider $LIBRARY_PATH?
@@ -110,16 +119,15 @@
 
     idx += 1
 
-# add -rpath flags in front
-cmd_args = cmd_args_rpath + cmd_args
+if add_rpath_args:
+    # try to make sure that RUNPATH is not used by always injecting --disable-new-dtags
+    cmd_args_rpath.insert(0, flag_prefix + '--disable-new-dtags')
 
-cmd_args_rpath = [flag_prefix + '-rpath=%s' % inc for inc in rpath_include]
+    # add -rpath options for paths listed in rpath_include
+    cmd_args_rpath = [flag_prefix + '-rpath=%s' % inc for inc in rpath_include] + cmd_args_rpath
 
-if not version_mode:
-    cmd_args = cmd_args_rpath + [
-        # try to make sure that RUNPATH is not used by always injecting --disable-new-dtags
-        flag_prefix + '--disable-new-dtags',
-    ] + cmd_args
+    # add -rpath flags in front
+    cmd_args = cmd_args_rpath + cmd_args
 
 # wrap all arguments into single quotes to avoid further bash expansion
 cmd_args = ["'%s'" % a.replace("'", "''") for a in cmd_args]


Test patch:
diff --git a/test/framework/toolchain.py b/test/framework/toolchain.py
--- a/test/framework/toolchain.py
+++ b/test/framework/toolchain.py
@@ -1925,10 +1925,26 @@ def test_rpath_args_script(self):
         self.assertEqual(out.strip(), "CMD_ARGS=(%s)" % ' '.join(cmd_args))
 
         # verify that no -rpath arguments are injected when command is run in 'version check' mode
-        cmd = "%s g++ '' '%s' -v" % (script, rpath_inc)
-        out, ec = run_cmd(cmd, simple=False)
-        self.assertEqual(ec, 0)
-        self.assertEqual(out.strip(), "CMD_ARGS=('-v')")
+        for extra_args in ["-v", "-V", "--version", "-dumpversion", "-v -L/test/lib"]:
+            cmd = "%s g++ '' '%s' %s" % (script, rpath_inc, extra_args)
+            out, ec = run_cmd(cmd, simple=False)
+            self.assertEqual(ec, 0)
+            self.assertEqual(out.strip(), "CMD_ARGS=(%s)" % ' '.join(["'%s'" % x for x in extra_args.split(' ')]))
+
+        # if a compiler command includes "-x c++-header" or "-x c-header" (which imply no linking is done),
+        # we should *not* inject -Wl,-rpath options, since those enable linking as a side-effect;
+        # see https://github.com/easybuilders/easybuild-framework/issues/3371
+        test_cases = [
+            "-x c++-header",
+            "-x c-header",
+            "-L/test/lib -x c++-header",
+        ]
+        for extra_args in test_cases:
+            cmd = "%s g++ '' '%s' foo.c -O2 %s" % (script, rpath_inc, extra_args)
+            out, ec = run_cmd(cmd, simple=False)
+            self.assertEqual(ec, 0)
+            cmd_args = ["'foo.c'", "'-O2'"] + ["'%s'" % x for x in extra_args.split(' ')]
+            self.assertEqual(out.strip(), "CMD_ARGS=(%s)" % ' '.join(cmd_args))
 
     def test_toolchain_prepare_rpath(self):
         """Test toolchain.prepare under --rpath"""