Repository: mesonbuild/meson

Issue:
-Wl,-rpath-link for secondary dependencies causes linking failure when there are two prefixes
This is a regression of Meson 0.52.0. https://github.com/mesonbuild/meson/pull/5647 breaks it and reverting 7b9c348102792030859ed0001a51416506a0a092 fixes the problem. Tag the author @SoapGentoo and reviewers @jpakkane, @dcbaker and @textshell .

It is typical for a development environment to have at least two prefixes, one for the development work under the home directory and one for libraries installed on the system. We have to ensure that libraries under the development prefix is always preferred to libraries under the system prefix. Otherwise, the linker may report undefined reference errors because the version provided by the system may be too old.

The way https://github.com/mesonbuild/meson/pull/5647 adds `-Wl,-rpath-link` breaks the development environment managed by JHBuild seriously because everything linking to GLib is likely to trigger undefined reference errors with GNU ld. LLVM lld isn't affected because it just ignores `-Wl,-rpath-link`. There are two problems in the current implementation of `-Wl,-rpath-link` in `PkgConfigDependency`:

1. It only adds `-Wl,-rpath-link` for `secondary_libpaths`. It is possible for a secondary dependency to come from `prefix_libpaths`, so it should add `-Wl,-rpath-link` for `prefix_libpaths` as well. `prefix_libpaths` should come before `secondary_libpaths` because it should be preferred.
2. It appears on the command line too early. In file `mesonbuild/backend/ninjabackend.py`, class `NinjaBackend`, function `generate_link`, `linker.get_dependency_link_args` is called before `linker.build_rpath_args`. This means `-Wl,-rpath-link` arguments for external dependencies returned by `linker.get_dependency_link_args` come before `-Wl,-rpath-link` arguments for internal dependencies returned by `linker.build_rpath_args`, which is wrong because we should always prefer internal libraries to external libraries.

Let's see an example here. I was trying to build `pixman` with JHBuild.

 * The development prefix `/home/lantw44/gnome/devinstall/lib` has GLib, GTK, Pango, FriBidi.
 * The system prefix `/usr/local/lib` has PCRE, libintl, libiconv.

Here are the error messages:
```
FAILED: demos/gradient-test 
clang -B/home/lantw44/.local/bin  -o demos/gradient-test 'demos/03bf7ed@@gradient-test@exe/gradient-test.c.o' -L/home/lantw44/gnome/devinstall/lib -L/usr/local/lib -Wl,--as-needed -Wl,--no-undefined -march=corei7 -B/home/lantw44/.local/bin -pipe -g3 -Og -gz -fdebug-macro -B/home/lantw44/.local/bin -Wl,--compress-debug-sections=zlib -fuse-ld=bfd -Wl,--start-group demos/libdemo.a test/libtestutils.a pixman/libpixman-1.so.0.38.5 -Wl,-rpath-link,/usr/local/lib /home/lantw44/gnome/devinstall/lib/libglib-2.0.so -L/home/lantw44/gnome/devinstall/lib -lintl /home/lantw44/gnome/devinstall/lib/libgtk-x11-2.0.so /home/lantw44/gnome/devinstall/lib/libgdk-x11-2.0.so /home/lantw44/gnome/devinstall/lib/libpangocairo-1.0.so /home/lantw44/gnome/devinstall/lib/libatk-1.0.so /home/lantw44/gnome/devinstall/lib/libcairo.so /home/lantw44/gnome/devinstall/lib/libgdk_pixbuf-2.0.so /home/lantw44/gnome/devinstall/lib/libgio-2.0.so /home/lantw44/gnome/devinstall/lib/libpangoft2-1.0.so /home/lantw44/gnome/devinstall/lib/libpango-1.0.so /home/lantw44/gnome/devinstall/lib/libgobject-2.0.so /usr/local/lib/libintl.so /home/lantw44/gnome/devinstall/lib/libharfbuzz.so /usr/local/lib/libfontconfig.so /usr/local/lib/libfreetype.so -fopenmp -Wl,-rpath-link,/usr/local/lib -L/home/lantw44/gnome/devinstall/lib -lintl /usr/local/lib/libpng16.so /usr/lib/libz.so -lm -Wl,--end-group -fopenmp -fopenmp '-Wl,-rpath,$ORIGIN/:$ORIGIN/../test:$ORIGIN/../pixman' -Wl,-rpath-link,/home/lantw44/gnome/build/pixman/demos -Wl,-rpath-link,/home/lantw44/gnome/build/pixman/test -Wl,-rpath-link,/home/lantw44/gnome/build/pixman/pixman
/usr/local/bin/ld.bfd: /home/lantw44/gnome/devinstall/lib/libpango-1.0.so: undefined reference to `fribidi_get_par_embedding_levels_ex'
/usr/local/bin/ld.bfd: /home/lantw44/gnome/devinstall/lib/libpango-1.0.so: undefined reference to `fribidi_get_bracket'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```

Function `_set_libs` of class `PkgConfigDependency` calls `pkg-config` three times, and here are results under the JHBuild environment.

This is `out`.
```
$ PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 pkg-config --libs glib-2.0
-L/home/lantw44/gnome/devinstall/lib -lglib-2.0 -lintl
```

This is `out_raw`.
```
$ pkg-config --libs glib-2.0 
-L/home/lantw44/gnome/devinstall/lib -lglib-2.0 -lintl 
```

This is `out_all`.
```
$ pkg-config --libs glib-2.0 --static
-L/home/lantw44/gnome/devinstall/lib -lglib-2.0 -lintl -pthread -liconv -L/usr/local/lib -lpcre -D_THREAD_SAFE -pthread
```

We can see that `-L/usr/local/lib` only appers in `out_all`, so we have:
 * `prefix_libpaths = ['/home/lantw44/gnome/devinstall/lib']`
 * `secondary_libpaths = ['/usr/local/lib']`
 * `system_libpaths = []`

They get translated to command line arguments as `-Wl,-rpath-link,/usr/local/lib -L/home/lantw44/gnome/devinstall/lib`.

Assume that there is an executables which need both GLib and GTK. Meson processes GLib first and put `-Wl,-rpath-link,/usr/local/lib -L/home/lantw44/gnome/devinstall/lib` on the command line. Therefore, when the linker searches for direct dependencies, it finds `/home/lantw44/gnome/devinstall/lib` first. However, when the linker searches for secondary dependencies, it finds `/usr/local/lib` first. This means that the system prefix is now even more important than the project itself because it comes before `-Wl,-rpath-link,/home/lantw44/gnome/build/pixman/demos -Wl,-rpath-link,/home/lantw44/gnome/build/pixman/test -Wl,-rpath-link,/home/lantw44/gnome/build/pixman/pixman` added by `linker.build_rpath_args`.

This is obviously wrong. The result is that the linker finds FriBidi from the wrong directory and report undefined reference errors. GTK needs Pango, and Pango needs FriBidi. The linker finds both GTK and Pango correctly because they are direct dependencies, but it finds FriBidi wrongly because it is a secondary dependency. It should find the one installed in `/home/lantw44/gnome/devinstall/lib`, not `/usr/local/lib`.

I think the solution is to
1. Add `-Wl,-rpath-link` for both `prefix_libpaths` and `secondary_libpaths`.
2. Separate `-Wl,-rpath-link` from other arguments, so they can be added in the correct order.


Discussion:
Doing points 1. seems easy, how do you want to accomplish point 2.?
I just thought of a case where point 1 might be insufficient. Assume that an executable needs two libraries. One is installed in `/usr/local/lib`, and the other one is installed in `/home/lantw44/gnome/devinstall/lib`. If the first one puts `-Wl,-rpath-link,/usr/local/lib` and the second one puts `-Wl,-rpath-link,/home/lantw44/gnome/devinstall/lib`, it may cause problems for secondary dependencies of the second library when `-Wl,-rpath-link,/usr/local/lib` appears first. Does meson know how to sort dependencies with `PKG_CONFIG_PATH`?

In a JHBuild environment, JHBuild controls the search order of direct dependencies with `LDFLAGS=-L/home/lantw44/gnome/devinstall/lib -L/usr/local/lib` and the search order of secondary dependencies with `LD_LIBRARY_PATH=/home/lantw44/gnome/devinstall/lib:/usr/local/lib`, so there was no `-Wl,-rpath-link` pointing to the outside of the build tree. Now it stops working because `-Wl,-rpath-link` added by meson overrides `LD_LIBRARY_PATH`.

For point 2, I think the least risky way is to tell `generate_link` to filter out `-Wl,-rpath-link` and move them to the bottom, but it will make the code of `generate_link` look ugly. Another possible way is to make `get_dependency_link_args` return a `(normal flags, rpath link flags)` tuple, so `generate_link` can decide how to use them.

I wonder in what situation rpath-link needs to be in an order that is different to normal runtime rpath. I.e. can we take some hints from rpath construction?

Sadly rpath-link is a mess. i still have the nagging feeling with should just `--allow-shlib-undefined` this from space to make this all go away.

You say lld does not trigger this. Can you try to find out why? I.e. does it just silently always use `--allow-shlib-undefined` semantics? Some thing different?
> I wonder in what situation rpath-link needs to be in an order that is different to normal runtime rpath. I.e. can we take some hints from rpath construction?

I think yes, they should be the same, but currently meson only handles `-rpath` for internal libraries. External libraries found by `-L` flags returned by pkg-config are not added to `-rpath` (unless pkg-config only returns one `-l` flag, which I consider it as a bug: https://github.com/mesonbuild/meson/pull/4324). Users can set `LD_LIBRARY_PATH` to find external libraries, but doing so may trigger another bug: https://github.com/mesonbuild/meson/issues/1635.

If meson is going to use `-L` flags to construct `-rpath`, beware that pkgconf doesn't sort `-L` flags. This is probably a bug. I just reported it to pkgconf todayL https://todo.sr.ht/~kaniini/pkgconf/7. I also attempted to fix the order in meson by sorting `-L` by `PKG_CONFIG_PATH`: https://github.com/mesonbuild/meson/pull/4325. But this still doesn't help the issue I mentioned in https://github.com/mesonbuild/meson/issues/6027#issuecomment-541334015, which requires meson to find the order working for all dependencies instead of just one dependency.

> Sadly rpath-link is a mess. i still have the nagging feeling with should just `--allow-shlib-undefined` this from space to make this all go away.

But doing so does not help when building an executable, right?

> You say lld does not trigger this. Can you try to find out why? I.e. does it just silently always use `--allow-shlib-undefined` semantics? Some thing different?

lld ignores `-rpath-link`. It recognizes it for compatibility with GNU ld, but it does nothing when `-rpath-link` is used. https://reviews.llvm.org/D18269
I think the key take-away from that PR is this:
> I agree with your argument that we should ignore that option. gold completely ignores that option, and the GNU ld's behavior to try to resolve undefined symbols in all shared object files at (static-)link time seems broken to me.
(from https://reviews.llvm.org/D18269#378717)

What does meson loose if we just ask ld.bfd nicely to do the same thing as lld (and possibly ld.gold)? Is the build really in the business of trying to verify that system/prefix installed libraries' dependencies actually match? Can a build even really know?
I ran `grep` on a few autotools and cmake build directories, and I didn't find any use of `-rpath-link` there. Probably they just assume users will set `LD_LIBRARY_PATH` themselves to satisfy the need of ld.bfd, but setting `LD_LIBRARY_PATH` can create problems for cmake and meson.

Resolving patch:
diff --git a/mesonbuild/dependencies/base.py b/mesonbuild/dependencies/base.py
--- a/mesonbuild/dependencies/base.py
+++ b/mesonbuild/dependencies/base.py
@@ -33,7 +33,6 @@
 from ..compilers import clib_langs
 from ..environment import BinaryTable, Environment, MachineInfo
 from ..cmake import CMakeExecutor, CMakeTraceParser, CMakeException
-from ..linkers import GnuLikeDynamicLinkerMixin
 from ..mesonlib import MachineChoice, MesonException, OrderedSet, PerMachine
 from ..mesonlib import Popen_safe, version_compare_many, version_compare, listify, stringlistify, extract_as_list, split_args
 from ..mesonlib import Version, LibType
@@ -748,18 +747,16 @@ def _set_cargs(self):
                                       (self.name, err))
         self.compile_args = self._convert_mingw_paths(self._split_args(out))
 
-    def _search_libs(self, out, out_raw, out_all):
+    def _search_libs(self, out, out_raw):
         '''
         @out: PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 pkg-config --libs
         @out_raw: pkg-config --libs
-        @out_all: pkg-config --libs --static
 
         We always look for the file ourselves instead of depending on the
         compiler to find it with -lfoo or foo.lib (if possible) because:
         1. We want to be able to select static or shared
         2. We need the full path of the library to calculate RPATH values
         3. De-dup of libraries is easier when we have absolute paths
-        4. We need to find the directories in which secondary dependencies reside
 
         Libraries that are provided by the toolchain or are not found by
         find_library() will be added with -L -l pairs.
@@ -793,18 +790,6 @@ def _search_libs(self, out, out_raw, out_all):
                 continue
             if arg.startswith('-L') and arg[2:] not in prefix_libpaths:
                 system_libpaths.add(arg[2:])
-        # collect all secondary library paths
-        secondary_libpaths = OrderedSet()
-        all_args = self._convert_mingw_paths(shlex.split(out_all))
-        for arg in all_args:
-            if arg.startswith('-L') and not arg.startswith(('-L-l', '-L-L')):
-                path = arg[2:]
-                if not os.path.isabs(path):
-                    # Resolve the path as a compiler in the build directory would
-                    path = os.path.join(self.env.get_build_dir(), path)
-                if path not in prefix_libpaths and path not in system_libpaths:
-                    secondary_libpaths.add(path)
-
         # Use this re-ordered path list for library resolution
         libpaths = list(prefix_libpaths) + list(system_libpaths)
         # Track -lfoo libraries to avoid duplicate work
@@ -812,12 +797,8 @@ def _search_libs(self, out, out_raw, out_all):
         # Track not-found libraries to know whether to add library paths
         libs_notfound = []
         libtype = LibType.STATIC if self.static else LibType.PREFER_SHARED
-        # Generate link arguments for this library, by
-        # first appending secondary link arguments for ld
+        # Generate link arguments for this library
         link_args = []
-        if self.clib_compiler and self.clib_compiler.linker and isinstance(self.clib_compiler.linker, GnuLikeDynamicLinkerMixin):
-            link_args = ['-Wl,-rpath-link,' + p for p in secondary_libpaths]
-
         for lib in full_args:
             if lib.startswith(('-L-l', '-L-L')):
                 # These are D language arguments, add them as-is
@@ -887,26 +868,6 @@ def _set_libs(self):
         libcmd = [self.name, '--libs']
         if self.static:
             libcmd.append('--static')
-        # We need to find *all* secondary dependencies of a library
-        #
-        # Say we have libA.so, located in /non/standard/dir1/, and
-        # libB.so, located in /non/standard/dir2/, which links to
-        # libA.so. Now when linking exeC to libB.so, the linker will
-        # walk the complete symbol tree to determine that all undefined
-        # symbols can be resolved. Because libA.so lives in a directory
-        # not known to the linker by default, you will get errors like
-        #
-        #   ld: warning: libA.so, needed by /non/standard/dir2/libB.so,
-        #       not found (try using -rpath or -rpath-link)
-        #   ld: /non/standard/dir2/libB.so: undefined reference to `foo()'
-        #
-        # To solve this, we load the -L paths of *all* dependencies, by
-        # relying on --static to provide us with a complete picture. All
-        # -L paths that are found via a --static lookup but that are not
-        # contained in the normal lookup have to originate from secondary
-        # dependencies. See also:
-        #   http://www.kaizou.org/2015/01/linux-libraries/
-        libcmd_all = [self.name, '--libs', '--static']
         # Force pkg-config to output -L fields even if they are system
         # paths so we can do manual searching with cc.find_library() later.
         env = os.environ.copy()
@@ -920,13 +881,9 @@ def _set_libs(self):
         # gnome.generate_gir + gnome.gtkdoc which need -L -l arguments.
         ret, out_raw, err_raw = self._call_pkgbin(libcmd)
         if ret != 0:
-            raise DependencyException('Could not generate libs for %s:\n%s\n' %
-                                      (self.name, err_raw))
-        ret, out_all, err_all = self._call_pkgbin(libcmd_all)
-        if ret != 0:
-            mlog.warning('Could not determine complete list of dependencies for %s:\n%s\n' %
-                         (self.name, err_all))
-        self.link_args, self.raw_link_args = self._search_libs(out, out_raw, out_all)
+            raise DependencyException('Could not generate libs for %s:\n\n%s' %
+                                      (self.name, out_raw))
+        self.link_args, self.raw_link_args = self._search_libs(out, out_raw)
 
     def get_pkgconfig_variable(self, variable_name, kwargs):
         options = ['--variable=' + variable_name, self.name]
diff --git a/run_unittests.py b/run_unittests.py
--- a/run_unittests.py
+++ b/run_unittests.py
@@ -5465,45 +5465,6 @@ def test_pkgconfig_internal_libraries(self):
                       override_envvars={'PKG_CONFIG_PATH': pkg_dir})
             self.build()
 
-    @skipIfNoPkgconfig
-    def test_pkgconfig_secondary_dependencies(self):
-        '''
-        Check that Meson gets -Wl,-rpath-link right for secondary dependencies
-
-        This test requires at least two libraries, as -Wl,-rpath-link is only
-        required for dependencies of dependencies (i.e. secondary dependencies).
-        '''
-        with tempfile.TemporaryDirectory() as tempdirname:
-            testdirbase = os.path.join(self.unit_test_dir, '67 rpath-link secondary')
-
-            # build libA
-            testdirlibA = os.path.join(testdirbase, 'libA')
-            testlibAprefix = os.path.join(tempdirname, 'libAprefix')
-            self.init(testdirlibA, extra_args=['--prefix=' + testlibAprefix,
-                                               '--libdir=lib',
-                                               '--default-library=shared'], default_args=False)
-            self.build()
-            self.install(use_destdir=False)
-
-            # build libB (uses libA)
-            pkg_dir = [os.path.join(testlibAprefix, 'lib/pkgconfig')]
-            self.new_builddir()
-            testdirlibB = os.path.join(testdirbase, 'libB')
-            testlibBprefix = os.path.join(tempdirname, 'libBprefix')
-            self.init(testdirlibB, extra_args=['--prefix=' + testlibBprefix,
-                                               '--libdir=lib',
-                                               '--default-library=shared'], default_args=False,
-                      override_envvars={'PKG_CONFIG_PATH': ':'.join(pkg_dir)})
-            self.build()
-            self.install(use_destdir=False)
-
-            # build executable (uses libB, secondary dependency on libA)
-            pkg_dir.append(os.path.join(testlibBprefix, 'lib/pkgconfig'))
-            self.new_builddir()
-            self.init(os.path.join(testdirbase, 'app'),
-                      override_envvars={'PKG_CONFIG_PATH': ':'.join(pkg_dir)})
-            self.build()
-
     @skipIfNoPkgconfig
     def test_static_archive_stripping(self):
         '''


Test patch:
diff --git a/test cases/unit/67 rpath-link secondary/app/app.c b/test cases/unit/67 rpath-link secondary/app/app.c
deleted file mode 100644
--- a/test cases/unit/67 rpath-link secondary/app/app.c	
+++ /dev/null
@@ -1,4 +0,0 @@
-#include <stdio.h>
-#include <libb.h>
-
-int main() { printf("The answer is: %d\n", libB_func()); }
diff --git a/test cases/unit/67 rpath-link secondary/app/meson.build b/test cases/unit/67 rpath-link secondary/app/meson.build
deleted file mode 100644
--- a/test cases/unit/67 rpath-link secondary/app/meson.build	
+++ /dev/null
@@ -1,5 +0,0 @@
-project('app', ['c'])
-
-b = dependency('test-b')
-
-executable('app', 'app.c', dependencies : [b])
diff --git a/test cases/unit/67 rpath-link secondary/libA/liba.c b/test cases/unit/67 rpath-link secondary/libA/liba.c
deleted file mode 100644
--- a/test cases/unit/67 rpath-link secondary/libA/liba.c	
+++ /dev/null
@@ -1 +0,0 @@
-int libA_func() { return 42; }
diff --git a/test cases/unit/67 rpath-link secondary/libA/liba.h b/test cases/unit/67 rpath-link secondary/libA/liba.h
deleted file mode 100644
--- a/test cases/unit/67 rpath-link secondary/libA/liba.h	
+++ /dev/null
@@ -1 +0,0 @@
-int libA_func();
diff --git a/test cases/unit/67 rpath-link secondary/libA/meson.build b/test cases/unit/67 rpath-link secondary/libA/meson.build
deleted file mode 100644
--- a/test cases/unit/67 rpath-link secondary/libA/meson.build	
+++ /dev/null
@@ -1,14 +0,0 @@
-project('lib', ['c'])
-
-a = library('test-a', 'liba.c', install: true)
-
-install_headers(files('liba.h'))
-
-import('pkgconfig').generate(
-    a,
-    version: '0.0',
-    description: 'test library',
-    filebase: 'test-a',
-    name: 'test library',
-    subdirs: ['.']
-)
diff --git a/test cases/unit/67 rpath-link secondary/libB/libb.c b/test cases/unit/67 rpath-link secondary/libB/libb.c
deleted file mode 100644
--- a/test cases/unit/67 rpath-link secondary/libB/libb.c	
+++ /dev/null
@@ -1,3 +0,0 @@
-#include <liba.h>
-
-int libB_func() { return libA_func(); }
diff --git a/test cases/unit/67 rpath-link secondary/libB/libb.h b/test cases/unit/67 rpath-link secondary/libB/libb.h
deleted file mode 100644
--- a/test cases/unit/67 rpath-link secondary/libB/libb.h	
+++ /dev/null
@@ -1 +0,0 @@
-int libB_func();
diff --git a/test cases/unit/67 rpath-link secondary/libB/meson.build b/test cases/unit/67 rpath-link secondary/libB/meson.build
deleted file mode 100644
--- a/test cases/unit/67 rpath-link secondary/libB/meson.build	
+++ /dev/null
@@ -1,16 +0,0 @@
-project('lib', ['c'])
-
-libA_dep = dependency('test-a')
-
-b = library('test-b', 'libb.c', install: true, dependencies : libA_dep)
-
-install_headers(files('libb.h'))
-
-import('pkgconfig').generate(
-    b,
-    version: '0.0',
-    description: 'test library',
-    filebase: 'test-b',
-    name: 'test library',
-    subdirs: ['.']
-)