Repository: mesonbuild/meson

Issue:
Support thin archives for static libraries that are not installed
I have a project which generates several executables which (partially) share the same source files.
Since each executable is first compiled in its own subdirectory, all these shared source files are recompiled for every executable. This is undesirable if I know that identical object files will be produced (that is, the compiler options are identical).

Is there a way to specify the name of the build-subdirectory to avoid this repeated redundant compilation?

I am aware that one could use static libraries to work around this problem, but besides being overkill this also has several disadvantages: It also has the overhead of building the library, and moreover, to support e.g. proper support for LTO it requires linker plugins etc.


Discussion:
LTO works just fine across static libraries. The overhead of creating static libraries is small, but we should actually start supporting "thin archives" for static libraries that are never installed. That would completely remove that overhead too, for cases in which the static library contains many object files.
> LTO works just fine across static libraries.

This information is wrong. I first believed this information and unfortunately made a buggy release which breaks for people who have not configured linker plugins:

Only if gcc has linker plugins installed (e.g. if /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins contains a symlink to the current gcc's liblto_plugin.so.0.0.0) static libraries compiled with -flto (and without -ffat-lto-objects) can be accessed correctly.

Here is the (correct) bug report which I received when people do not have the plugin installed:
https://github.com/vaeth/mv-overlay/issues/2

Interesting, I wonder if it's a distro-specific toolchain thing. Maybe Gentoo should enable the liblto plugin by default?

Could you provide a test case for LTO via static archives? The last time I tested it, it worked for me on Fedora. I would like to investigate this more.
Strange. Now I cannot reproduce it anymore. Of course, my toolchain has changed meanwhile, so there may be a lot of causes. I am sure, everything I did last time to get the error was to remove my manually set symlink `/usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins/liblto_plugin.so.0.0.0` (on amd64) or `/usr/i686-pc-linux-gnu/binutils-bin/lib/bfd-plugins/liblto_plugin.so.0.0.0` (on i686).

I append a somewhat minimal meson.build which should produce the error (though currently it doesn't, here). Note the usage of -fno-fat-lto-objects so that really some sort of ar plugin/wrapper must be used to access the static archive.
```meson
project('test','cpp',version:'0')
run_command(find_program('sh'), '-c', '''
echo 'int dummy(int i);int dummy(int i){return i;}' >sub.cc
echo 'int dummy(int i);int main(){return dummy(0);}' >main.cc
''')
args=['-use-linker-plugin','-flto','-fno-fat-lto-objects']
add_global_arguments(args,language:'cpp')
add_global_link_arguments(args,language:'cpp')
executable('main','main.cc',link_with:static_library('sub','sub.cc'))
```
Now I found the reason:

meson defaults to gcc-ar which does not require a linker plugin. The linker plugin is required if one uses the original binutil's ar. For instance, on my gentoo system the above meson.build fails when called after
```export AR=ar```
or, equivalently, (on an amd64 system)
```export AR=x86_64-pc-linux-gnu-ar```
or (on an x86 system)
```export AR=i686-pc-linux-gnu-ar```
Gentoo's meson.eclass by default exports this architecture-specific AR variable (alongside with architecture-specific CC, CXX, STRIP) depending on cross-compilation mode. This behaviour appears to be correct to me (note that also other compilers than gcc might be used, especially when cross-compiling). Unfortunately, it does not work without the symlinks mentioned in the previous comment.

@nirbheek I do not know why gentoo does not set the mentioned symlinks by default. I suppose that it has something to do with its cross-compiler support and for supporting switching gcc versions on the fly. (In the moment, things seem to work when the symlink points to a non-matching gcc version's library, but this might be accidental and change.)
A corresponding bug is reported for gentoo https://bugs.gentoo.org/show_bug.cgi?id=630066

Independent of that, it would be nice to have a way to avoid creating a static archive which is actually unneeded since the .o files could be accessed directly in the corresponding subdirectory.
> Independent of that, it would be nice to have a way to avoid creating a static archive which is actually unneeded since the .o files could be accessed directly in the corresponding subdirectory.

People have requested an "object library" of sorts from time to time, but their use-cases can always be fulfilled by static libraries, and a thin static library would cover even the 'hundreds of objects' use-case. We are hesitant to add an 'object bundle' unless absolutely needed because a static library is already supposed to be that.

For instance, if there are performance problems or features needed that make the use of a static library suboptimal.
This feature request doesn't seem to have a high demand. Not sure if there is any plan to implement this in the future

Resolving patch:
diff --git a/mesonbuild/backend/ninjabackend.py b/mesonbuild/backend/ninjabackend.py
--- a/mesonbuild/backend/ninjabackend.py
+++ b/mesonbuild/backend/ninjabackend.py
@@ -2790,7 +2790,7 @@ def get_target_type_link_args(self, target, linker):
             if target.import_filename:
                 commands += linker.gen_import_library_args(self.get_import_filename(target))
         elif isinstance(target, build.StaticLibrary):
-            commands += linker.get_std_link_args()
+            commands += linker.get_std_link_args(not target.should_install())
         else:
             raise RuntimeError('Unknown build target type.')
         return commands
diff --git a/mesonbuild/linkers/linkers.py b/mesonbuild/linkers/linkers.py
--- a/mesonbuild/linkers/linkers.py
+++ b/mesonbuild/linkers/linkers.py
@@ -59,7 +59,7 @@ def get_base_link_args(self, options: 'KeyedOptionDictType') -> T.List[str]:
     def get_exelist(self) -> T.List[str]:
         return self.exelist.copy()
 
-    def get_std_link_args(self) -> T.List[str]:
+    def get_std_link_args(self, is_thin: bool) -> T.List[str]:
         return []
 
     def get_buildtype_linker_args(self, buildtype: str) -> T.List[str]:
@@ -167,23 +167,16 @@ def __init__(self, exelist: T.List[str], machine: str):
         VisualStudioLikeLinker.__init__(self, machine)
 
 
-class ArLinker(StaticLinker):
-
-    def __init__(self, exelist: T.List[str]):
-        super().__init__(exelist)
-        self.id = 'ar'
-        pc, stdo = mesonlib.Popen_safe(self.exelist + ['-h'])[0:2]
-        # Enable deterministic builds if they are available.
-        if '[D]' in stdo:
-            self.std_args = ['csrD']
-        else:
-            self.std_args = ['csr']
-        self.can_rsp = '@<' in stdo
+class ArLikeLinker(StaticLinker):
+    # POSIX requires supporting the dash, GNU permits omitting it
+    std_args = ['-csr']
 
     def can_linker_accept_rsp(self) -> bool:
-        return self.can_rsp
+        # armar / AIX can't accept arguments using the @rsp syntax
+        # in fact, only the 'ar' id can
+        return False
 
-    def get_std_link_args(self) -> T.List[str]:
+    def get_std_link_args(self, is_thin: bool) -> T.List[str]:
         return self.std_args
 
     def get_output_args(self, target: str) -> T.List[str]:
@@ -193,16 +186,36 @@ def rsp_file_syntax(self) -> RSPFileSyntax:
         return RSPFileSyntax.GCC
 
 
-class ArmarLinker(ArLinker):  # lgtm [py/missing-call-to-init]
+class ArLinker(ArLikeLinker):
+    id = 'ar'
 
     def __init__(self, exelist: T.List[str]):
-        StaticLinker.__init__(self, exelist)
-        self.id = 'armar'
-        self.std_args = ['-csr']
+        super().__init__(exelist)
+        stdo = mesonlib.Popen_safe(self.exelist + ['-h'])[1]
+        # Enable deterministic builds if they are available.
+        stdargs = 'csr'
+        thinargs = ''
+        if '[D]' in stdo:
+            stdargs += 'D'
+        if '[T]' in stdo:
+            thinargs = 'T'
+        self.std_args = [stdargs]
+        self.std_thin_args = [stdargs + thinargs]
+        self.can_rsp = '@<' in stdo
 
     def can_linker_accept_rsp(self) -> bool:
-        # armar can't accept arguments using the @rsp syntax
-        return False
+        return self.can_rsp
+
+    def get_std_link_args(self, is_thin: bool) -> T.List[str]:
+        # FIXME: osx ld rejects this: "file built for unknown-unsupported file format"
+        if is_thin and not mesonlib.is_osx():
+            return self.std_thin_args
+        else:
+            return self.std_args
+
+
+class ArmarLinker(ArLikeLinker):
+    id = 'armar'
 
 
 class DLinker(StaticLinker):
@@ -212,7 +225,7 @@ def __init__(self, exelist: T.List[str], arch: str, *, rsp_syntax: RSPFileSyntax
         self.arch = arch
         self.__rsp_syntax = rsp_syntax
 
-    def get_std_link_args(self) -> T.List[str]:
+    def get_std_link_args(self, is_thin: bool) -> T.List[str]:
         return ['-lib']
 
     def get_output_args(self, target: str) -> T.List[str]:
@@ -291,16 +304,9 @@ def get_linker_always_args(self) -> T.List[str]:
         return ['-r']
 
 
-class AIXArLinker(ArLinker):
-
-    def __init__(self, exelist: T.List[str]):
-        StaticLinker.__init__(self, exelist)
-        self.id = 'aixar'
-        self.std_args = ['-csr', '-Xany']
-
-    def can_linker_accept_rsp(self) -> bool:
-        # AIXAr can't accept arguments using the @rsp syntax
-        return False
+class AIXArLinker(ArLikeLinker):
+    id = 'aixar'
+    std_args = ['-csr', '-Xany']
 
 
 def prepare_rpaths(raw_rpaths: str, build_dir: str, from_dir: str) -> T.List[str]:
@@ -1119,7 +1125,7 @@ def __init__(self, exelist: T.List[str]):
         self.id = 'ar'
         self.std_args = ['-r']
 
-    def get_std_link_args(self) -> T.List[str]:
+    def get_std_link_args(self, is_thin: bool) -> T.List[str]:
         return self.std_args
 
     def get_output_args(self, target: str) -> T.List[str]:
diff --git a/unittests/allplatformstests.py b/unittests/allplatformstests.py
--- a/unittests/allplatformstests.py
+++ b/unittests/allplatformstests.py
@@ -1447,7 +1447,7 @@ def build_static_lib(self, compiler, linker, source, objectfile, outfile, extra_
             link_cmd = ['ar', 'csr', outfile, objectfile]
         link_cmd = linker.get_exelist()
         link_cmd += linker.get_always_args()
-        link_cmd += linker.get_std_link_args()
+        link_cmd += linker.get_std_link_args(False)
         link_cmd += linker.get_output_args(outfile)
         link_cmd += [objectfile]
         self.pbcompile(compiler, source, objectfile, extra_args=extra_args)


Test patch: