Repository: mesonbuild/meson

Issue:
Massive regression in performance of the ninja backend
I noticed that despite the perf improvements in the interpreter run time that were achieved for 0.46, the configure time for gtk+ actually went up by a few seconds. So I added some profiling and found this:

`intr.run()` time went from [25.2s](https://github.com/mesonbuild/meson/files/1943956/intr-0.45.1.log) to [13.8s](https://github.com/mesonbuild/meson/files/1943958/intr-0.47.0.dev1.log)
`g.generate()` time went from [4.2s](https://github.com/mesonbuild/meson/files/1943959/generate-0.45.1.log) to [*20.6s*](https://github.com/mesonbuild/meson/files/1943960/generate-0.47.0.dev1.log)

The above links are all profile stat dumps which can be parsed with the [pstats module](https://docs.python.org/3/library/profile.html#module-pstats). I haven't looked into the profiles properly yet.


Discussion:
Looks like this was because of #3094:

```
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   536    0.225    0.000   16.008    0.030 mesonbuild/backend/ninjabackend.py:2425(guess_external_link_dependencies)
116446    1.412    0.000    7.769    0.000 mesonbuild/backend/ninjabackend.py:2417(guess_library_absolute_path)
```

16 seconds out of 20.6 seconds is spent in that function, with 7 seconds spent just finding the absolute path for all libraries. Probably because we're re-computing the same stuff over and over.

CCing @textshell.
I will not be able to look into this until later this week.
But that sounds like either an insane amount of library directories or an insane amount of duplication of libraries?
If it is the later i wonder if #3414 would make a difference by reducing duplicated libraries (by generating abs paths earlier and thus allowing recently added dedup code that only works with abs paths to remove duplicates)?

Resolving patch:
diff --git a/mesonbuild/backend/ninjabackend.py b/mesonbuild/backend/ninjabackend.py
--- a/mesonbuild/backend/ninjabackend.py
+++ b/mesonbuild/backend/ninjabackend.py
@@ -2429,8 +2429,8 @@ def guess_external_link_dependencies(self, linker, target, commands, internal):
         #   https://sourceware.org/bugzilla/show_bug.cgi?id=22843
         # * Meson optimizes libraries from the same build using the symbol extractor.
         #   Just letting ninja use ld generated dependencies would undo this optimization.
-        search_dirs = []
-        libs = []
+        search_dirs = OrderedSet()
+        libs = OrderedSet()
         absolute_libs = []
 
         build_dir = self.environment.get_build_dir()
@@ -2451,23 +2451,24 @@ def guess_external_link_dependencies(self, linker, target, commands, internal):
                         break
                 if not os.path.isabs(path):
                     path = os.path.join(build_dir, path)
-                search_dirs.append(path)
+                search_dirs.add(path)
             elif item.startswith('-l'):
                 if len(item) > 2:
-                    libs.append(item[2:])
+                    lib = item[2:]
                 else:
                     try:
-                        libs.append(next(it))
+                        lib = next(it)
                     except StopIteration:
                         mlog.warning("Generated linker command has '-l' argument without following library name")
                         break
+                libs.add(lib)
             elif os.path.isabs(item) and self.environment.is_library(item) and os.path.isfile(item):
                 absolute_libs.append(item)
 
         guessed_dependencies = []
         # TODO The get_library_naming requirement currently excludes link targets that use d or fortran as their main linker
         if hasattr(linker, 'get_library_naming'):
-            search_dirs += linker.get_library_dirs()
+            search_dirs = list(search_dirs) + linker.get_library_dirs()
             prefixes_static, suffixes_static = linker.get_library_naming(self.environment, 'static', strict=True)
             prefixes_shared, suffixes_shared = linker.get_library_naming(self.environment, 'shared', strict=True)
             for libname in libs:
diff --git a/mesonbuild/compilers/c.py b/mesonbuild/compilers/c.py
--- a/mesonbuild/compilers/c.py
+++ b/mesonbuild/compilers/c.py
@@ -44,6 +44,8 @@
 
 
 class CCompiler(Compiler):
+    library_dirs_cache = {}
+
     def __init__(self, exelist, version, is_cross, exe_wrapper=None, **kwargs):
         # If a child ObjC or CPP class has already set it, don't set it ourselves
         if not hasattr(self, 'language'):
@@ -157,7 +159,7 @@ def get_include_args(self, path, is_system):
     def get_std_shared_lib_link_args(self):
         return ['-shared']
 
-    def get_library_dirs(self):
+    def get_library_dirs_real(self):
         env = os.environ.copy()
         env['LC_ALL'] = 'C'
         stdo = Popen_safe(self.exelist + ['--print-search-dirs'], env=env)[1]
@@ -167,6 +169,12 @@ def get_library_dirs(self):
                 return libstr.split(':')
         return []
 
+    def get_library_dirs(self):
+        key = tuple(self.exelist)
+        if key not in self.library_dirs_cache:
+            self.library_dirs_cache[key] = self.get_library_dirs_real()
+        return self.library_dirs_cache[key][:]
+
     def get_pic_args(self):
         return ['-fPIC']
 


Test patch: