Repository: easybuilders/easybuild-easyblocks

Issue:
Qt  Make/Ninja interaction
When building Qt5 5.12.8 I observed that the QtWebEngine part was very slow. It uses Ninja inside Make and for some reason Ninja only spawned one g++ process at a time. Now I did use something non-standard, namely using a Gentoo Prefix Ninja, which is a fork with support for the GNU Make jobserver (https://github.com/Kitware/ninja), and that was what caused this interaction.

However when putting ninja 1.10.0 as a builddependency I see this:
```
Checking for system ninja... no
```
and it then proceeds to build and use an included Ninja in (`qtwebengine/src/3rdparty/ninja/ninja`), which is Ninja version 1.8.2 in this case.
From a pure EB perspective I see two issues here:

1. The Qt make procedure has this check (also in 5.14.1): 
```     
        contains(version, "1.[7-9].*"): return(true)
        qtLog("Ninja version too old")
```
Ninja 1.10 fails this test and that's why it uses the included Ninja instead of the builddep.

2. Ninja by default (ie. not the one with GNU Make jobserver support) will gobble up all cores. If you want to avoid that you need to pass it an explicit `-jN` flag, via `prebuildopts = "NINJAFLAGS='-j8'"` (see https://patchwork.ozlabs.org/project/buildroot/patch/20180629143521.9018-1-gael.portay@savoirfairelinux.com/ )


Discussion:
Regarding issue 1: Qt 5.15.0 uses the following check:
```
        contains(version, "1\.([7-9]|1[0-9])\..*"): return(true)
        qtLog("Ninja version too old")
```
So a simple patch should be sufficient here.

Regarding issue 2: Couldn't we use something like `prebuildopts = "NINJAFLAGS='-j%(parallel)s'"` in the easyconfig file? Or is the Ninja build overlapping with some other compilations spawned by `make` such that the jobserver approach would be required to effectively limit parallelism?
Two days ago I was building QT5 for 2019b on our Sandybridge cluster.  The build node has 32GB and it failed about 1h in, with an error “terminated”.  We normally build with 16 cores for that.  Telling EB to give it only 5 cores, made it fail about 2h in, again with error “terminated”.  I took a different node with 96GB and it build.  So did it month ago on our Haswell cluster, where the build node has 64GB.   I had been looking at the build using “htop”.   I had the impression during some parts of the build process more than the EB allowed 16 threads were active.  Link step?  Some testing?

To me the signature is consistent with "out of memory".  I noticed (htop) threads can have around a GB of RSS.  If you have more than 16 running, 32GB is eaten very quickly.

This might or might not be another manifestation of the issue described here.
Small update on this: just setting the `$NINJAFLAGS` environment variable to something like `-j10` is enough, then you see `ninja -j10` popping up in the installation log (without setting `$NINJAFLAGS`, so just see `ninja -v`, which implies it uses all the cores it can see).

Resolving patch:
diff --git a/easybuild/easyblocks/q/qt.py b/easybuild/easyblocks/q/qt.py
--- a/easybuild/easyblocks/q/qt.py
+++ b/easybuild/easyblocks/q/qt.py
@@ -30,11 +30,13 @@
 import os
 from distutils.version import LooseVersion
 
+import easybuild.tools.environment as env
 import easybuild.tools.toolchain as toolchain
 from easybuild.easyblocks.generic.configuremake import ConfigureMake
 from easybuild.framework.easyconfig import CUSTOM
 from easybuild.tools.build_log import EasyBuildError
 from easybuild.tools.filetools import apply_regex_substitutions
+from easybuild.tools.modules import get_software_root
 from easybuild.tools.run import run_cmd_qa
 from easybuild.tools.systemtools import get_cpu_architecture, get_glibc_version, get_shared_lib_ext
 from easybuild.tools.systemtools import AARCH64, POWER
@@ -154,6 +156,13 @@ def configure_step(self):
         ]
         run_cmd_qa(cmd, qa, no_qa=no_qa, log_all=True, simple=True, maxhits=120)
 
+        # Ninja uses all visible cores by default, which can lead to lack of sufficient memory;
+        # so $NINJAFLAGS is set to control number of parallel processes used by Ninja;
+        # note that $NINJAFLAGS is not a generic thing for Ninja, it's very specific to the Qt5 build procedure
+        if LooseVersion(self.version) >= LooseVersion('5'):
+            if get_software_root('Ninja'):
+                env.setvar('NINJAFLAGS', '-j%s' % self.cfg['parallel'])
+
     def build_step(self):
         """Set $LD_LIBRARY_PATH before calling make, to ensure that all required libraries are found during linking."""
         # cfr. https://elist.ornl.gov/pipermail/visit-developers/2011-September/010063.html


Test patch: