Repository: spack/spack

Issue:
Spack uses too many processes by default and runs out of memory
Related to #327. I know there was a lot of discussion about it in the previous issue.

Rather than just a performance optimization, this actually causes builds to fail.
Too many processes active at once leads to very strange gcc 'internal compiler error' messages. I *think* this is gcc running out of memory, but have not been able to confirm. 

By default, Spack spins up 160 make jobs, which very quickly burns the available memory (C++ compilations involving Trilinos). Lowering the number of make jobs makes the compiler errors go away.

I believe this should be fairly easy to fix because of the use of the compiler wrapper. The compiler wrapper could check available memory on the node upon exiting with an error. If the available memory is very low, this would be a "non-fatal error" and would trigger a reduction in the number of make jobs.

The notion of a "non-fatal error" that can be diagnosed and fixed might be useful in general if there are other situations like this that might arise.

*make -> error
*diagnose and fix
*make (pick up where left off)




Discussion:
You can manually limit the amount of parallelism by doing `spack install -j20 yourpackage`, or do that setting in your `config.yaml` file via the option `build_jobs: 20`
Yes, that what was I did to get it to work for now (already noted in issue description). I like Spack and am trying to promote Spack usage to a somewhat distrustful audience. This would be a very strange error for a "non-expert." Almost all of our codes are going to involve memory-intensive C++ builds. Our users might very quickly give up on Spack if they get bizarre errors that don't show up when they build outside of Spack.
In #10627, I suggested setting the default by some kind of sliding
scale.  Up to 8, use all cores, then start to back off with a cap of
16 or 20.

My point is that for machines with >> 100 cores (or hardware threads),
then all available cores is not a good default.  In my case, I
accidentally brought down a power8 with 48 cores and 4x HT when spack
ran `-j 192`.  It's just too easy to make a mistake like this.

Of course, this is only the default.  A user can always write `-j 64`
if that's appropriate for your machine.
I agree.  I don't know of any packages for which 160-way builds are significantly faster than 20-way builds.  Would someone like to submit a PR for the backoff feature?  Then we'd have max 20 (or so), unless overridden by `-j`.

I think that spack is still an outlier compared to other build tools that people would have come across and the default should be to use 1 make job.  In general tools don't parallelise builds unless they are asked too, and changing the defaults is a much simpler solution than adding extra code to try to create some kind of pseudo-intelligent solution.
I have had the same problem with Ninja, which parallelizes by default (I think). But then it uses TOO many processes.  I agree that fewer is a better default. I guess a counterargument would be some recent build tools are parallelizing by default.  I like "slow and successful" as a better default than "fast and failed."
I think spack should just limit to 20 jobs by default.  This should work on
99% of systems where it currently does not.  Let’s not overthink this one.

On Wed, Apr 10, 2019 at 09:32 Jeremy <notifications@github.com> wrote:

> I have had the same problem with Ninja, which parallelizes by default (I
> think). But then it uses TOO many processes. I agree that fewer is a better
> default. I guess a counterargument would be some recent build tools are
> parallelizing by default. I like "slow and successful" as a better default
> than "fast and failed."
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/spack/spack/issues/11072#issuecomment-481692349>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AB1cd_VzPBRbRQ5PtFXH8W1dZzufGtIDks5vfef2gaJpZM4cU3vq>
> .
>

Agreed. I like 20 better than 1.
So how does that look?  spack uses ``num_cores if num_cores <= 20 else 20`` jobs.  Presumably this adds a second configuration variable of ``max_jobs`` or similar?

I also think 20 is still too high.  I use several HPCs where if I used 20 cores to build on the login nodes I would be yelled at by sysadmins or kicked by monitoring scripts.
> So how does that look?

Fine by me.  Just remember that there are REAL and VIRTUAL cores these days.  My guess is Spack should base its -j on the number of virtual cores.

@alalazo  Thanks for tackling this one!

Resolving patch:
diff --git a/lib/spack/spack/cmd/common/arguments.py b/lib/spack/spack/cmd/common/arguments.py
--- a/lib/spack/spack/cmd/common/arguments.py
+++ b/lib/spack/spack/cmd/common/arguments.py
@@ -5,6 +5,7 @@
 
 
 import argparse
+import multiprocessing
 
 import spack.cmd
 import spack.config
@@ -86,6 +87,7 @@ def __call__(self, parser, namespace, jobs, option_string):
                   '[expected a positive integer, got "{1}"]'
             raise ValueError(msg.format(option_string, jobs))
 
+        jobs = min(jobs, multiprocessing.cpu_count())
         spack.config.set('config:build_jobs', jobs, scope='command_line')
 
         setattr(namespace, 'jobs', jobs)
@@ -94,7 +96,8 @@ def __call__(self, parser, namespace, jobs, option_string):
     def default(self):
         # This default is coded as a property so that look-up
         # of this value is done only on demand
-        return spack.config.get('config:build_jobs')
+        return min(spack.config.get('config:build_jobs'),
+                   multiprocessing.cpu_count())
 
     @default.setter
     def default(self, value):
diff --git a/lib/spack/spack/config.py b/lib/spack/spack/config.py
--- a/lib/spack/spack/config.py
+++ b/lib/spack/spack/config.py
@@ -100,7 +100,7 @@
         'verify_ssl': True,
         'checksum': True,
         'dirty': False,
-        'build_jobs': multiprocessing.cpu_count(),
+        'build_jobs': min(16, multiprocessing.cpu_count()),
     }
 }
 


Test patch:
diff --git a/lib/spack/spack/test/cmd/common/arguments.py b/lib/spack/spack/test/cmd/common/arguments.py
--- a/lib/spack/spack/test/cmd/common/arguments.py
+++ b/lib/spack/spack/test/cmd/common/arguments.py
@@ -20,14 +20,38 @@ def parser():
     yield p
     # Cleanup the command line scope if it was set during tests
     if 'command_line' in spack.config.config.scopes:
-        spack.config.config.remove_scope('command_line')
+        spack.config.config.scopes['command_line'].clear()
 
 
-@pytest.mark.parametrize('cli_args,expected', [
+@pytest.fixture(params=[1, 2, 4, 8, 16, 32])
+def ncores(monkeypatch, request):
+    """Mocks having a machine with n cores for the purpose of
+    computing config:build_jobs.
+    """
+    def _cpu_count():
+        return request.param
+
+    # Patch multiprocessing.cpu_count() to return the value we need
+    monkeypatch.setattr(multiprocessing, 'cpu_count', _cpu_count)
+    # Patch the configuration parts that have been cached already
+    monkeypatch.setitem(spack.config.config_defaults['config'],
+                        'build_jobs', min(16, request.param))
+    monkeypatch.setitem(
+        spack.config.config.scopes, '_builtin',
+        spack.config.InternalConfigScope(
+            '_builtin', spack.config.config_defaults
+        ))
+    return request.param
+
+
+@pytest.mark.parametrize('cli_args,requested', [
     (['-j', '24'], 24),
-    ([], multiprocessing.cpu_count())
+    # Here we report the default if we have enough cores, as the cap
+    # on the available number of cores will be taken care of in the test
+    ([], 16)
 ])
-def test_setting_parallel_jobs(parser, cli_args, expected):
+def test_setting_parallel_jobs(parser, cli_args, ncores, requested):
+    expected = min(requested, ncores)
     namespace = parser.parse_args(cli_args)
     assert namespace.jobs == expected
     assert spack.config.get('config:build_jobs') == expected