Change:
CI time quadrupled when the new pass manager was enabled: Last CI before the new pass manager was enabled: CI with the new pass manager was enabled: Right now that does not really matter much because Github's Apple CI runners use very old hardware and take about the same time, but once that is fixed this will block a faster CI. No idea if that platform has any significant userbase. Maybe disable the new PM for that platform or report it upstream?
This regression absolutely needs to be reported upstream, as LLVM dev talk has mentioned completely removing the old pass manager within 1~2 versions.
Most time is apparently being spent compiling : This crate usually takes less than 30 seconds to build.
The build timeout in might be caused by this as well.
That is very surprising and am wondering what we could be doing for that not to be O(n).
cc I am a little tempted to suggest a patch that passes -Zno-new-pass-manager or w/e on s390x, but that seems pretty unfortunate as well, given the relatively quick deprecation timeline.
Is there anything I can do to unblock ? Or will it require the patch that is suggesting?
Just checking, has the regression been reported upstream yet? I didn't see a report when I searched the LLVM Bugzilla. (I don't have an LLVM Bugzilla account, so I can't report it.)
I'm going to try to find the culprit, but in the meantime to unblock folks that hack could be an extra target condition in . Note that the host arch is apparently not the issue, since these are all running on x8664, cross-compiling to various targets.
I have the docker build running locally, with LLVM debuginfo, and shows a couple hotspots. First it was sitting here at about 31% of the samples: Then it moved away from that to a new hotspot at 28%: Next it moved to this at 49%: It's still going, but I'm not sure this is helpful enough to keep watching this way...
See also , which is not s390x specific.
Based on those results, it looks to me (albeit as someone who's not knowledgeable about LLVM internals) like maybe it has some really large data structures that are slow to process. Could that be it?
I have just tried for with the new pass manager disabled and it built in about 30 minutes locally. So this change is definitely blocked by this issue.
I managed to collect for the troublesome crate: So it took almost 15 minutes from to (running the pass manager), growing from 2 to 33 MB of LLVM bitcode. Then it took almost an hour to actually codegen that into an object file. At one point during that hour, I noticed the process was using 24GB of resident memory! In that 4 million lines of LLVM IR, about 2.7 million is just in 2 instantiations of ! That longest line of characters is the line below: I cut them off with "etc.", but the full and lists are each entries long! AFAICS, all of those predecessors are simple blocks like the one I included here. The second has the same thing but only characters long, I think just from a couple shorter number lengths. The same blocks in (before pm) look like this:
Here's the input bitcode before the pass manager: Even behaves badly, while runs quickly without blowing up the bitcode.
cc -- this is another new-pm problem, although it appears to be specific to SystemZ.
Filed: (now migrated to llvm/llvm-project)
Assigning priority as discussed in the of the Prioritization Working Group. label -I-prioritize +P-high
Status: disabled newPM when targeting s390x, but this is a temporary solution because oldPM will be removed eventually. The underlying issue does seem to be excessive inlining, similar to and others from newPM.
If anyone is wondering why this one only occurs on SystemZ: Apparently this target just increases all inlining threshold by a factor of three...
Aha, thanks! The only other targets that change that multiplier are NVPTX (5) and AMDGPU (11).

Discussion:
// The new pass manager is enabled by default for LLVM >= 13. // This matches Clang, which also enables it since Clang 13. <del> // FIXME: There are some perf issues with the new pass manager // when targeting s390x, so it is temporarily disabled for that // arch, see https://github.com/rust-lang/rust/issues/89609 user_opt.unwrap_or_else(|| target_arch != "s390x" && llvm_util::get_version() >= (13, 0, 0)) </del> <ins> // There are some perf issues with the new pass manager when targeting // s390x with LLVM 13, so enable the new pass manager only with LLVM 14. // See https://github.com/rust-lang/rust/issues/89609. let min_version = if target_arch == "s390x" { 14 } else { 13 }; user_opt.unwrap_or_else(|| llvm_util::get_version() >= (min_version, 0, 0)) </ins> }

<del> Subproject commit e29ac13bc97e26f886c3bfe72f9135e994c3cd0a </del> <ins> Subproject commit c8eccf626fb5bb851b2ade93af8851ca1523807f </ins>

// run-pass <del> // ignore-wasm32-bare FIXME(#93923) llvm miscompilation </del> use std::collections::HashMap; use std::path::Path;