Change:
// The new pass manager is enabled by default for LLVM >= 13. // This matches Clang, which also enables it since Clang 13. <del> // FIXME: There are some perf issues with the new pass manager // when targeting s390x, so it is temporarily disabled for that // arch, see https://github.com/rust-lang/rust/issues/89609 user_opt.unwrap_or_else(|| target_arch != "s390x" && llvm_util::get_version() >= (13, 0, 0)) </del> <ins> // There are some perf issues with the new pass manager when targeting // s390x with LLVM 13, so enable the new pass manager only with LLVM 14. // See https://github.com/rust-lang/rust/issues/89609. let min_version = if target_arch == "s390x" { 14 } else { 13 }; user_opt.unwrap_or_else(|| llvm_util::get_version() >= (min_version, 0, 0)) </ins> }

Discussion:
Discovered in I don't have reduced code, but the testsuite fails on when compiled in mode, specifically on this line: It fails when compiled from x8664 host: or with a native i686 host toolchain: I expected to see this happen: the test should pass Instead, this happened: FAILED The left is wrong, and the right is correct. They both come from the same generated sieve, so that part of the code appears to have worked correctly. That leaves the difference between the loop and the , and while does implement both and , the differences are small. The same test passes on i686 in debug mode, as well as x8664 in debug or release. It most recently worked on: stable Rust 1.58.1 Both beta and nightly are failing the same way. modify labels: +regression-from-stable-to-beta -regression-untriaged
I just tried under qemu to see if the bug is a 32-bit thing, but that passes fine.
It passes on i686 release with .
The relevant difference with the NewPM seems to be that the next() method gets inlined now.
Assigning priority as discussed in the of the Prioritization Working Group. label -I-prioritize +P-critical
Just tested this with the toolchain and it still fails, so this issue is not fixed in LLVM 14.
does have an attribute, and if I remove that, the test passes. But even is not enough to make oldPM fail. Also, forcing newPM with 1.58.1 still passes as well! Between stable and beta, the most significant change looks like the patch to disable deferred inlining -- and the test does pass on beta and nightly with . (That doesn't mean the inlining decision is wrong -- it's probably just creating circumstances that are buggy in another pass.)
and I just found a slight variant that shows a similar failure on x86_64....
We're discussing in this . After discovering the variant where x86_64 fails, we found that it also failed on stable back to 1.55, but passed on 1.54. Bisecting between those took me to , a simple addition, which I take to mean it just perturbed the code enough to expose a pre-existing codegen bug. So, I'm removing the regression label, and we can treat it like any ongoing bug.

FYI I took the test from the primal crate and reduced it down to a single (large!) file: I suspect this is still too absurdly large to turn into LLVM bitcode to file a bug with LLVM. But, just as i write this, I realize it might suffice as input for the bitcode automatic reduction tools that LLVM distributes. Looking in that next.
This is flagged as P-critical, but I do not believe it is a 1.59.0 release blocker. (And I think the release team .) The investigation, covered in , determined this bug is visible as far back as 1.55.0. So: should it be downgraded to P-high? I think the nature of the bug (outright miscompilation) warrants us covering it on a weekly basis in T-compiler meeting, at least for now. So I won't downgrade it yet; we can let the team decide if it should be downgraded during regular triage meeting.
Upstream issue: Not a great reduction, but my attempts to construct a minimal MIR test failed.
The does work for me, thanks!
Priority label has been mentioned during T-compiler weekly meeting . This will be reviewed in a follow-up meeting.

Context: - the test had to be disabled there to get a fix in With running the UI tests targeting wasm32 causes this failure with LLVM asserts enabled: With LLVM asserts disabled it produces invalid wasm bytecode: Manually executing the rustc command printed by the UI test and adding generates the following IR . Running that through (built by bootstrap) also produces invalid bytecode (checked with ). Tested on base commit + PR (currently ) ping llvm
Hey LLVM ICE-breakers! This bug has been identified as a good "LLVM ICE-breaking candidate". In case it's useful, here are some [instructions] for tackling these sorts of bugs. Maybe take a look? Thanks! <3 [instructions]: https://rustc-dev- cc
Quick observations: The failing assert: Looking at the , it seems like the only way this assert could fail is if the value is too large to fit in the specified number of bytes. There are two calls to , and one of them has a check for out-of-range values already, so it's probably in .
No longer blocks the PR, it now disables the test on wasm32 instead.
Upstream issue:

CI time quadrupled when the new pass manager was enabled: Last CI before the new pass manager was enabled: CI with the new pass manager was enabled: Right now that does not really matter much because Github's Apple CI runners use very old hardware and take about the same time, but once that is fixed this will block a faster CI. No idea if that platform has any significant userbase. Maybe disable the new PM for that platform or report it upstream?
This regression absolutely needs to be reported upstream, as LLVM dev talk has mentioned completely removing the old pass manager within 1~2 versions.
Most time is apparently being spent compiling : This crate usually takes less than 30 seconds to build.
The build timeout in might be caused by this as well.
That is very surprising and am wondering what we could be doing for that not to be O(n).
cc I am a little tempted to suggest a patch that passes -Zno-new-pass-manager or w/e on s390x, but that seems pretty unfortunate as well, given the relatively quick deprecation timeline.
Is there anything I can do to unblock ? Or will it require the patch that is suggesting?
Just checking, has the regression been reported upstream yet? I didn't see a report when I searched the LLVM Bugzilla. (I don't have an LLVM Bugzilla account, so I can't report it.)
I'm going to try to find the culprit, but in the meantime to unblock folks that hack could be an extra target condition in . Note that the host arch is apparently not the issue, since these are all running on x8664, cross-compiling to various targets.
I have the docker build running locally, with LLVM debuginfo, and shows a couple hotspots. First it was sitting here at about 31% of the samples: Then it moved away from that to a new hotspot at 28%: Next it moved to this at 49%: It's still going, but I'm not sure this is helpful enough to keep watching this way...
See also , which is not s390x specific.
Based on those results, it looks to me (albeit as someone who's not knowledgeable about LLVM internals) like maybe it has some really large data structures that are slow to process. Could that be it?
I have just tried for with the new pass manager disabled and it built in about 30 minutes locally.

So this change is definitely blocked by this issue.
I managed to collect for the troublesome crate: So it took almost 15 minutes from to (running the pass manager), growing from 2 to 33 MB of LLVM bitcode. Then it took almost an hour to actually codegen that into an object file. At one point during that hour, I noticed the process was using 24GB of resident memory! In that 4 million lines of LLVM IR, about 2.7 million is just in 2 instantiations of ! That longest line of characters is the line below: I cut them off with "etc.", but the full and lists are each entries long! AFAICS, all of those predecessors are simple blocks like the one I included here. The second has the same thing but only characters long, I think just from a couple shorter number lengths. The same blocks in (before pm) look like this:
Here's the input bitcode before the pass manager: Even behaves badly, while runs quickly without blowing up the bitcode.
cc -- this is another new-pm problem, although it appears to be specific to SystemZ.
Filed: (now migrated to llvm/llvm-project)
Assigning priority as discussed in the of the Prioritization Working Group. label -I-prioritize +P-high
Status: disabled newPM when targeting s390x, but this is a temporary solution because oldPM will be removed eventually. The underlying issue does seem to be excessive inlining, similar to and others from newPM.
If anyone is wondering why this one only occurs on SystemZ: Apparently this target just increases all inlining threshold by a factor of three...
Aha, thanks! The only other targets that change that multiplier are NVPTX (5) and AMDGPU (11).