Change:
--set target.x86_64-unknown-linux-gnu.ranlib=/rustroot/bin/llvm-ranlib  --set llvm.thin-lto=true  --set llvm.ninja=false  <del> --set rust.jemalloc  --set rust.use-lld=true </del> <ins> --set rust.jemalloc </ins> ENV SCRIPT ../src/ci/pgo.sh python3 ../x.py dist  --host $HOSTS --target $HOSTS  --include-default-paths 

Discussion:
since / the linux rustc executables distributed by rustup are no longer executable: The toolchain before that merge ( ) works fine for me. My target (is that the right term for it?) is
I think it would be good to get the dynamic linker version, since perf (which also uses Linux) didn't run into this...
If this is your "default" nightly toolchain that you tried to upgrade and it borked, then it is considered the host toolchain by rustup.
this toolchain was installed via to be precise this is (not yet..) the master toolchain, but will be in a few hours. Executing different s by path directly like has always worked regardless of rustup until now.
cc
I think we should revert that PR until we figure out why this is broken.
Heh, that was fast :D I expected obscure errors, but outright failing to execute ... well, that's bad. Filed
So the obvious question is: How was this not caught in CI?
it seems to not happen on every platform, CI obviously ran the tests successfully and tested in a bunch of platforms with old glibc versions / toolchains and it worked on those too.
May we see ?
Need to finish investigation, but hopefully this is now not affecting anyone.
`
I'm not running that kernel, but rustc runs fine for me in the container, using podman on a Fedora 36 host.
was being at the same time as passing a flag with a value of when there apparently is also a value of <sub(which in this context could maybe mean "safe to use with any dynamic linker because it behaves as if LLVM was used originally" or something else like that)</sub. IMO we should reland right away (possibly gated on the nightly release channel so we don't accidentally end up with obscure bugs leaking into stable) and have a few weeks of waiting period to let any reports trickle in about LLD breaking our linking. And only then we could start adding flags (we just need "the next nightly" to be scientific, but still, if LLD does have some weird bugs that haven't been seen before but does hit them, we need to know). I would also recommend landing before , just so we have a better chance at finding the broken link in the LLD -ICF(safe) -ICF(all) chain.

LLD was not landed standalone because it's a regression for us to do so (binary size increases). I suppose we could temporarily land such a change, but it seems like we still are failing to reproduce the bug here, so I'm hesitant to say we'd be able to verify that LLD alone (or safe, or any other options) still cause the problem or not. Maybe we can rely on to test, but that seems horrible (and could be transient, nondeterministic, etc.)
I already did tests with safe ICF and LLD only. Both fail for . So even with just using LLD, it's broken in some environments.
Ah, thanks for testing, I wasn't been able to gather that detail from either the Zulip thread or this thread. That's the kind of situation I was imagining switching only one aspect at a time would find. <hr/The only thing I can think of for diagnosing LLD is doing something like this: for 2 toolchains, using vs for linking, but with the rest of the build unaffected (i.e. ideally only the should be rebuilt between the two cases) And then diffing the two outputs. The order might differ, but ideally the is enough, and the symbol sizes shouldn't differ unless there's weird linker relaxation stuff going on. (I'm assuming we're using , but if saw breakage going from to LLD, it might be worth trying in case it can replicate the issue) <hr/Another experiment that might be interesting is trying to integrate , if it doesn't break in some more obvious reason - it's probably not ready for nightly, but it could shed some light.
Well, ignore all that, this one was user error: had set, causing this behavior: (system libs omitted for brevity) Note how in case, instead of doing the relative traversal specified in the rpath of , it actually uses the fully absolute path to the toolchain. That happened because had this value for : <hr/We can probably reland the original PR (or just turning LLD on if we want to) and see if anything else breaks.
Aha! And here's why setup only broke now: Unrelated to the LLD change, in the same nightly, we stopped changing 's file hash - .

I believe at the time had as the toolchain, and so was being used from there, instead of from the toolchain being executed. EDIT: a few more nightlies, looks like might be what's causing the file hash to change? Whereas always has the same file hash because we force it to: