Change:
aarch64-unknown-linux-musl binaries crash immediately when built using Rust nightly since 2018-02-05, including the current beta, 1.25.0-beta.9. This happens in debug and release builds. It works fine with 2018-02-04, or with stable Rust 1.24.1. This is building on an x86_64-unknown-linux-gnu host, which shows no errors or warnings, and running on an embedded Linux 4.9 device. I tried this code: A fresh . (Same thing with a completely empty "main {}" function.) I expected to see this happen: The binary should run without panicking. With nightly 2018-02-04, it looks like this, through strace: Instead, with 2018-02-05+, this happened: With strace: Here's the backtrace from the core, using gdb 7.1.2: : Any nightly Rust since 2018-02-05, installed through rustup. Example: My is from GCC 7.2.0 via Buildroot 2017.08. Note: I've tried adding per but it made no difference.
I see the libc repo had a similar-sounding issue, but it was back in November 2017, and Rust builds have been working for me until February 5 2018. It may have prevented more visibility, though. See: rust-lang/libc and
A colleague found that dynamically linking libgcc (and libc) using "target-feature=-crt-static" works around this issue.
I wonder if it's related to
If indeed it worked in 2018-02-04 and failed in 2018-02-05, then the suspicious commit range is .. (), presuming I am interpreting this rustup output correctly
can you validate the commits from the two nightlies? That commit range looks a bit suspicious. Just include the output from each rustc that you tested with.
triage: P-high We should figure out what is happening here.
Sure! Here are the outputs. Working 2018-02-04 nightly: Non-working 2018-02-05 nightly: I also bisected with the beta releases using rustup and found that 2018-02-13 (1.24.0-beta.12) works, but the next beta 2018-02-20 (1.25.0-beta.2) crashes. Last working beta: First crashing beta: I also confirmed that the just-released 1.25.0-beta.10 and nightly-2018-03-14 still crash. My process is basically to the version, the version, , and . (I used because I didn't know had a argument until just now.)
Now I see what Niko meant about a potentially suspicious commit range... aside from an RLS/rustfmt change from , there were only two merges: (which frankly I don't understand :)) and to disable ThinLTO.
Could you check which one of the three PRs are the cause? the nightly toolchain the build artifacts from these three commits in ... for each PR with
Sure, thanks for making the tool! : works. : works. : crashes. That's the ThinLTO change from fixing . Please let me know any other information I can provide!
Thanks! Looks like another trusting-trust issue then .
I tried with "-Z", "thinlto=no" and with "-Z", "thinlto=yes" in my 's rustflags, but it crashed either way.
Fascinating. That was our guess from the compiler team meeting, though it seemed unlikely as that PR ought to improve reliability in general.
I could use any suggestions for how to reproduce this problem =) Some have suggested qemu? That said, I'm not sure where to start debugging this. , be an LLVM problem? I'm sort of hoping that one of the LLVM upgrades will make it go away. =) In any case, reproducing it would be a start.
Yeah, I think qemu-aarch64 is the way to go, and perhaps the "bleeding edge" aarch64/musl toolchain from -- I'll see if I can build a repro using those tools.
Here's how I reproduced from scratch: I used a Fedora 27 EC2 instance, though anything recent with qemu should be fine - I downloaded the aarch64/musl "bleeding edge" toolchain from and extracted it in the home directory, to get the linker: I set to: Install rustup using Install toolchains and targets: Hello world: Observe success: Observe crash:
This appears to be fixed again on nightly-2018-03-16, I guess because of
I can confirm it started working for me as well in the latest nightly! This is with nightly-2018-03-17; nightly-2018-03-16 still crashed. To be specific: Crashes: Works! I'll work on bisecting to pinpoint the merge that fixed it. Don't want this to reoccur!
Unfortunately rustup-toolchain-install-master wasn't able to fetch the artifacts for all the intervening commits; I'm guessing the artifacts either aren't uploaded for every build, they're just not uploaded yet, or some commits were built together. Anyway, it seems to work with all of these commits, but I'm not confident without being able to test all of them and confirm the negative case.
To try to narrow it down a bit more, and confirm my findings, I tested the 2018-03-17 nightly with a few different settings. I checked with release and debug builds, and with "-Z", "thinlto=yes" and "-Z", "thinlto=no" in rustflags in , and all four combinations worked. I tested with 'opt-level = "s"' and that was fine. (Output size is quite important to me.) However, adding "lto = true" to causes the crash again! I dug into the combinations a bit: debug build, no incremental compilation, lto = true =no crash! :) debug build, no incremental compilation, lto = false =no crash debug build, incremental compilation, lto = false =no crash release build, no incremental compilation, lto = true =crash! :( release build, no incremental compilation, lto = false =no crash release build, incremental compilation, lto = false =no crash So I tried to figure out why the release build crashed when the debug build didn't. Adding "codegen-units = 1" still crashed. Adding "opt-level = 0" stopped the crash! "opt-level = 1" was still OK. "opt-level = 2" crashed! (And 'opt-level = "s"' and 'opt-level = 3' crash too, as you might expect.) I hope this helps -- it's something about going from opt-level 1 to 2 with LTO that's still crashing, even with the new nightly. Non-LTO builds, and LTO builds with opt-level 0/1, are now fine.
btw, just authored this awesome project: which might be worth trying out here (even if you've already got it narrowed down).
I'm just catching up -- glad that it's (partly at least) fixed. That also gives us a lot of data about what could be causing the problem.
Two other potentially interesting pieces of data: I didn't try opt-level = "z" in my last round of testing, but I can confirm that one works with 2018-03-17. I wouldn't have expected that, when "s" crashes. I had earlier reported "A colleague found that dynamically linking libgcc (and libc) using "target-feature=-crt-static" works around this issue." He says that this no longer works with the 2018-03-17 nightly that fixes some of the statically linked cases.
So it seems clear that I am not the best person to investigate this, and I've also got a few other things to look into right now, so I don't really have time. I'm not sure who is, but we've got a lot of data collected. So the plan is that I (or someone else) will summarize the current state: This is how to reproduce: it works with this rustc, fails with that one what commits are in that range and we'll put out a call and see if anyone can help us narrow down the problem. I suspect some kind of LLVM bug here still and it'd be nice to know what to pin it on.
Looking at the backtrace, it seems that this issue is related to thread-local storage. It seems that this function is causing a segfault: I will have a look at the disassembly & relocations generated for that TLS access, I think something strange might be happening at LTO/link time.
OK, so I think that I've found the source of the bug: For some reason, the addresses of all TLS variables are offset by an additional 0x10. This behavior happens in nightly-2018-02-05 (broken) but not in nightly-2018-02-04 (good). I think this may have gone unnoticed in the past since all TLS was shifted by 0x10, and the TLS was zero-initialized. In this specific case, one of the bytes of the TLS data has an initial value of 0x3, but due to the 0x10 shift it is accessed at the wrong offset by the program.
Now I'm completely stumped as to what actually caused this bug. It seems like a bug in LLVM rather than rustc, possibly related to LTO since the linker is getting confused about TLS offsets.
visited for triage. It seems we haven't made progress since the last report. I am wondering whether we can enlist someone to act as a local "LLVM LTO bug identification" expert...
can we confirm that it is an LTO problem?
To me it looks like a linker bug: the TLS relocations are being resolved to the wrong value. Since it is very unlikely that the linker has been broken this whole time, I would blame it on LTO somehow interfering with the linker.
-- question: do you think you can narrow this down to just LLVM IR inputs that reflect the error, so we can open a bug on the LLVM side?
triage: P-medium Next steps are to diagnose the LLVM problem. Filing under
I have a minimal reproduction: The bug only seems to trigger when libstd is linked into the final binary. The expected output is , which is the initial value of the TLS variable. However when libstd is linked in, the output is because the TLS offsets are incorrect. Bad version: Good version:
Switching the linker between bfd, gold and lld doesn't seem to make any difference.
There's a TLS-related fix in musl that applies to aarch64 and some other architectures; it will probably be in the 1.1.20 release. I wonder if it helps with this!
That might very well be the solution! I noticed that the section in libstd was aligned to 32 bytes, which may very well explain why it's not being handled correctly. So basically, my earlier hypothesis is incorrect: the compiler/linker are calculating the TLS offsets correctly, it's just that musl isn't handling over-aligned TLS sections correctly.
When new nightly build expected to appear with this fix? I'm trying to use to get fresh build myself but always run into compilation issues.
- I've confirmed that your minimal reproduction prints when musl has applied, and otherwise.
I have build my project with rust compiled from sources with fresh musl - and can confirm it works.
Are you cross compiling? I'm trying cross compile to test the tls fix but I get the error from and
Use this command as a workaround:

Discussion:
export CFLAGS="-fPIC $CFLAGS" <del> MUSL=musl-1.1.18 </del> <ins> # FIXME: remove the patch when upate to 1.1.20 MUSL=musl-1.1.19 </ins> # may have been downloaded in a previous run if [ ! -d $MUSL ]; then curl https://www.musl-libc.org/releases/$MUSL.tar.gz | tar xzf - <ins> # Patch to fix https://github.com/rust-lang/rust/issues/48967 cd $MUSL &&  curl "https://git.musl-libc.org/cgit/musl/patch/?id=610c5a8524c3d6cd3ac5a5f1231422e7648a3791" | patch -p1 &&  cd - </ins> fi cd $MUSL