Change:
Upstream issue: The version of used in the (and possibly the one too?) standard library has a bug where it will sometimes walk off the end of the segment containing the section and segfault. I've struggled to reproduce this except in some proprietary code, but the backtrace looks like: There's a proposed fix in the linked LLVM issue, which can possibly be patched into libunwind before building the musl target if taking an updated libunwind isn't possible.
I ran into this as well.
There doesn't seem to be any movement on the LLVM issue since I raised it, not even an assignee. Do you know if this is normal behaviour for them? Or if I've mis-raised the issue?
Running into this as well at OneSignal
I can reproduce this on latest stable on an open source project: On the second run of the final command, the background process will segfault.
After much investigation, I think there's two bugs here, one that LLVM's unwinder will consider unreadable memory as part of the section, and one that the rust compiler creates invalid (unusual?) unwind information. The crash is specifically happening when trying to find unwind information for the frame above (not but the "c-runtime" main. Since the frames above the C entry point are provided by the runtime (musl in this case), they are contained in / which, in rust's musl-targeting stdlib have been provided. The provided object files have no unwind information in them: This in and of itself isn't a huge problem, the code stops trying to unwind when it can't find unwind information for the next frame (), so it should stop after which would be fine. so long as we can safely fail to find the unwind information. The issue is that libunwind's logic for searching for FDE (frame description entry) for the parent frame is pretty all-encompassing and runs in many phases, and one of these crashes (sometimes) if there searched for address is not present in the unwind information). At a high level the process looks like () Look for a "compact unwinding table" Look for "DWARF unwind information" Look for ARM EHABI unwind info Look for dynamically created DWARF unwind entries For Rust x8664 musl binaries, the compiler has provided DWARF unwind information, so we fall into the second bullet (): No compact encoding hint, so the first section isn't interesting Search the section for an index for the frame - not present Check the local cache in case we've seen this frame before - we haven't Scan the whole of the section for the address of interest - CRASH! Digging further, we see that the logic for scanning the sector looks like (): Read a length field If it's 0 stop, we're done - rustc never seems to generate this "end-of-information" marker Skip over CIE entries For non-CIE entries work out if the address we're looking for is in the range of the entry, if so we've got our FDE Jump forward by the length of the CFI entry In each loop we check that the newly found CIE entry is in a reasonable place (between and ). If it's not, bail out for safety and fail the lookup. Unfortunately, as in the linked issue originally is incorrectly set too big (it's set to which is almost always wrong) so the unwind code will step right off the end of the segment and into one of: (LSDA) - This starts with a signature that is approximately , which is bigger than the error introduced by the miscalculation, so (after failing to parse the LSDA as a CFI entry) it jumps forward out of the range and bails out because the of the "reasonable location" check I mentioned above. The gap before (if the did not have a length that's a multiple of 32 and thus there's padding between the sections). In this case when it tries to read the next entry one of three things happens: The padding bytes happen to have a large value (when interpreted as a ) and we're in a similar position to the case where there's no padding, fail to parse the CFI entry, try to jump to the next one, realise we're jumping out of the allowed range and bail out for safety. The padding bytes are medium sized when interpreted as a , in this case, after the failed parse, the unwinder jumps far enough to be off the end of the loaded segment, but not far enough to be past the so the sanity check passes and the unwinder attempts to read the next CFI entry, reads a 32-bit number from an invalid address and... :boom: The padding bytes are small when interpreted as a , and the unwinder jumps into a random part of the LSDA and we're back in a situation that's functionally similar to when we're reading the padding bytes and the same three options are still applicable in the next lookup. So, what's the bug(s) here? LLVM is miscalculating the section length (the original issue) Rustc (or the linker?) doesn't include the marker at the end of Either of these would fix this issue, probably both should be done?
Ah, there's another possible route for the crash: If the LOAD segement is really big, the signature at the start of the LSDA might not be big enough to jump past leading to an invalid read and a :boom:
I've ran into this issue as well. What's the status of this?
I've been looking at a proper fix for this and neither of the two paths I proposed above are yeilding simple results: LLVM miscalculating the section length: The simple fix I proposed in the upstream issue allows the unwinder to read arbitrary parts of the LOAD segment before bailing, which might have unexpected effects A more complex fix needs to calculate the section length at runtime, which is tricky as sections don't exist at runtime... Neither the nor section contains a length field so we're not simply told the length Maybe read the executable off disk (look in ) and parse the section table? Might not be readable to the current user. Might not even exist (not sure if this is possible, can you create a linux executable from an in-memory image?) Add terminator entry to the end of the section This relies on the static linker's behaviour, which is a shame as allows you to bring your own linker and both seem inconsistent about whether they add terminator entries, sometimes they do, sometimes they don't This feels like something that rustc can't fix locally or rely on externally Another option would be for LLVM to skip walking the section if the lookup in the fails. It's unlikely to ever reveal anything new, and it seems that this section is just un-walkable with any kind of reliability.
Here are two tested workarounds, neither very pretty, but both work: to your Cargo config a build, extract the link script from the linker verbose output the lines: .ehframe : ONLYIFRO { KEEP ((.ehframe)) (.ehframe.) } ... .ehframe : ONLYIFRW { KEEP ((.ehframe)) (.ehframe.) } them to: .ehframe : ONLYIFRO { KEEP ((.ehframe)) (.ehframe.) LONG(0x0) } ... .ehframe : ONLYIFRW { KEEP ((.ehframe)) (.ehframe.) LONG(0x0) } ( is the CIE terminator) off this script somewhere in Cargo config (probably under the Musl target) an object containing just the terminator: 2, Add to your Cargo config (probably under the Musl target)
Hi! Is this still an issue? If so, is there perhaps a simpler way to reproduce this than the instructions linked above?
The setup that's needed is: A large LOAD segment (the one containing and ) Specifically the section needs to be deep into the segment How deep it has to be is roughly determined by the first of the section So something like the following should trigger the fault: This builds a huge section, which lives before the section so should lead to the crash. Hedging words because I can't reproduce the issue on my development machine (I run out of RAM and get OOM-killered).
Hmm, that doesn't seem to be sufficient (tested on the current stable and nightly). I get this output when building and running this on Arch Linux: I also get the same output when using to capture the backtrace.
I noticed the wrong ehSectionEnd calculation independently and filed
It looks like the gcc driver on Alpine adds the file to the link, which includes a terminator in . It seems that the rustc driver doesn't add something like this? (LLVM has a that has an .ehframe terminator, so maybe that's relevant to rustc.) FWIW, it looks like LLVM's LLD linker automatically adds a terminator to the end of .ehframe, even if one isn't present in the linker inputs, so that could be a workaround: I think libunwind should stop scanning .ehframe when it finds unwind info from .ehframehdr (i.e. the GNUEH_FRAME segment).
Rustc's musl-targeting link includes: I don't exactly know where the files come from (they're shipped with the compiler toolchain, e.g. mine are in ) but looking at the musl codebase, there's no terminator in : (Intriguingly there's also no in the musl codebase, suggesting Rustc gets these objects from somewhere else?)_ I wouldn't expect to see a or since the rustc musl target doesn't include a C++ runtime and musl libc doesn't define either of these objects for the C runtime either.
It looks like musl's crt1 is a C file, not assembly. crt/{S,r,}crt1.c When crtbegin and crtend were to LLVM (), there was some discussion about which project is responsible for which aspects of the CRT begin/end files. Apparently the status quo on Linux is that: the compiler (e.g. libgcc) provides crtbegin/crtend the libc provides crti/crte (and maybe crt1, too?) That is the case on my gLinux (i.e. Debian) system: /usr/lib/x8664-linux-gnu/crt[1ie].o all come from the libc6-dev:amd64 package /usr/lib/gcc/x8664-linux-gnu/9/{crtbegin.o,crtend.o} come from libgcc-9-dev:amd64 The .ehframe terminator is usually in crtend, which is part of libgcc (not glibc or musl). LLVM's compiler-rt also provides crtbegin/crtend files. The gcc and clang drivers link a crtend object even for C programs. On Alpine, the gcc driver links a libgcc crtend object. It looks like rustc typically invokes the driver, but when it's targeting musl, it enters a mode where it provides the start files explicitly, and in this mode, it doesn't include crtbegin/crtend. For Alpine, I see , , , and . For rustc/glibc, I don't see any of those flags, so the cc driver uses the default CRT begin/end/i/n/1 files. I see that rustc documents a lot of this at The unwinder needs the .ehframe terminator, because the .ehframehdr search table is optional, and because .ehframehdr has the start address of .ehframe, but not its size. ( / ). I don't think it's reasonable to open the ELF file indicated by dlpiname -- in principle, that duplicates the work done by the loader, and I can think of many specific things that wouldn't work. (e.g. the vdso, the executable w/glibc, Android's androiddlopen_ext and ability to map DSOs from zip files, unlinked files, etc).
Using self-contained mode is not generally recommended for serious work. If the compiler heuristics enable it by default, then it can be explicitly disabled with , then gcc will be used for linking without and will find all the necessary CRT objects by itself. Then it's important to make sure that gcc for the right target (musl in this case) is used by rustc.
If the compiler heuristics enable it by default Here's a FIXME for improving the heuristic for musl targets:
Thanks for that, very interesting. I think there are two separate use cases for musl-targeting builds: an environment where there's a full "native musl" toolchain (like on Alpine etc) where I think you're correct that we shouldn't use and should let the toolchain sort it out for us though I can't see an easy way to tell if a real toolchain is available a generic linux environment where the goal is fully static binaries (for easy of redistribution/embedding in thin container images/etc.) where there's likely not a native toolchain and rustc is forced to work out the answers for itself (as an aside no longer works as the "linker" for musl targets, since it doesn't support which is enabled in rustc as of the last release) I'm worrying about the latter case and I think from above the fix for that is: to add the terminator in builds as well as ones and include it in the target package alongside the (so rustup etc fetch it) the link command rustc emits for the musl target to include before Some side questions: Does the above sound sensible to people? Do we need to include any of too? (I think not, but I'm no expert :shrug:) Do other targets need similar treatment? musl?
In the latter case it would make sense to me for rustc to invoke an actual linker (///etc) since at the moment it goes via a compiler but has to tell the compiler to not do anything and just to pass the link args to the linker anyway. This is off topic for this discussion though.
If we need to improve the self-contained mode specifically, then I think we can just ship the begin/end objects and link to them like gcc does obsoleting the comment cited in That appears to be the simplest solution. If that doesn't work out (e.g. due to licensing), then the necessary parts can be to and linked as , as you suggested above.
Which / objects were you thinking? 's? LLVM's? If you're right and it's the compiler that ships the begin/end then maybe it's right for to have it's own (though given that the link might contain code compiled by random compilers (from scripts) as well as compiled by , which is "the compiler" for the sake of this discussion)?
's? LLVM's? I don't know, but looks like they should be compatible. At least on Ubuntu both gcc and clang link to gcc's objects. The gcc ones can be just copied by rustbuild if the license allows, the LLVM ones can be used if we need to build them by ourselves (they are a part of compiler-rt).
adds crtbegin/crtend objects to Rust distribution and links to them in self-contained mode as was suggested in , so it should fix this issue.
I still have this problem with clang's compiler-rt:

Discussion:
tool::Rustfmt, tool::Miri, tool::CargoMiri, <del> native::Lld </del> <ins> native::Lld, native::CrtBeginEnd </ins> ), Kind::Check | Kind::Clippy { .. } | Kind::Fix | Kind::Format => describe!( check::Std,

DependencyType::TargetSelfContained, ); } <ins> let crt_path = builder.ensure(native::CrtBeginEnd { target }); </ins> for &obj in &["crtbegin.o", "crtbeginS.o", "crtend.o", "crtendS.o"] { <del> let src = compiler_file(builder, builder.cc(target), target, obj); </del> <ins> let src = crt_path.join(obj); </ins> let target = libdir_self_contained.join(obj); builder.copy(&src, &target); target_deps.push((target, DependencyType::TargetSelfContained));

fs::write(&self.path, self.hash.as_deref().unwrap_or(b"")) } } <ins> #[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)] pub struct CrtBeginEnd { pub target: TargetSelection, } impl Step for CrtBeginEnd { type Output = PathBuf; fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> { run.path("src/llvm-project/compiler-rt/lib/crt") } fn make_run(run: RunConfig<'_>) { run.builder.ensure(CrtBeginEnd { target: run.target }); } /// Build crtbegin.o/crtend.o for musl target. fn run(self, builder: &Builder<'_>) -> Self::Output { let out_dir = builder.native_dir(self.target).join("crt"); if builder.config.dry_run { return out_dir; } let crtbegin_src = builder.src.join("src/llvm-project/compiler-rt/lib/crt/crtbegin.c"); let crtend_src = builder.src.join("src/llvm-project/compiler-rt/lib/crt/crtend.c"); if up_to_date(&crtbegin_src, &out_dir.join("crtbegin.o")) && up_to_date(&crtend_src, &out_dir.join("crtendS.o")) { return out_dir; } builder.info("Building crtbegin.o and crtend.o"); t!(fs::create_dir_all(&out_dir)); let mut cfg = cc::Build::new(); if let Some(ar) = builder.ar(self.target) { cfg.archiver(ar); } cfg.compiler(builder.cc(self.target)); cfg.cargo_metadata(false) .out_dir(&out_dir) .target(&self.target.triple) .host(&builder.config.build.triple) .warnings(false) .debug(false) .opt_level(3) .file(crtbegin_src) .file(crtend_src); // Those flags are defined in src/llvm-project/compiler-rt/lib/crt/CMakeLists.txt // Currently only consumer of those objects is musl, which use .init_array/.fini_array // instead of .ctors/.dtors cfg.flag("-std=c11") .define("CRT_HAS_INITFINI_ARRAY", None) .define("EH_USE_FRAME_REGISTRY", None); cfg.compile("crt"); t!(fs::copy(out_dir.join("crtbegin.o"), out_dir.join("crtbeginS.o"))); t!(fs::copy(out_dir.join("crtend.o"), out_dir.join("crtendS.o"))); out_dir } } </ins>

# riscv targets currently do not need a C compiler, as compiler_builtins # doesn't currently have it enabled, and the riscv gcc compiler is not # installed. <del> ENV CC_mipsel_unknown_linux_musl=mipsel-openwrt-linux-gcc  </del> <ins> ENV CFLAGS_armv5te_unknown_linux_musleabi="-march=armv5te -marm -mfloat-abi=soft"  CFLAGS_arm_unknown_linux_musleabi="-march=armv6 -marm"  CFLAGS_arm_unknown_linux_musleabihf="-march=armv6 -marm -mfpu=vfp"  CFLAGS_armv7_unknown_linux_musleabihf="-march=armv7-a"  CC_mipsel_unknown_linux_musl=mipsel-openwrt-linux-gcc  </ins> CC_mips_unknown_linux_musl=mips-openwrt-linux-gcc  CC_mips64el_unknown_linux_muslabi64=mips64el-linux-gnuabi64-gcc  CC_mips64_unknown_linux_muslabi64=mips64-linux-gnuabi64-gcc 

<ins> # only-linux # ignore-32bit -include ../tools.mk all: $(RUSTC) eh_frame-terminator.rs $(call RUN,eh_frame-terminator) | $(CGREP) '1122334455667788' objdump --dwarf=frames $(TMPDIR)/eh_frame-terminator | $(CGREP) 'ZERO terminator' </ins>

<ins> // run-pass #![feature(backtrace)] #[derive(Clone, Copy)] struct Foo { array: [u64; 10240], } impl Foo { const fn new() -> Self { Self { array: [0x1122_3344_5566_7788; 10240] } } } static BAR: [Foo; 10240] = [Foo::new(); 10240]; fn main() { let bt = std::backtrace::Backtrace::force_capture(); println!("Hello, world! {:?}", bt); println!("{:x}", BAR[0].array[0]); } </ins>