Change:
# way to produce "super compatible" binaries. # # See: https://github.com/rust-lang/rust/issues/34978 <del> ENV CFLAGS_x86_64_unknown_linux_musl=-Wa,-mrelax-relocations=no </del> <ins> # And: https://github.com/rust-lang/rust/issues/59411 ENV CFLAGS_x86_64_unknown_linux_musl="-Wa,-mrelax-relocations=no -Wa,--compress-debug-sections=none  -Wl,--compress-debug-sections=none" </ins> ENV HOSTS=x86_64-unknown-linux-musl  CC_x86_64_unknown_linux_musl=x86_64-linux-musl-gcc 

Discussion:
The last successful build was with the Rust nightly released on 2019-03-14 (): https://travis- The first failing build was with the Rust nightly released on 2019-03-15 (): https://travis- There weren't any changes on my end. (Last commit to ripgrep master was 2019-03-03.) Here's the pertinent error: Unfortunately, I can't seem to reproduce this on my local system. One possible difference between my local system (Archlinux) and the Ubuntu CI on Travis is that Ubuntu might be using an older version of via the package. I have locally. According to the logs in the link above, CI is using (for both the last successful build and the first failing build).
Exactly the same problem with exactly the same setup here: https://travis-
Works fine with : https://travis-
I get the same error message for on a new hello-world project (as generated by cargo new). This is on debian jessie. (Don't ask. Need to build for old servers...) Reproducible locally, so if I can help debugging this, please tell me where to look.
It looks like it depends on the version of binutils: With 2.28-5 from debian/stretch I can build the binary successfully, with 2.25-5+deb8u1 from debian/jessie I get the mentioned error.
I was afraid something like that can happen when tinkering around but I was able to compile on Alpine (native musl host) and Ubuntu (with musl target). I'll see if I can reproduce it. sidenote: Ubuntu 14.04 becomes EOL next month.
I was able to reproduce it in Ubuntu 14.04 container and verified that fixes it.
Thank you!!
Sorry for breaking it, when I was testing dozen of different distros I didn't even think of running it on distro with so old binutils.
Yeah unfortunately Ubuntu 14.04 is the default Linux VM on Travis. :-(
triage: P-high. driveby: approving PR .
triage: assigning self since I cannot assign
Did this make it into a recent nightly? It looks like the nightly is still tripping over this bug: https://travis- (I'm seeing about switching to xenial.)

Weird, I reproduced it locally and tested it before opening PR. I'll look into it today (CEST time).
I can easily reproduce it installing toolchain with but when I install freshly build toolchains using Dockerfiles I cannot reproduce it no matter how hard I try. I'll try few ideas using builds from Rust and report here in a few days.
The issue only manifests when building with and is set to something, building with without environment variables produces working libraries. Technical informations: There are 2 : one from and another one from . Their sizes are interesting: without (no errors): : 85,2 KiB : 85,3 KiB with (link errors): : 85,2 KiB : 397,4 KiB I'll try upgrading musl but I'm not convinced it will help. After some digging I found this line to cause the problem: cc can I ask you for some directions where I should look or how long those old platforms should be supported?
we don't really have a hard-and-fast policy, but rather "we keep things working as long as it's reasonable". If it's not reasonable to keep supporting something then we can explicitly evaluate at that time if it feels right to drop.
I've tried pushing harder when building C toolchain, upgrading musl to 1.1.21, GCC to 6.5.0 and even downgrading binutils to 2.25.1 (long time ago backward compatibility broke when binutils were upgraded to 2.26). Without success. Disabling this makes libraries compatible with older buintuils: Noticed one more thing, before there was single sized 526,3 KiB. After that PR there are two with sizes: 85,2 KiB and 397,4 KiB. The bigger one causes linking issues.
I'm pretty lost here myself so I don't think I can really provide much guidance. I don't even know if this is still an open or closed issue, or if we're looking to go beyond what's already in-tree. Could a summary be redone of what's currently happening?
It probably should be reopened. The short description is that musl builds are failing on Ubuntu Trusty, which is the default Linux environment on Travis.
It still fails with errors from the first comment.

First time I tried to fix it with it worked on my machine but nightly builds were still broken. Later I discovered it happens only on and not on builds and narrowed it to (this time reproduced the error with locally built compiler). I've tried forcing for building GCC and musl, upgrading musl to 1.1.21, GCC to 6.5.0 and even downgrading binutils to 2.25.1 (latest release that should be compatible with older binutils) but nothing helped. Before (introduction of musl host toolchain) there was single with size 526,3 KiB. After there are two in coming from different parts of Rust. Their size differs whether debuginfo is enabled or not: no debuginfo (aka build, no errors): from stage1-rustc: 85,2 KiB from stage1-std: 85,3 KiBdebuginfo (aka build, link errors): from stage1-rustc: 85,2 KiB from stage1-std: 397,4 KiB - this one is causing issues
Ok so instead of
er, Ok instead of golfing versions for various tools, it looks like a "hello world" fails in 14.04 and succeeds in 16.04 (ubuntu). The main difference here is binutils that we're interested in, and it jumped from 2.19 to 2.23 in those two revisions. I wasn't able to quickly test any other versions of binutils, but turns up: I suspect that's what's happening here where binutils in Ubuntu 14.04 didn't understand compressed debug sections but in Ubuntu 16.04 it does. Presumably at some point in our toolchain we're producing compressed debug sections either by accident or as part of some build script. My guess is that either the upgraded version of gcc turns it on by default or musl does or something like that. We probably need to figure out where needs to be passed through (or maybe it's just doing that through )
It also looks like may be the gcc flag to pass through. can you test passing that to the musl toolchain build and see if it fixes the issue?
:confused: Ubuntu 14.04 uses and 16.04 uses . Musl is ELF just like glibc, PE/COFF is for MinGW.

Ah oops I was looking at glibc versions rather than binutils versions. That being said I think it's ironically the same culprit:
As I said earlier I tried switching to use binutils 2.25.1 and I'm quite confident it was picked because I've got errors saying is invaild. I'll try tomorrow if Rust didn't accidentally pick binutils from system instead of musl-toolchain but it seems unlikely. You can ping me on Discord BTW.
this isn't an aspect of binutils or such to build the toolchain, it has to do with the binaries we're producing using features that aren't supported by older toolchains. It doesn't matter what we build the musl toolchain with so much as how we build it. Using nightly-2019-03-15 compiling hello world works, using 2019-03-16 fails. Inspecting gets us: This seems highly likely to be related to compression of debuginfo at least, if not more aspects of the binaries.
That looks clearer. I assumed it would use binutils from musl-toolchain because we point Rust to use GCC from it: I can try to point it to musl-toolchain binutils 2.25.1 or disable compression, whatever you like and open PR tomorrow (or reply here if it doesn't help).
This has slipped to beta now I believe.
calls so my downgrade to binutils 2.25.1 should have fixed this. Probably Rust's build systems at some point calls to build musl libs. Anyway disabling debug sections compression fixed the issue: thank you for help!