Change:
opt_nosave llvm-release-debuginfo 0 "build LLVM with debugger metadata" opt_nosave debuginfo 0 "build with debugger metadata" opt_nosave debuginfo-lines 0 "build with line number debugger metadata" <ins> opt_nosave debuginfo-only-std 0 "build only libstd with debugging information" </ins> opt_nosave debug-jemalloc 0 "build jemalloc with --enable-debug --enable-fill" valopt localstatedir "/var/lib" "local state directory"

Discussion:
When I have any compilation error(s) there is a rather long delay (around 14s on my machine) before rustc finally exits. Successful compilation is very fast. I am on Ubuntu 16.04 in VMware Workstation 12.5 for Windows virtual machine. Steps to reproduce: :
If it helps, that same version of rustc on Windows 10 works OK for me, there is no noticeable delay after compilation error reporting.
Actually I failed to report that I was using Ubuntu 16.04 virtual machine on VMware Workstation 12 (most recent version) on native Windows 10. I just have tested for this issue on Ubuntu 16.04 on bare metal, and rustc worked fine. The issue seems to be related to virtualization somehow.
Note that the compiler continues after it hits the first error, so the compiler could possibly have been doing real compilation work (looking for more errors), so perhaps that's what's happening here?
Have you considered profiling rustc to see where it is spending most of those 13 seconds?
here is what I've gathered from 'perf' the results from my Ubuntu 16.04 virtual machine running on VMware Workstation (only few lines from the top for brevity): Clearly the problem is function. I will also put here the results from bare metal for comparison.
results for Ubuntu 16.04 running bare metal:
I have just realized that on my virtual machine I had environment variable RUSTBACKTRACE=1, whereas on my bare-metal I had not. Indeed when setting RUSTBACKTRACE=1 on both machines the results are more or less the same (there is a very long delay before rustc exits). So the problem is not with virtualization after all. Definitely the problem lays in backtrace code somewhere. I started to notice this issue somewhere around a week ago, as I am updating rust nightly very often (daily).
Possibly duplicate of
14 beta exhibits this issue. If it goes stable it will be a significant headache/puzzle for everyone who leaves RUSTBACKTRACE set. cc seems to be a duplicate issue. In the same 10/25 nightly that introduced this, the release hello world executable size on grew from ~650k to ~2MB, not sure if that may be related.

This is negatively impacting the alternate playground, which always sets RUST_BACKTRACE, and is now present in released 1.14.
Hey, this has made it's way to stable and it is a rather serious problem I think. perhaps it should have some labels/assignee applied? We face it in IntelliJ Rust: We run cargo commands with by default, so this leaks to the compiler even though we are interested only in the bactraces for the user's code. If it is useful, here's a flamegraph of the hard working rustc:
sounds reasonable to me. I'll nominate for discussion in libs triage as I believe this is basically entirely about libbacktrace
What makes libbacktrace so slow, anyway? I've definitely noticed the very long lags in the past.
Judging by and the , I'm going to guess that something is being eagerly allocated when it doesn't need to be.
I've tried to track this down, but been unable to reproduce with nightly on macOS 10.12.2 or stage1 and stage2 compilers on Ubuntu 16.04 (on ). While I do notice a slight increase (~100-400 ms) when either the compiler ICEs or a crate panics, I sort of expect that since more work is being done. However, I can reproduce with nightly (rustc 1.16.0-nightly ( 2017-01-03)) on Ubuntu 16.04, which sort of confuses me. I'm not sure what the difference between those and stage1/stage2 builds on the same machine is. To summarize: This makes benchmarking and testing changes difficult/impossible, since I can't reproduce with a locally built compiler. I'm not really sure what can be said here. I also confirmed that this is not due to the wrapper script that rustbuild introduces. Exact measurements below. is empty in all cases. macOS 10.12.2, rustc 1.16.0-nightly ( 2017-01-03). Ubuntu 16.04, rustc 1.16.0-nightly ( 2017-01-03) -- same as on macOS Ubuntu 16.04, Stage 1 Compiler () at . Ubuntu 16.04, Stage 2 Compiler () at .

It may also be of interested that utilizing strace, it seems that the stage1/stage2 compilers benchmarked above issue hundreds of mmap calls, approximately page-by-page () in case it matters, while the nightly rustc issues thousands, from what I can tell. It almost feels like it's loading the libraries in , possibly. It seems to iterate over those (with open calls), but that might not be during the panic. Hopefully all of this can help narrow this issue down. I'd be happy to collect any other statistics that could be helpful.
interesting! I wonder if this means it's all related to how we're building the nightlies. The main differences I can think of there are: We're compiling with a much older gcc most likely (4.7) We're compiling against a much older glibc (2.5 IIRC) LLVM assertions are enabled (but just for Rust code) Debuginfo is enabled in the compiler Now that I list these out, did you enable debuginfo in the compiler you produced locally? IIRC that was a historical problem and I don't think we ever fixed that.
is empty in all cases. I see delay even if is called with non existent file: .
This occurs with Rust 1.14 stable on Linux for me.
oh I should say releases*, not just nightlies. They're all built the same way modulo llvm-assertions (which we can now rule out if it's a problem on stable)
FWIW, here's the reproduction steps I have:
For some reason, this reminds me of Most likely unrelated, but symptoms are somewhat similar.
This reproduces when I use a debuginfo+debug assertions rustc I built, but does not reproduce when I use a non-DI rustc. 70% of CPU time is spent in .
Now that I list these out, did you enable debuginfo in the compiler you produced locally? IIRC that was a historical problem and I don't think we ever fixed that. Yes. I have debug info and optimizations enabled locally ().
Okay, I've done some more debugging/profiling locally. The results seem to imply that nightly builds produce about ~6 million more calls to in . Why this is, I don't know, and I don't really know how to find out either.

I've attempted to run locally, but using the results from that also doesn't appear to show a slowdown with . What system are you on? I'm trying to figure out why you see a slow down with stage 1 (locally built) rustc with debug info and I don't. Nightly callgrind results: Master callgrind results:
The libs team discussed this issue during triage today and our conclusions were: So far the assumption is that the addition of debuginfo in the compiler is the cause of this problem, further bullets are predicated on this assumption. One possibility is to only ship debuginfo-enabled MSVC compilers (the initial motivation for the change) Another possibility is to only ship debuginfo-enabled standard libraries, not compilers. We'd want to investigate these routes to see which is the most plausible.
The rustc isn't debuginfo-enabled anyway, because nobody bothered to ship the PDB files with it yet. The only debuginfo is the codeview in the libraries which affects binaries created by rustc. Because has no downsides to debuginfo in the libraries, if you're going to remove it from other targets please retain it for the msvc targets. Also, can we finally ship the PDB files already?
We could also add a way to libstd to disable backtrace printing - the backtrace is only read on ICEs.
So far the assumption is that the addition of debuginfo in the compiler is the cause of this problem, further bullets are predicated on this assumption. But what is the underlying cause? I don't really know what the amount of debug info is, but 10 seconds with 100% on the modern CPU seems like a huge amount of work. Or is processing debug info that expensive?
I think this happens for all normal compiler errors, not just ICEs, right? A normal compiler error will also use a to abort the compiler thread, and that will transitively print a backtrace. Right now the backtrace is suppressed, though, so it doesn't show up on stdout. (IIRC) I don't believe investigation has been done to the underlying cause. I agree it's quite alarming!
Sure enough. A "normal" compiler exit causes a . A hook in libstd could suppress backtrace generation in that case.
So the actual bug here has been by and has an which I doubt will ever see activity.

The gist of the bug is: libbacktrace has a custom allocator, backed by This custom allocator maintains a linked list of free memory When allocating, the linked list is checked first We're then hitting a pathological case where we're always looking at the whole linked list to realize we need to call anyway. These allocations all seem to come from , and presumably with debuginfo there's a lot more function entries. We have a few ways to solve this: Stop generating debuginfo for the compiler, and pray that it has "fewer function entries" to run much more quickly. Locally this took runtime from 0.013s without a backtrace to 0.072s with a backtrace. (before it took 6s with a backtrace) Delete and force the fallback to and that libbacktrace uses. claims that this is still slower than not generating a backtrace (e.g. 0.1s to 1.2s). Could be a good stopgap though. (note the concern about not always being right I believe is not relevant for our own use case). I also personally measured an increase from 0.013s to 0.097s locally when taking this strategy. Change to allocate less, unsure how this would be done. Removing the apparently didn't help, though, so I suspect the problem is much deeper. Move to a different strategy entirely, like , which may be much more performant. Summarizing the timings with the program where doesn't exist, I got: Thoughts on which route we should take? I'm personally leaning towards "no debuginfo in the compiler"
I'm also going to tag this as a stable-to-stable regression as this essentially regressed from 1.13.0
I would prefer that we go with the malloc allocation strategy, since people working on rustc itself want debug info for the compiler (and we might want it in nightly for easier debugging of ICEs) and the performance tradeoff is minimal. Exploring gimli and other non-libbacktrace options can happen after February 2nd, I think, when we delete Makefiles and can start using crates off .
oh sorry landed more quickly than I thought it would! To clarify and such still enable debuginfo in the compiler. I only changed releases. That means that local development should still work out just fine.

I personally prefer to not change libbacktrace much to make future vendoring efforts easier, but that's mostly just me.
What I really want is for us to adopt some kind of library that works across platforms, so that mac can enjoy line numbers too! (I'm not sure of the status on windows here...)