Change:
<del> Subproject commit cd87134ab77e6bacb2128137065b328b9c35e0e5 </del> <ins> Subproject commit a6f4c1bb07d58df5956d2c49be68547143f77b18 </ins>

Discussion:
I got a segfault / LLVM assertion failure while compiling a target with ThinLTO enabled. This is the same failure as in , albeit with a different root cause. In my case, I was dealing with a global named . Note the presence of two extensions. This means that this was a local which was promoted to a global symbol twice. The of doesn't handle this case correctly; it strips off both extensions: The fix is actually trivial; just have to change to so that it only removes the last_ extension. Leaving this issue open to track the fix being cherry-picked.
Filed a bug .
This might still be a rustc bug, since apparently a double-promotion is unexpected. Working on a minimal repro now.
is this something that we could/should be pulling in the ICE-breakers on?
triage: P-high. Assigning to and removing I-nominated label.
Possibly, but I’ll need to put a usable reproducer somewhere. That tends to be more difficult when the target is Fuchsia. I’ve had limited success doing all the minimizing myself. I’ll see if I can reproduce on other targets, and upload a reproducer if so.
So, from the output of , what appears to be happening is that we run the same bitcode through the ThinLTO optimization pipeline twice. Once when outputting an rlib, and once when linking to that rlib. More specifically: rlib crate ( in this case) gets compiled with ThinLTO and optimizations enabled. We run the bitcode through the ThinLTO optimization pipeline. results in a lot of files that look like binary crate is compiled which depends on the rlib. It is also compiled with ThinLTO and optimizations. again results in a lot of files that look like (except and everything after it are missing, because it crashes) the file is the output of the ThinLTO pipeline from the previous step The offending symbol is promoted to a global once in the first optimization run and again in the second, which accounts for the two suffixes on it. The crash occurs because it's not expected for a symbol to be promoted twice. I can't think of a reason why we would run the same pipeline twice on the same bitcode. Is there one?
cc
are you compiling with , or just normal ?

The former will run some bitcode through thinlto twice, but the latter shouldn't. If the bug happens by running bitcode through twice, that's like where it's coming from?
Ahh you were right, it has to do with the flags! For the rlib I wasn't passing , but for the executable I was. As soon as I started passing for both, the problem went away and we only sent the code through the pipeline once. (EDIT: I was using in both) Apparently we've been doing this for a very long time in Fuchsia without problems. So is it expected that we should be passing to both?
Oh weird, so I didn't expect that to fix anything nor did I think that the compiler does what it does! Looks like it boils down to which explains: The final executable means that all bitcode will run through ThinLTO. If the rlib bitcode was previously run through ThinLTO, then it would be run through twice. That match statement, however, means that if you () then we skip the thinlto step in rlibs assuming you'll do it at the end. So it looks like, yes, you'll want to pass to all compilations, not just the final one. And that should do the trick! I would still consider this an LLVM bug though because it should in theory work both ways...
Agreed on this still being a bug. Also, as it turns out, the old way of doing it is faster for our build! Presumably since we're doing more work up front. For progeny, I'm posting the data here. Here are a couple runs before adding to the rlib builds: Here are a couple runs after: This is on a 72-core machine building a pretty large graph of Rust crates (many executables).
Interesting, not the results I would have expected! If you have a lot of executables though and you're running ThinLTO on all of them then you're likely basically optimizing the same code over and over. Before each crate was individually pretty heavily optimized, doing those optimizations only once instead of once for each executable. Afterwards, however, ThinLTO-per-crate doesn't happen and only happens at the end. This probably produces roughly equivalent code in terms of speed, but each executable has to run more optimizations.

Right, so by doing more of that work upfront in the rlibs, we save time optimizing the code for each executable. I think there's still a more efficient way of doing all this. For one, we could disable the actual object file output for rlibs. Also, when compiling the executable, the whole ThinLTO pipeline is run again on the individual crate bitcode before building summaries and distributing the codegen to workers. If there were a way to disable that and tell the compiler to assume that rlib bitcode has already been optimized, we could save some time here. Perhaps we could make that automatic with a flag that emits the summary with the rlib, and detect that it's already there in the executable compile. Of course I'm talking about how to optimize a completely-unintended way of using Rust's ThinLTO pipeline. But since it happens to be faster, it might be worth experimenting with. :)
Heh perhaps! In theory optimization pipelines are quite flexible and we can basically design whatever we want, it's all just a matter of implementation and measurement at some point.
Teresa Johnson left some on the downside of doing it this way. To summarize, it can affect what gets inlined (for better or for worse), and it disables whole program devirtualization. We'd have to do some more tinkering with the pipeline and measurement to avoid those issues.
Also, a fix was landed upstream:
Oh something I forgot to reply to earlier This is one of the subjects on actually, and yeah would help quite a bit in your case since all the codegen happening in each crate isn't actually used. A good point! It sounds like this only happens though if we compile with the equivalent of which I don't think we're currently doing? If we do that though this is something good to keep in mind! Yay!
Ah that's all very relevant, thanks. Subscribed :) I opened to track this. Not expecting it to happen anytime soon, but it seems good to have a tracking issue for it.

Some code went from compiling in <10 seconds to (hours) in between commits .. (unfortunately I don't have a smaller range right now). The regression only happens when ThinLTO is enabled. Full LTO doesn't have the slowdown. Here's the which is causing the regression. The function has about 20 points, which I suspect might be related. It doesn't seem to have anything to do with inlining the awaited function (I replaced it with a panic and linking was still slow). ~Here's the LLVM IR for the fast and slow cases (built with the above commits of rustc, respectively): ~ EDIT: See for a reproducer cc
. is the range and the following is the list of bors commits in that range: Auto merge of - RalfJung:miri, r=alexcrichton Auto merge of - Centril:rollup-v843h4a, r=Centril Auto merge of - Xanewok:update-rls, r=Xanewok Auto merge of - ehuss:update-cargo-books, r=alexcrichton Auto merge of - estebank:trait-impl-lt-mismatch, r=nikomatsakis Auto merge of - tmandry:rollup-g20uvkh, r=tmandry Auto merge of - matthiaskrgr:submoduleupd, r=Manishearth Auto merge of - eddyb:eval-always-considered-harmful, r=michaelwoerister Auto merge of - michaelwoerister:fix-issue-, r=alexcrichton Auto merge of - Centril:rollup-qrgwnt6, r=Centril Auto merge of - Centril:rollup-9i8ev23, r=Centril Auto merge of - pietroalbini:scriptify-ci-config, r=alexcrichton Auto merge of - estebank:variants, r=petrochenkov Auto merge of - matthewjasper:stabilize-const-constructor, r=Centril Auto merge of - Centril:rollup-y6b2qbf, r=Centril Auto merge of - Centril:rollup-bzlo74f, r=Centril Auto merge of - eddyb:spanned-inferred-outlives,

r=nikomatsakis Auto merge of - pnkfelix:issue--trait-based-structural-match, r=matthewjasper Auto merge of - estebank:point-at-assoc-type, r=nikomatsakis Auto merge of - flip1995:clippyup, r=Manishearth Auto merge of - hermitcore:rusty-hermit, r=alexcrichton Auto merge of - Centril:rollup-28jtjfc, r=Centril Auto merge of - RalfJung:miri, r=oli-obk Auto merge of - eddyb:promo-sanity, r=oli-obk Auto merge of - JohnTitor:rollup-mr6crka, r=JohnTitor Auto merge of - Centril:rollup-arlxgch, r=Centril Auto merge of - Manishearth:clippyup, r=Manishearth Auto merge of - Centril:rollup-v40xke9, r=Centril Auto merge of - Centril:rollup-upfsvrb, r=Centril Auto merge of - mati865:msys2-bug, r=pietroalbini Auto merge of - Mark-Simulacrum:rustc-dev-split, r=pietroalbini Auto merge of - Centril:rollup-0zth66f, r=Centril Auto merge of - bovinebuddha:objectsafefordispatch, r=nikomatsakis Auto merge of - JohnTitor:rollup-fkcr85k, r=JohnTitor Auto merge of - lzutao:clippy-up, r=Manishearth Auto merge of - popzxc:refactor-libtest, r=wesleywiser Auto merge of - alexcrichton:remove-emscripten-backend, r=Mark-Simulacrum Auto merge of - Centril:rollup-00glhmb, r=Centril
I had to double check to be sure, but the first bad commit was , "Update hashbrown to 0.6.2" The only reason I can come up with for this is that the change in hashbrown caused some IR to be emitted in a different order, and this triggered a bug somewhere, possibly LLVM, but before invoking the linker.

Cc
I finally have a reproducer that works with (the one built in the rust tree). I reproduced with the following line: (I've also used the and in the longest runs I've done, but pretty sure those flags aren't necessary.)
Reduced:
For future reference, here's how I got the above reproducer. Note that this is for a hang, but could easily be adapted for crashing as well. I had already minimized the rust source code manually, which allowed me to identify the exact function that was causing the hang. It turned out that pulling out a self-contained example was really difficult or impossible; this bug seemed to depend on a certain set of things being in a given codegen unit, for example. If I had some code that depended on items Foo, Bar, Baz, then pulled those dependencies and the offending function into their own file, the problem suddenly went away. I had already run on rustc and produced a flamegraph to get a basic idea of where time was being spent. Pro-tip: was the only mode that gave me good backtrace info for both rustc and LLVM. I could see it was spending a significant amount of time in LLVM. The page of the rustc guide was very helpful here. Note that the following steps are easier if you can set . In my case, this made the issue go away, so I couldn't. rustc with until it hung that for each codegen unit there were a number of files (N) that looked like ( is the codegen unit, is the stage bit which varies) for one codegen unit, 10, which had fewer than N files. This is the one that was hanging. sorting all of these files by creation timestamp, it's easy to see which step is missing. In this case, the last file in codegen unit 10 was . The stage after was . Okay, so the hang is occurring in the pass manager. That matches what I was seeing in perf. we have the input bitcode before the pass manager runs. But that isn't enough to reproduce the failure outside of rustc; rustc configures the pass manager with a specific set of passes, each with their own configuration. Reproducing that configuration in a tool like proved very difficult.

I did instead was to use rustc's option to print the LLVM IR for the function for each pass, until the compiler hung. Now I had the LLVM IR for the function right before the pass that was hanging (instcombine). LLVM IR for the function printed by is not self-contained. It references other things. This is where the bitcode files above come in handy. I obtained the LLVM IR from the bitcode file above using , found the IR for the offending function, and replaced it with the IR output by . patching IR like this doesn't quite work. The main problem is debuginfo; the debuginfo metadata (looks like ) references global tables that aren't printed by .[1] What I did was run a regex over the newly patched-in IR to remove these debuginfo attributes. The vim-flavored regex was something like . Also, I had to comment out every line that contained (). , run that ll file through with the pass that's hanging. In my case, . With any luck, will accept the IR and hang. If you get errors you can also try with , though this never helped me. [1]: Disabling debuginfo doesn't quite work, since you're probably depending on libraries (like libstd) that were compiled with debuginfo, and their code has been inlined into yours at this point. Congratulations! You now have a reproducer, but it may be huge. I used both and (in that order) to reduce the LLVM IR down to a manageable size. There are on how to do this. 's interface is simpler, but it couldn't handle all the function attributes I had in my IR (which bugpoint kindly stripped out). For minimizing hangs, what you want to do is use with a threshold that's big enough to prevent false positives. In my case, the offending IR would hang forever, but a different version of the same IR would optimize in 0.2s. I used 4 seconds as my argument to . has an exit code of 124 when it times out. This is your "interesting" case. If the exit code is nonzero but not 124, that means the minimizer produced invalid input; this is not interesting. Here's the script I used with bugpoint: The bugpoint run took around 9 hours in my case.

llvm-reduce was much shorter, since the case was already minimized. One final annoyance is that these two tools expect inverted exit codes from your interesting-ness script (I had to swap and above when running with llvm-reduce). They both check that your input is considered "interesting," so you'll know right away if you need to switch it up.
filed an issue upstream:
Patch has landed: