Change:
Some code went from compiling in <10 seconds to (hours) in between commits .. (unfortunately I don't have a smaller range right now). The regression only happens when ThinLTO is enabled. Full LTO doesn't have the slowdown. Here's the which is causing the regression. The function has about 20 points, which I suspect might be related. It doesn't seem to have anything to do with inlining the awaited function (I replaced it with a panic and linking was still slow). ~Here's the LLVM IR for the fast and slow cases (built with the above commits of rustc, respectively): ~ EDIT: See for a reproducer cc
... is the range and the following is the list of bors commits in that range: Auto merge of - RalfJung:miri, r=alexcrichton Auto merge of - Centril:rollup-v843h4a, r=Centril Auto merge of - Xanewok:update-rls, r=Xanewok Auto merge of - ehuss:update-cargo-books, r=alexcrichton Auto merge of - estebank:trait-impl-lt-mismatch, r=nikomatsakis Auto merge of - tmandry:rollup-g20uvkh, r=tmandry Auto merge of - matthiaskrgr:submoduleupd, r=Manishearth Auto merge of - eddyb:eval-always-considered-harmful, r=michaelwoerister Auto merge of - michaelwoerister:fix-issue-, r=alexcrichton Auto merge of - Centril:rollup-qrgwnt6, r=Centril Auto merge of - Centril:rollup-9i8ev23, r=Centril Auto merge of - pietroalbini:scriptify-ci-config, r=alexcrichton Auto merge of - estebank:variants, r=petrochenkov Auto merge of - matthewjasper:stabilize-const-constructor, r=Centril Auto merge of - Centril:rollup-y6b2qbf, r=Centril Auto merge of - Centril:rollup-bzlo74f, r=Centril Auto merge of - eddyb:spanned-inferred-outlives, r=nikomatsakis Auto merge of - pnkfelix:issue--trait-based-structural-match, r=matthewjasper Auto merge of - estebank:point-at-assoc-type, r=nikomatsakis Auto merge of - flip1995:clippyup, r=Manishearth Auto merge of - hermitcore:rusty-hermit, r=alexcrichton Auto merge of - Centril:rollup-28jtjfc, r=Centril Auto merge of - RalfJung:miri, r=oli-obk Auto merge of - eddyb:promo-sanity, r=oli-obk Auto merge of - JohnTitor:rollup-mr6crka, r=JohnTitor Auto merge of - Centril:rollup-arlxgch, r=Centril Auto merge of - Manishearth:clippyup, r=Manishearth Auto merge of - Centril:rollup-v40xke9, r=Centril Auto merge of - Centril:rollup-upfsvrb, r=Centril Auto merge of - mati865:msys2-bug, r=pietroalbini Auto merge of - Mark-Simulacrum:rustc-dev-split, r=pietroalbini Auto merge of - Centril:rollup-0zth66f, r=Centril Auto merge of - bovinebuddha:objectsafefordispatch, r=nikomatsakis Auto merge of - JohnTitor:rollup-fkcr85k, r=JohnTitor Auto merge of - lzutao:clippy-up, r=Manishearth Auto merge of - popzxc:refactor-libtest, r=wesleywiser Auto merge of - alexcrichton:remove-emscripten-backend, r=Mark-Simulacrum Auto merge of - Centril:rollup-00glhmb, r=Centril
I had to double check to be sure, but the first bad commit was , "Update hashbrown to 0.6.2" The only reason I can come up with for this is that the change in hashbrown caused some IR to be emitted in a different order, and this triggered a bug somewhere, possibly LLVM, but before invoking the linker.
Cc
I finally have a reproducer that works with (the one built in the rust tree). I reproduced with the following line: (I've also used the and in the longest runs I've done, but pretty sure those flags aren't necessary.)
Reduced:
For future reference, here's how I got the above reproducer. Note that this is for a hang, but could easily be adapted for crashing as well. I had already minimized the rust source code manually, which allowed me to identify the exact function that was causing the hang. It turned out that pulling out a self-contained example was really difficult or impossible; this bug seemed to depend on a certain set of things being in a given codegen unit, for example. If I had some code that depended on items Foo, Bar, Baz, then pulled those dependencies and the offending function into their own file, the problem suddenly went away. I had already run on rustc and produced a flamegraph to get a basic idea of where time was being spent. Pro-tip: was the only mode that gave me good backtrace info for both rustc and LLVM. I could see it was spending a significant amount of time in LLVM. The page of the rustc guide was very helpful here. Note that the following steps are easier if you can set . In my case, this made the issue go away, so I couldn't. rustc with until it hung that for each codegen unit there were a number of files (N) that looked like ( is the codegen unit, is the stage bit which varies) for one codegen unit, 10, which had fewer than N files. This is the one that was hanging. sorting all of these files by creation timestamp, it's easy to see which step is missing. In this case, the last file in codegen unit 10 was . The stage after was . Okay, so the hang is occurring in the pass manager. That matches what I was seeing in perf. we have the input bitcode before the pass manager runs. But that isn't enough to reproduce the failure outside of rustc; rustc configures the pass manager with a specific set of passes, each with their own configuration. Reproducing that configuration in a tool like proved very difficult. I did instead was to use rustc's option to print the LLVM IR for the function for each pass, until the compiler hung. Now I had the LLVM IR for the function right before the pass that was hanging (instcombine). LLVM IR for the function printed by is not self-contained. It references other things. This is where the bitcode files above come in handy. I obtained the LLVM IR from the bitcode file above using , found the IR for the offending function, and replaced it with the IR output by . patching IR like this doesn't quite work. The main problem is debuginfo; the debuginfo metadata (looks like ) references global tables that aren't printed by .[1] What I did was run a regex over the newly patched-in IR to remove these debuginfo attributes. The vim-flavored regex was something like . Also, I had to comment out every line that contained (). , run that ll file through with the pass that's hanging. In my case, . With any luck, will accept the IR and hang. If you get errors you can also try with , though this never helped me. [1]: Disabling debuginfo doesn't quite work, since you're probably depending on libraries (like libstd) that were compiled with debuginfo, and their code has been inlined into yours at this point. Congratulations! You now have a reproducer, but it may be huge. I used both and (in that order) to reduce the LLVM IR down to a manageable size. There are on how to do this. 's interface is simpler, but it couldn't handle all the function attributes I had in my IR (which bugpoint kindly stripped out). For minimizing hangs, what you want to do is use with a threshold that's big enough to prevent false positives. In my case, the offending IR would hang forever, but a different version of the same IR would optimize in 0.2s. I used 4 seconds as my argument to . has an exit code of 124 when it times out. This is your "interesting" case. If the exit code is nonzero but not 124, that means the minimizer produced invalid input; this is not interesting. Here's the script I used with bugpoint: The bugpoint run took around 9 hours in my case. llvm-reduce was much shorter, since the case was already minimized. One final annoyance is that these two tools expect inverted exit codes from your interesting-ness script (I had to swap and above when running with llvm-reduce). They both check that your input is considered "interesting," so you'll know right away if you need to switch it up.
filed an issue upstream:
Patch has landed:

Discussion:
<del> Subproject commit cd87134ab77e6bacb2128137065b328b9c35e0e5 </del> <ins> Subproject commit a6f4c1bb07d58df5956d2c49be68547143f77b18 </ins>