Change:
originally posted in Hi! I've found a regression in compilation introduced in , searched nightlies: from nightly-2021-08-17 to nightly-2021-09-10 regressed nightly: nightly-2021-08-26 searched commits: from to regressed commit: <details<summarybisected with <a href=''cargo-bisect-rustc</av0.6.0</summaryHost triple: x86_64-pc-windows-msvc Reproduce with: </detailsThe issue is that compilation succeeds in a acceptable time before this change, but now it's seemingly taking forever, I have not tried this on linux, only windows so far. I'm not sure how I should report this because I'm not able to reproduce it outside my large workspace where I bisected this. Does anyone have an idea to what this could be or have a good way to figure out what the cause is? some notable dependencies: actix-web, sqlx, serde, chrono, ring, tokio, tracing, reqwest,anyhow,eyre,async-trait and many more I've tried which outputs at the end and then just hangs. How do I proceed?
cc
One way to minimize could be to see which crate in your workspace is triggering the hang. E.g., run for each of the crates in the workspace and find the one with the fewest dependencies (in-workspace dependencies, or external). Then you can try to minimize that crate further by deleting different parts of the code until the hang disappears.
The crate/step which is failing is the final one. I managed to delete parts of the code to make the hang disappear but I'm not certain I've actually pinpointed it exactly yet. I let it run and seems it failed on
ran with and the build completed after a while. The memory usage jumps up dramatically, from 138 MB to 7GB to 20GB, and then down again to repeat. Same behaviour without those flags. This is probably not a hang, just increased complexity causing longer run-time. EDIT: On older nightly, the max memory usage is around 1.5GB
Managed to minimize it somewhat. Was not able to remove all dependencies but most of them. takes 40-45 seconds to compile from clean state. takes 125-500 seconds to compile from clean state. (and uses 5-7GB of memory sometimes) I believe the issue is to do with the client traits and impls and tracing instrumentation. Replacing with a todo!() makes compilation sane again.
ping cleanup I think it would be useful to further minimize the above Gist.
Hey Cleanup Crew ICE-breakers! This bug has been identified as a good "Cleanup ICE-breaking candidate". In case it's useful, here are some [instructions] for tackling these sorts of bugs. Maybe take a look? Thanks! <3 [instructions]: https://rustc-dev- cc
I took a profile of a debug build of rustc building that gist. This definitely doesn't seem to be a hang per se, there's always some work going on, it's just taking a really really long time. There's no one specific function we're spending all our time in either, its more like death by a thousand papercuts. Although we are spending about 25% of the time in hashmap-related code, but I imagine that's normal for rustc?
Hmm, I haven't looked at rustc profiles before, but 25% in hashmap-related code (I assume you mean code in the hashmap library—) sounds unusual to me. You could try profiling other trait-using code that doesn't slow rustc down a ton and see how much of the time is in hashmap code.
No, 25% may even be low. rustc is very HashMap intensive in most profiles.
diff of profdata on as base and as change. . Also some flamegraphs: [before], [after] [before]: [after]: can publish the raw profdata if wanted
I'm confused as to why there is no analysis in the flamegraph of after.
Hi all. Thanks a ton for the help on this. Especially It's definitely helpful to see that the majority of the time is spent during the codegen phase. Definitely I think getting a more minimized repro will help me get a fix for this.
That's gonna be tricky. Even just removing a few of the calls to tracing, something that I'd expect to have no side effects, would drastically reduce the compilation time. Same for deleting code that gets linted as unreachable.
Perhaps if you expand the macros (using something like ), you could remove parts of the calls to tracing and see what the effect is?
It's probably not so much total compile time, as much as time spent in
One thing I just spotted in my profiling is lots of stacks of collectitemsrec that are over 50 recursive calls deep. Is that expected? Is it worthwhile/possible to change this method to be iterative instead of recursive?
Managed to get it quite small, but still directly depends on and . removing and makes the memory footprint smaller. removing makes the problem dissappear, same if you just call directly. I tried to simulate with a few async fns that do things across await boundaries but couldn't replicate the complexity. I'm sure it's possible though. Thanks for the hint about using expanded, I did do that before but didn't think to remove parts of tracing.
Managed to make it 0-dep Compiles on the playground on stable, timeout/sigkill on current playground nightly (2021-09-14) and beta ofc Note: Can still be minimized. Just noticed for example that check_conn can just be an body edit: reduced more, noticeable better compiler performance though
For future endeavors like this, I think it'd be very useful to develop a fuzz-like tool that in a working compiler: all crates and removes unused types/paths (optionally) Sprinkle/replace in method bodies (and replacing method calls) in a fashion such that we work upwards from , branching if needed. That was basically the workflow I used to get to this.
It can't inline dependencies from other crates, but there's  which I've used successfully multiple times. (In combination with creduce, as its docs recommend.)
It was not helpful when I tried it in this case, maybe it'll work better now that's it's all inlined.
Fantastic! Good job! That would be helpful. The smaller the file, the better (though at least with this, I can probably get to this sometime this week).
Stable/beta/nightly debug/release builds all work for me for both linked playgrounds. Is there anything else required to reproduce the issue?
The issue is the memory usage and time. The problem is much more pronounced in the larger example. Time the difference between the two and you should see the issue. Edit: it doesnt compile for me on nightly playground: line 11: 8 Killed timeout --signal=KILL Can this issue be labeled I-compilemem and I-compiletime?
label +I-compiletime +I-compilemem -I-hang
I only get timeouts with beta/nightly if I use the default ''Run'', but not if I choose ''Build''. With Build and nightly I get: for the first link and for the second. With stable the times are 0.82s and and 4.83s but the timings vary widely on playground that this does not help much. Why does Run time out only for beta/nightly while Build works for all?
The regressed commit is currently not on stable, only beta and nightly, it will be promoted to stable soon. Hopefully if a fix can be found it can be beta backported
Not sure if this is helpful, but I was able to reproduce the latest minimal playground locally by making a new bin crate and compiling with the latest nightly toolchain with debug output. I can only see the hang when logs are output, otherwise it just take a few seconds. Eventually the logs get to: Where the last line repeats thousands and thousands of times
Last one I think, the more I minimize it, the quicker it compiles (but still slower than 2021-08-25).
Assigning priority as discussed in the of the Prioritization Working Group. label -I-prioritize +P-medium
status update: I've been trying to look into this. Given the complexity of the MCVE, it's very difficult. When running the MCVE with and letting it run for minutes, here is just a random snippet of the last few lines of output: A couple things to note: We've created region variables We've created 4359 universes for placeholder regions. There are some extremely gnarly types here. Given that I see a bunch of late-bound regions, it's not surprising that we might be doing a normalization somewhere that we weren't doing before.
As a sanity check, I compiled with and it has fixed the issue in my workspace, thank you for looking into this!
It's a bit unfortunate I wasn't able to find a "real" fix, but as Niko said, a decent stopgap measure.
Looks like the stopgap measure is causing problems... at least if the analysis in is correct, the 'fix' is causing ICEs in Miri when rustc is built with debug assertions.

Discussion:
// to make sure we don't forget to fold the substs regardless. match *ty.kind() { <del> ty::Opaque(def_id, substs) => { </del> <ins> // This is really important. While we *can* handle this, this has // severe performance implications for large opaque types with // late-bound regions. See `issue-88862` benchmark. ty::Opaque(def_id, substs) if !substs.has_escaping_bound_vars() => { </ins> // Only normalize `impl Trait` after type-checking, usually in codegen. match self.param_env.reveal() { Reveal::UserFacing => ty.super_fold_with(self), Reveal::All => { <del> // N.b. there is an assumption here all this code can handle // escaping bound vars. </del> let recursion_limit = self.tcx().recursion_limit(); if !recursion_limit.value_within_limit(self.depth) { let obligation = Obligation::with_depth(

// Wrap this in a closure so we don't accidentally return from the outer function let res = (|| match *ty.kind() { <del> ty::Opaque(def_id, substs) => { </del> <ins> // This is really important. While we *can* handle this, this has // severe performance implications for large opaque types with // late-bound regions. See `issue-88862` benchmark. ty::Opaque(def_id, substs) if !substs.has_escaping_bound_vars() => { </ins> // Only normalize `impl Trait` after type-checking, usually in codegen. match self.param_env.reveal() { Reveal::UserFacing => ty.super_fold_with(self), Reveal::All => { <del> // N.b. there is an assumption here all this code can handle // escaping bound vars. </del> let substs = substs.super_fold_with(self); let recursion_limit = self.tcx().recursion_limit(); if !recursion_limit.value_within_limit(self.anon_depth) {