jiebi/CodeConvo huggingface dataset:
Code change:
LLVMRustThinLTOPatchDICompileUnit(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } <ins> extern "C" void LLVMRustThinLTORemoveAvailableExternally(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } </ins> #endif // LLVM_VERSION_GE(4, 0)

Related discussion:
text: Regarding the compilation time difference between pre- and post-trans inlining, my hypothesis would be that sometimes pre-trans inlining will lead to more code being eliminated early on (as in the case of regex apparently) and sometimes it will have the opposite effect. I suspect that there's room for improvement by tuning which LLVM passes we run specifically for ThinLTO. Or do we do that already?
Heh it's true yeah, I'd imagine that there's always room for improvement in pass tuning in Rust :). Right now we perform (afaik) 0 customization of any pass manager in LLVM. All of the normal optimization passes, LTO optimization passes, and ThinLTO optimization passes are all the same as what's in LLVM itself.
I wanted to also take the time and tabulate all the results from the to make sure it's all visibile in one place. Note that all the timings below are comparing a release build to a release build with 16 CGUs and ThinLTO enabled Improvements All of the following compile times improved, sorted by most improved to least improved Regressions The following crates regressed in compile times, sorted from smallest regression to largest alexcrichton's attempt to reproduce the regressions Here I attempt to reproduce the regressions on my own machine with Unfortunately the only regression I was able to reproduce was the rust-belt regression. I'll be looking more into that.
Looking into , one interesting thing I've found is that the crate takes longer to compile with ThinLTO than it would otherwise. Looking at it appears that one codegen unit in this crate takes 99% of the time in LLVM. This codegen unit appears to be basically entirely dominated by . Almost all of the benefit of multiple codegen units is spreading out the work across all CPUs. Enabling ThinLTO to avoid losing any perf is fundamentally doing more work than what's already happening at , but we get speedups across the board in most cases. If we have one huge CGU, split it in two, and work on those in parallel then we've got 50% of the original time to run ThinLTO (in a perfect world). If ThinLTO takes more than 50% of the time then we'd have a build time regression, but that's almost never happening.
Code change:
LLVMRustThinLTOPatchDICompileUnit(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } <ins> extern "C" void LLVMRustThinLTORemoveAvailableExternally(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } </ins> #endif // LLVM_VERSION_GE(4, 0)

Related discussion:
text: What's happening here is that we have one huge CGU, but when we split it up we still have one huge CGU. This means that the CGU which may take ~1% less time in LLVM only gives us a tiny sliver of a window to run ThinLTO passes. In the case of the crate this means that adding ThinLTO passes is an overall build time regression. So generalizing even further, I think that enabling ThinLTO and multiple CGUs by default is going to regress compile time performance in any crate where our paritioning implementation doesn't actually partition very well. In the case of we've got one huge function which hogs almost all the optimization time (I think) and the entire crate is basically dominated by that one function. I would be willing to conclude, however, that such a situation is likely quite rare. Almost all other crates in the ecosystem will benefit from the partitioning which should basically evenly split up a crate. maybe you have thoughts on this though?
Thank you so much for collecting and analyzing such a large amount of data, Your conclusions make sense to me. Unevenly distributed CGU sizes are also a problem for incremental compilation; e.g. the test sees 90% CGU re-use, yet compile time is almost the same as from-scratch. In the non-incremental case it might be an option to detect the problematic case right after partitioning and then switch to non-LTO mode? I'm not sure it's worth the extra complexity though. For cases where there is one big CGU but that CGU contains multiple functions, we should be able to rather easily redistribute functions to other CGUs. We'd only need a metric for the size of a . The number of MIR instructions might be a sufficient heuristic here. That would not help for but it might help for other crates. In conclusion, judging from the table above, I think we should make the default sooner rather than later. Open questions I see: What to do about ? I would have thought that it would always be a clear win to disable this when ThinLTO is enabled but that doesn't seem to be the case. Are there hardware configurations where this never is a win? E.g. should we default to traditional compilation if ?
Code change:
LLVMRustThinLTOPatchDICompileUnit(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } <ins> extern "C" void LLVMRustThinLTORemoveAvailableExternally(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } </ins> #endif // LLVM_VERSION_GE(4, 0)

Related discussion:
text: FWIW, as the author of , which is a made-for-fun Piston game, I am more than happy to take a small compile regression when it seems like vast majority of crates get huge real-time improvements!
This seems like a good thing to have in our back pocket, but I'm also wary of trying to do this by default. For example the case is one where we may wish to disable ThinLTO but the one massive function could also critically rely on a function in a different CGU being inlined? That sort of case would be difficult for us to determine... In any case though I agree that we probably don't need the complexity just yet so I think we can defer this for later. Agreed! So far all the crate's I've seen the O(instructions) has been quite a good metric for "how long this takes in all LLVM-related passes", so counting MIR sounds reasonable to me as well. Again though I also feel like this is ok to have in our back pocket. I'd want dig more into the minor change example, I'm curious how awry the CGU distribution is there and whether a heuristic like this would help. Quite surprisingly I've yet to see any data that it's beneficial to compile time in release mode or doesn't hurt runtime. (all data is that it hurts both compile time and runtime performance!) That being said we're seeing such big wins from ThinLTO on big projects today that we can probably just save this as a possible optimization for the future. On the topic of inline functions I recently dug up again (later closed in favor of , but the latter may no longer be true today). I suspect that may "fix" quite a bit of the compile time issue here without coming at a loss of performance? In any case possibly lots of interesting things we could do there. Also a good question! I find this to be a difficult one, however, because the CGU number will affect the output artifact, which means that if we do this sort of probing it'll be beneficial for performance but come at the cost of more difficult deterministic builds. You'd have to specify the CGUs manually or build on the same-ish hardware to get a deterministic build I think? I think though that if you have 1 CPU then this is universaly a huge regression.
Code change:
LLVMRustThinLTOPatchDICompileUnit(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } <ins> extern "C" void LLVMRustThinLTORemoveAvailableExternally(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } </ins> #endif // LLVM_VERSION_GE(4, 0)

Related discussion:
text: With one CPU we'd split the preexisting one huge CGU into N different ones, probably take roughly the same amount of time to opimize those, and then tack on ThinLTO and more optimization passes. My guess is that with one CPU we'd easily see 50% regressions. With 2+ CPUs however I'd probably expect to see benefits to compile time. Anything giving us twice the resources to churn the cgus more quickly should quickly start seeing wins in theory I think. For now though the deterministic builds wins me over in terms of leaving this as-is for all hardware configurations. That and I doubt anyone's compiling Rust on single-core machines nowadays!
One thing I think that's also worth pointing out is that up to this point we've mostly been measuring the runtime of an entire . That's actually, I believe, the absolute worst case scenario for where ThinLTO will provide benefit. Despite this, it's showing huge improvements for lots of projects! The benefit of ThinLTO and multiple CGUs is leveraging otherwise idle parallelism on the build machine. It's overall increasing the amount of work the compiler does. For a from scratch, though, you typically already have tons of crates compiling for the first half of the build in parallel. In that sense there's not actually any idle parallelism. Put another way, ThinLTO and multiple CGUs should only be beneficial for builds which dont have many crates compiling in parallel for long parts of the build. If a build is 100% parallel for the entire time then ThinLTO will likely regress compile time performance. Now you might realize, however, that one very common case where you're only building one crate is in an incremental build! Typically if you do an incremental build you're only building a handful of crates, often serially. In that sense I think that there's some massive wins of ThinLTO + multiple CGUs in incremental builds rather than entire crate builds. Although improving both is of course great as well :)
For deterministic builds you have to do some extra configuration anyway (e.g. path remapping) so I would not consider that a blocker. And I guess there are single core VMs around somewhere.
Code change:
LLVMRustThinLTOPatchDICompileUnit(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } <ins> extern "C" void LLVMRustThinLTORemoveAvailableExternally(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } </ins> #endif // LLVM_VERSION_GE(4, 0)

Related discussion:
text: But all of this is such a niche case that I don't really care MIR-only RLIBs should put us into a pretty good spot regarding this, so I'm quite confident that we're on the right path overall.
That's really interesting. For incremental compilation enabled the situation might be different (because pre-trans inlining hurts re-use) but for the non-incremental case it sounds like it's pretty clear what to do.
Yeah, maybe we should revisit this at some point. Although I have to say the current solution of only and symbols and nothing in between is really nice and simple.
Hm yeah that's a good point about needing configuration anyway for deterministic builds. It now seems like a more plausible route to take! Also that's a very interesting apoint about incremental and inlining on our own end... Maybe we should dig more into those ThinLTO runtime regressions at some point! Also yeah I don't really want to change how we trans inline functions just yet, I do like the simplicity too :)
out of curiosity how did you figure out this and which function it was. I've wanted to look into why the webrender build is so slow and would welcome tips.
oh sure I'd love to explain! So I originally found as a problematic crate when compiling rust-belt as it just took awhile and I decided to dig deeper. I checked out the crate and ran: That drops in the current directory, and opening that up I see: ! The graph here isn't always the easiest to read, but we've clearly got two huge bars, both of which correspond to taking a huge amount of time for that one CGU (first is optimization, second is ThinLTO + codegen). Next I ran: and that command dumps a bunch of IR files into . Our interesting CGU is so I opened up (we sure do love our long filenames). Inside that file it was 70k lines and some poking around showed that one function was 66k lines of IR. I sort of forget now how at this point I went from that IR to determining there was a huge function in there though...
I don't know if this is the right place, but I think it should be considered whether the symbol issues reported in the Nightly section here: should be considered a blocker or not.
Code change:
LLVMRustThinLTOPatchDICompileUnit(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } <ins> extern "C" void LLVMRustThinLTORemoveAvailableExternally(LLVMModuleRef Mod) { report_fatal_error("ThinLTO not available"); } </ins> #endif // LLVM_VERSION_GE(4, 0)

Related discussion:
text: The tl;dr is that certain versions of llvm, (i've seen this on 3.8, and whatever llvm rustc nightly uses) appends seemingly random garbage to the end of some names, e.g. we get: instead of: This knocks the debuginfo out of sync (it doesn't have the garbage appended). I've been able to repro this with clang3.8 for c++ files as well (haven't tested on other llvm), and switching to 5.0 seems to have fixed the issue. I don't know if its within reach, but perhaps we should attempt upgrading to llvm 5.0 before releasing this on stable? Note I understand this is for release mode, but i see this in debug mode on rustc nightly right now as well...
Code change:
(&self_ty.kind, parent_pred) { if let ty::Adt(def, _) = p.skip_binder().trait_ref.self_ty().kind { <del> let id = self.tcx.hir().as_local_hir_id(def.did).unwrap(); let node = self.tcx.hir().get(id); </del> <ins> let node = self .tcx .hir() .as_local_hir_id(def.did) .map(|id| self.tcx.hir().get(id)); </ins> match node { <del> hir::Node::Item(hir::Item { kind, .. }) => { </del> <ins> Some(hir::Node::Item(hir::Item { kind, .. })) => { </ins> if let Some(g) = kind.generics() { let key = match &g.where_clause.predicates[..] { [.., pred] => {

Related discussion:
text: I tried code very similar to this: and it produces some type of type checking error when uncommenting the line beginning with . : Backtrace:
The error is likely due to the fact that the trait bounds on don't include for , though the compiler/type checker crashing is still likely a bug.
That code was introduced in , cc Submitted as a quick fix.
I can't get it to work in the same crate, just with two crates, where the first one depends on the other: modify labels: -E-needs-mcve
Thanks I included a test in the PR.
Code change:
/// Unix-specific extensions to `Permissions` #[stable(feature = "fs_ext", since = "1.1.0")] pub trait PermissionsExt { <del> /// Returns the underlying raw `mode_t` bits that are the standard Unix /// permissions for this file. </del> <ins> /// Returns the underlying raw `st_mode` bits that contain the standard /// Unix permissions for this file. </ins> /// /// # Examples ///

Related discussion:
text: On linux, , then run: This changes the permissions of to . Is this expected? masks away everything other than the lower 3 triads when it calls out to this code: Then sends these mode bits into into chmod which considers four triads, not three. Possibly useful references: and Should we change Metadata.permissions to preserve that triad, maybe by masking with instead of ?
I guess the solution depends on whether Unix's std::fs::Permissions is supposed to care about 9 bits or 12 (this is not an implementation detail since it's exposed to the programmer via of PermissionsExt). any thoughts on this?
It seems to me like we should extend the mask to the full 12 bits.
Seems reasonable to me to avoid masking where it makes sense!
Code change:
let ty::CoroutineClosure(_, parent_args) = *tcx.type_of(parent_def_id).instantiate_identity().kind() else { <del> bug!(); </del> <ins> bug!("coroutine's parent was not a coroutine-closure"); </ins> }; if parent_args.references_error() { return coroutine_def_id.to_def_id();

Related discussion:
text: <!-- [31mICE[0m: Rustc '-Zunstable-options --edition=2024 -ooutputfile -Zdump-mir-dir=dir' 'error: internal compiler error: impossible case reached', 'error: internal compiler error: impossible case reached' File: --auto-reduced (treereduce-rust): original: Version information ` Command: <!-- Include a backtrace in the code block by setting in your environment. E.g. . --<details<summary<strongProgram output</strong</summary<p</p</details<!-- query stack: [coroutinebymovebodydefid] looking up the coroutine by-move body for [analysis] running analysis passes on this crate -- label +F-asyncclosure
As a step of reducing your code examples, could you please ensure that is necessary to reproduce this ICE before mentioning it in the issue? This ICE reproduces on edition 2021, which is the current stable edition.
My project fails on nightly also while using edition 2021.
Yep, a fix is already up and it's waiting for approval. I recommend downgrading to a slightly older nightly until this is fixed.
Code change:
/// Sets the platform-specific value of errno #[cfg(all(not(target_os = "linux"), not(target_os = "dragonfly")))] // needed for readdir and syscall! <ins> #[allow(dead_code)] // but not all target cfgs actually end up using it </ins> pub fn set_errno(e: i32) { unsafe { *errno_location() = e as c_int } }

Related discussion:
text: Good question! There are 6 workarounds in for older Linux versions (that I could find). Increasing the minimum version to 2.6.32 (aka 3rd Kernel LTS, aka RHEL 6) would fix 5 of them. Code links are inline: uses the flag with  (mentioned above, in 2.6.23) uses with  ( in 2.6.24, there is also the mention of a bug occuring on "some linux kernel at some point") uses  to atomically set the flag on the pipe fds ( in 2.6.27) uses with  ( in 2.6.27) uses  to permit use of ( in 2.6.28) ~ ( in 4.5, not fixed by this proposal)~ As you can see, the workarounds fixed by this proposal all have a similar flavor. Originally posted by in
Kernel updated in .
As noted, needs a much newer kernel 4.5, so we'll still need that fallback. But all the stuff should be ok to assume available now.
The new minimal version is 2.6.27 from what I can see, so "Socket::accept uses accept4(2) to permit use of SOCK_CLOEXEC" still needs a fallback, right?
The builder is using 2.6.32 headers, so I would call that our new minimum.
Ah right; I am actually not sure where I got the 27 from...
The title of suggested 2.6.27, but we went a little higher to match the actual distros in use.
I know this is ancient history, but do you recall the problem with ? You introduced that comment about "some kernel" in , and you were the last to update it in . If the problem kernels were before 2.6.32, then I'd like to clean that up now.
Just digging in kernel history, I did find one hiccup in , but that bug was introduced and resolved during the 3.7 development cycle, not in a release.
IIRC we just mirrored what musl did at the time, which looks like it : Looks like the original mailing list thread is gone and there's no comments in the musl source, and I don't recall what happened here. I suspect it would be safe to remove those workarounds and just assume works one the first try until someone says differently.
Code change:
assert!(rows.iter().all(|r| r.len() == v.len())); <del> // FIXME(Nadrieril): Hack to work around type normalization issues (see #72476). let ty = matrix.heads().next().map_or(v.head().ty(), |r| r.ty()); </del> <ins> let ty = v.head().ty(); </ins> let is_non_exhaustive = cx.is_foreign_non_exhaustive_enum(ty); let pcx = PatCtxt { cx, ty, span: v.head().span(), is_top_level, is_non_exhaustive };

Related discussion:
text: <!-- Thank you for finding an Internal Compiler Error! If possible, try to provide a minimal verifiable example. You can read "Rust Bug Minimization Patterns" for how to create smaller examples. --When compiling on an Apple M1 processor: <!-- If you're using the stable version of the compiler, you should also check if the bug also exists in the beta or nightly versions. --: <!-- Include a backtrace in the code block by setting in your environment. E.g. . --<details<summary<strongBacktrace</strong</summary<p</p</details
Can also reproduce this when compiling on an Intel Macbook
I've encountered the same error on a different platform (x8664) when compiling . The project doesn't use cargo so repro is nontrivial, but from I can see that the ICE started occurring at which is a rollup of . <details<summary<strongBacktrace</strong</summary<p</p</details
Also seems to be hitting starlark-rust, , as per
Somewhat minimized standalone testcase: Builds on stable, fails on nightly. cc
Thanks for the minimization! Seems like we've hit again, but through a different code path. I'm investigating.
I think might be another duplicate of this, triggered when compiling the crate.
Code change:
let mut out = output.stderr.clone(); out.extend(&output.stdout); let out = String::from_utf8_lossy(&out); <ins> // Check to see if the link failed with "unrecognized command line option: // '-no-pie'" for gcc or "unknown argument: '-no-pie'" for clang. If so, // reperform the link step without the -no-pie option. This is safe because // if the linker doesn't support -no-pie then it should not default to // linking executables as pie. Different versions of gcc seem to use // different quotes in the error message so don't check for them. if sess.target.target.options.linker_is_gnu && (out.contains("unrecognized command line option") || out.contains("unknown argument")) && out.contains("-no-pie") && cmd.get_args().iter().any(|e| e.to_string_lossy() == "-no-pie") { info!("linker output: {:?}", out); warn!("Linker does not support -no-pie command line option. Retrying without."); for arg in cmd.take_args() { if arg.to_string_lossy() != "-no-pie" { cmd.arg(arg); } } info!("{:?}", &cmd); continue; } if !retry_on_segfault || i > 3 { break } </ins> let msg_segv = "clang: error: unable to execute command: Segmentation fault: 11"; let msg_bus  = "clang: error: unable to execute command: Bus error: 10"; if !(out.contains(msg_segv) || out.contains(msg_bus)) {

Related discussion:
text: In Ubuntu 15.10 and later the default GCC toolchain installed has been compiled with which causes and to be passed to during code generation and to during linking. Now, the linking part is what is important here as rust on most platforms uses as the linker for most targets. So, though no PIC or PIE code may have been generated by , the linker will still try to link as with PIE in mind, breaking when trying to compile with and the like. To work around this will need to see that the linker has been compiled with PIE as the default and pass to the linker if using GCC or something similar for other linkers. Code to do this can be seen in for Tarpaulin though it is only meant for Linux in that state and can be improved greatly if in tree. EDIT: Don't know why Github isn't linking cross-project PRs correctly right now.
Can this issue be resolved by always passing , except when compiling with ?
Sadly, no. In testing I have seen that any build of GCC that was not built with rejects the option and fails to run. This is why in the PR that I linked the linker was queried to see if it was compiled with before tacking on .
So it's came out from the discussion on an issue related to this in tarpaulin that passing -no-pie before -shared (and maybe then -static if it exists for static linkages as well) resolves the issue. I'm going to test this tarpaulin side and I can report back but I feel it would be best if this could be solved rustc side as it can affect any user in a similar situation.