Change:
Recent rustc seems to miscompile something somewhere in the dependency tree of Bevy, causing apps to crash with a segfault. The issue seems to only manifest when optimizations are enabled and codegen units are 1. The default cargo release configuration triggers it. Unoptimized builds are fine. Setting with also seems to be fine. I am on macOS 14.1 Sonoma, on Apple M1 Pro hardware. You can reproduce with any Bevy application. It is a huge code base, I know, I'm sorry I can't find a more minimal example. Clone the Bevy repo: all these revs have the issue: commit (i had that one laying around) commit (current latest) tag v0.12.0 tag v0.11.3 Run some example in release mode: The previous 1.74.0-beta.7 did not have the issue. I am not sure which nightly introduced the regression. I first noticed the issue when I installed the 2023-11-11 nightly, though it might also be present in earlier nightlies. I just installed 1.75.0-beta.1 and updated my nightly, and can confirm the issue is present there, too.
I cloned the repo and ran the command you provided, but got errors: with both stable and nightly on bevy commit did I do something wrong? what's your bevy commit?
I specifically tried: commit (i had that one laying around) commit (current latest, same as you) tag tag Yes, the previous 2 releases of Bevy (at least) are affected, too. I don't get your error, everything compiles without errors on my end. Have you tried followed by , to ensure you have a clean target directory and cargo isn't getting confused by outdated deps?
uhhh, I have run and both .... wtf cargo?? I tried it on another computer and it compiles. let's dismiss that as a me issue. I can't try it on this other computer for unrelated reasons though so I can't try reproducing it on linux as well to rule out macos-specific issues.
It would be useful if you could use to bisect the regression to a single PR, which would make it easier to figure out what's happening and whether it really is a miscompilation or just UB that's now going wrong in bevy's dep tree.
I was not able to reproduce this on Windows. Marking as O-MacOS, but I suspect that the problem is platform-independent and just only manifests in platform-specific bevy code.
OK. TIL about . Installing it and reading the documentation now. Will try it in a moment. Edit: I created a simple bevy project that should automatically exit after the first frame is rendered to the screen. This should be scriptable. It will exit successfully on older rustc, or segfault on new rustc. is running now, as I am typing this. Here is the bevy code if anyone else wants to try it:
I get On MacOS 14.0
Bisected down to: Regression in nightly-2023-10-19 Regression in
oh well, that is sadly not a very useful bisection. it enabled more inlining, causing more optimizations to be applied overall. anyways cause you may be interested in this, but I can't imagine that your PR is at fault can you try enabling LTO, which should get the same behavior on earlier versions and see whether that crashes as well and then bisect that? Additionally, it would be useful to use a debugger to figure out where the segfault occurred and maybe why it did. It may be possible to figure out what got compiled weirdly, which could lead us to be able to figure out whether it's UB in the code.
That makes me wonder if Bevy is doing this thing that Embassy is (was?) doing: i.e. if you are relying on the address of an empty function being the same every time it is inspected, it must be one of or and I'd advise the latter.
Just tested it. LTO does not cause a segfault in earlier versions, and does cause a segfault in new versions. It does not seem to affect the results. Also, like I said, setting in the new rustc does not result in segfault. Even though it should be enabling the most inlining optimizations. It is quite interesting. The issue occurs only with optimized builds with 1 codegen units, regardless of LTO.
Ran in lldb: Seems to be an invalid memory access in ? Included a backtrace, and disassembly of the function, if that helps.
Bevy does something similar for its system, but it doesn't rely on function pointer equality, instead it erases the function item into a trait object and uses its for checking, which shouldn't suffer from the same problem.
It looks like disabling cross-crate inlining "solves" the issue, i.e.:
WG-prioritization assigning priority (). label -I-prioritize +P-high
Console-only reproducer:
I can 1+ this, I was just doing the bevy tutorial and got to the part where I add DefaultPlugins leading to a seg fault. Once I it recompiled and ran as expected. Macbook Air - Apple M2 Clean cargo install using nightly Bevy 0.12 EDIT: Dropping from to in -also seems to prevent the seg fault Hope that aids in the troubleshooting effort :)
After reducing 's 150KLOC and 200+ dependencies: the "miscompilation" is not in , but indeed likely in . Here's a more minimal reproducer: There's some non-determinism involved, and likely from parallelism (and could depend on the hardware involved I'm not sure). The texture creation call above can: work fail with a segfault fail with a bus error fail with a wgpu validation error about the texture format being unusable on the device fail with an abort during some metal deallocation, fail with an abort during a free, For , updating to seems to work around the issue. At least, that was enough for me on the reproducer above. If someone wants to help, bisecting from 0.17.x to 0.18 should at least give an indication of what happened to change this behavior here (could be UB and/or missing synchronization, but just looking at the open issues there we can see mentions of data races and missing synchronization when using metal). That may help see why cross-crate inlining was involved here.
If it is a data race, -Zsanitizer=thread might be informative.
(I would be trying these myself but I just have a phone for a bit) Can you bisect along values of -Ccodegen-units and -Zcross-crate-inline-threshold? I would bisect between 1 and 1024 for both. If you can find a cross crate threshold that's required it wouldn't be difficult to stick some prints into it compiler to see what DefId is changing codegen mode.
(any other wgpu maintainers?) Do you have any knowledge/ideas that might help with figuring out why rustc might be miscompiling wgpu?
While we do have various races in places, that code is should be rather straightforward - no immediate ideas why that would be miscompiled.
This may have been fixed/worked around by Here are a few , but weren't enlightening to me. Maybe Ben will see something.
Trying to understand the behavior of LLVM given such a large program which is known to execute UB (data races and now null pointer dereferences) seems way too difficult to me. If I was working this (I'm not because it's holiday time, but maybe I can look into this after) I'd fix those bugs then reasses.
Raising the priority to signal that we want to track this and ensure it does not land on stable (wherever the root cause) label -P-high +P-critical
The cross-crate-inlining change I made essentially infers the presence of the attribute where it is deduced to be a reasonable addition. It will never infer on a function which is . I followed my own suggestion above to bisect along values of and deduced that the magic value is 21. Then I stuck some code into the compiler to print every that we analyze with a value of 21. Only a handful were printed in the build. I hacked up the contents of my to add to each of them in turn. The magic function is this: If I add to that function, reproducer above will segfault on stable.
leads me to , which if I put on prevents the segfaults. But above that is which is huge and does a lot of things, and adding to that function has no effect. So I think the unexpected codegen happens while optimizing , which at least I suppose narrows things down from the whole crate graph.
Oh ... great ... so this is a miscompilation bug that also exists in stable, which could have been triggered if some specific code was inlined. Now rustc just inlines it automatically. So it isn't a codegen regression, but an old/preexisting bug exposed by new rustc enabling more aggressive optimizations. Damn. I'm so glad I reported this. Never could have imagined that it would go so deep...
I am not yet convinced that this is a miscompilation as opposed to UB that is now exploited.
These 2 functions are the closest to the segfault in the backtrace, so the problem surely lies around there. We could likely extract them both and hopefully repro outside of wgpu.
I have not managed to separate the strangely-behaving code out from wgpu, but I have learned some things. is very strange. If I stick on it, LLVM generates this function signature: Which is exactly what you'd expect. But if I remove the attribute and leave cross-crate-inlining enabled, or add we get this signature: So LLVM has done some kind of cross-function optimization here. The IR for the function bodies looks exactly the same to me, except for the fact that one does two gepis and the other just uses its arguments. And of course they are lowered to slightly different assembly. Here is the one: And the one: Directly after both of these is a small jump table. But I think what's down there is irrelevant, because in the one, when we on it, contains an address that is comfortably outside this function. So I think whatever is going on here relies on getting this SRoA codegen, but also not be inlined. lldb has been supremely anti-helpful. It says breakpoints are set then blows past them (realizing this was happening took me a long time), and if I do too much instruction stepping it segfaults.
I think what's happening with the signature is argument promotion, where it replaces an indirect argument with a direct one.
I've removed 99% of in , but it still currently clocks in at 1000 lines or so. It's indeed way harder than one would hope. It's quite fiddly, maybe there's a magic value of the inlining threshold for different shapes, combined with various inline never/always attributes, but I didn't manage to easily merge more of the crates together without making the issue disappear; there are still 2 crates in addition to the main binary. There are still a bunch of backends/API/etc wgpu abstractions in the repro so it can still be reduced more, but I'm not sure I'll have much more time to dedicate to this. At the very least, there are still two dependencies out of three that shouldn't be hard to remove ( and ) . But there are limits/roadblocks to how much we can minimize without having thresholds/inline attributes that make this reproduce all the time, as some of these sizes impact inlining decisions. I still don't know if this is a miscompile or UB (maybe slightly leaning towards the former): if it's the latter, it's maybe not in : there's no unsafe code in the repro per se, but there's surely some in . it's possible that on mac exhibits an issue though. I think* I removed all the osx/metal specific code, it could be cool to test on aarch64-linux (maybe on the dev-desktops) and on an intel mac, to see what happens there. switching from to the std mutex and rwlock also made the segfault disappear but it's not clear whether it could just be some different behavior or different codegen that would stop triggering the issue. the number of variants in the enum seem to matter. There needs to be quite a lot or the problem disappears.
I have some ideas on how to shrink this further and I'm starting to make progress...
My latest progress brings the line count down to 221 lines, and there is precious little logic left, it's mostly bitflags and enum definitions. It's in a PR to the above repo. It got way more consistent when I found the path to remove the locks. I'm sure the remaining bits of could be cleaned up. And I didn't try removing any of the crate boundaries, though I suspect those are required. At this point I'm quite sure this is a miscompile. There's no unsafe code left in the repo, and Miri doesn't complain. I've tried running this on aarch64-unknown-linux-gnu and x8664-unknown-linux-gnu, and neither segfault. I doubt this is an OS issue, perhaps there is something different between the default aarch64-apple-darwin target and the aarch64-unknown-linux-gnu target that would be enlightening.
Also somehow the MIR inliner is still required. Though it's not clear if that's because the bug is in the MIR inliner or if only post-inlining do we generate the exact LLVM IR pattern that gets miscompiled.
The repro is now at around 150 lines, with 1 bin and 1 lib. It's minimal enough e.g. for a test or LLVM bug report. Doesn't fully turn off the inliner? Because the segfault reproduces for me even with this flag. In any case, the LLVM optimization pass that seems to start introducing the segfault is from the lib — to reproduce this on the latest revision (with or , the limit is 1404 on the lib). cc
Yes that should turn off the MIR inliner. I was aiming for a reproducer that has because that would turn off all MIR optimizations, and I found that when I that flag I stopped getting segfaults unless I enabled MIR inlining (which is a pass, but it also has its own special flag).
Maybe the smaller repro behaves slightly differently now, still segfaults for me, interestingly enough.
It's nice to know I'm not the only one experiencing random segfaults in my bevy apps on , and that it is being actively explored! I have no dev experience to offer, but I did notice another compiler bug () , but I'm pretty sure they are unrelated.
Nice reproduction. But I can't reproduce it on macOS x86_64. Purchasing M1. I have written very less Machine IR related code. But I'll try to look into it when the M1 arrives.
Upstream issue: I tried to submit a fix as well. Base on , I got
Hmm, I can only use nightly to reproduce, and I can't use local stage1/stage2 to reproduce. This could be a difference in build configuration. So I think this minimal reproduction is unstable. If we want to use this as a test case, I would expect a more stable test case. I a more detailed explanation of this issue at We should be able to create a must-be-dirty stack to reproduce.
I got a "perfect" test case. I can reproduce it with nightly, 1.74.1 and even 1.69.0. By examining the generated instructions, the following instruction sets to : godbolt:
Reopening to track beta backport in .
This can now be closed: this issue was fixed on nightly by the cherry-picked LLVM fix in , and fixed on beta by backporting that PR in , so it thankfully won't reach stable in a couple weeks. Thanks in particular to and

Discussion:
[submodule "src/llvm-project"] path = src/llvm-project url = https://github.com/rust-lang/llvm-project.git <del> branch = rustc/17.0-2023-09-19 </del> <ins> branch = rustc/17.0-2023-12-14 </ins> shallow = true [submodule "src/doc/embedded-book"] path = src/doc/embedded-book

config.llvm_link_shared.set(Some(true)); } } else { <del> config.llvm_from_ci = config.channel == "dev" && crate::core::build_steps::llvm::is_ci_llvm_available(&config, false); </del> <ins> config.llvm_from_ci = config.parse_download_ci_llvm(None, false); </ins> } if let Some(t) = toml.target {

download_ci_llvm: Option<StringOrBool>, asserts: bool, ) -> bool { <ins> let if_unchanged = || { // Git is needed to track modifications here, but tarball source is not available. // If not modified here or built through tarball source, we maintain consistency // with '"if available"'. if !self.rust_info.is_from_tarball() && self .last_modified_commit(&["src/llvm-project"], "download-ci-llvm", true) .is_none() { // there are some untracked changes in the the given paths. false } else { llvm::is_ci_llvm_available(&self, asserts) } }; </ins> match download_ci_llvm { <del> None => self.channel == "dev" && llvm::is_ci_llvm_available(&self, asserts), </del> <ins> None => self.channel == "dev" && if_unchanged(), </ins> Some(StringOrBool::Bool(b)) => b, // FIXME: "if-available" is deprecated. Remove this block later (around mid 2024) // to not break builds between the recent-to-old checkouts. Some(StringOrBool::String(s)) if s == "if-available" => { llvm::is_ci_llvm_available(&self, asserts) } <del> Some(StringOrBool::String(s)) if s == "if-unchanged" => { // Git is needed to track modifications here, but tarball source is not available. // If not modified here or built through tarball source, we maintain consistency // with '"if available"'. if !self.rust_info.is_from_tarball() && self .last_modified_commit(&["src/llvm-project"], "download-ci-llvm", true) .is_none() { // there are some untracked changes in the the given paths. false } else { llvm::is_ci_llvm_available(&self, asserts) } } </del> <ins> Some(StringOrBool::String(s)) if s == "if-unchanged" => if_unchanged(), </ins> Some(StringOrBool::String(other)) => { panic!("unrecognized option for download-ci-llvm: {:?}", other) }

assert_eq!(parse_llvm(""), if_unchanged); assert_eq!(parse_llvm("rust.channel = "dev""), if_unchanged); assert!(!parse_llvm("rust.channel = "stable"")); <del> assert!(parse_llvm("build.build = "x86_64-unknown-linux-gnu"")); assert!(parse_llvm( </del> <ins> assert_eq!(parse_llvm("build.build = "x86_64-unknown-linux-gnu""), if_unchanged); assert_eq!(parse_llvm( </ins> "llvm.assertions = true rn build.build = "x86_64-unknown-linux-gnu" rn llvm.download-ci-llvm = "if-unchanged"" <del> )); </del> <ins> ), if_unchanged); </ins> assert!(!parse_llvm( "llvm.assertions = true rn build.build = "aarch64-apple-darwin" rn llvm.download-ci-llvm = "if-unchanged"" ));

<del> Subproject commit 7738295178045041669876bf32b0543ec8319a5c </del> <ins> Subproject commit 2c4de6c2492d5530de3f19f41d8f88ba984c2fe2 </ins>