Change:
On any recent MSVC nightly, compiling with profile with results in either or depending on the crate. Many crates work, but others don't. Among those that fail some use SIMD. resolves to on my machine. This seems to be related and the LLVM upgrade, again. It does not happen when is not set. Everything works on but fails after , same as the aforementioned issue. I am not sure how to reproduce it in a single crate, but I will look into it. LLVM 9 just doesn't like AMD. Although, another issue of mine: also fails in a similar manner before the LLVM upgrade, so it's worth noting.
Here is the verbose build output: <details</detailsThe errors are the same with a clean build, but I used a subsequent attempt to cut down on the log size.
A simplified test case is simply adding to an empty crate.
seems to be partially responsible. Removing that fixes it. So it's not LTO at least.
I can also trigger this on the profile by changing: with does not trigger it.
Checking in from triage: This seems to be related to our LLVM upgrade. The linked issue () was blamed on and fixed by via a LLVM submodule update (). cc and -- Any thoughts on what's going on here?
Tagging as P-high for now. Not sure who to assign to.
(Sound this be labeled as I-unsound?)
Is it possible to get a backtrace for the segfault? I don't have a windows system (or a zen system for that matter) to reproduce this on. I don't know whether Windows has assertion-enabled builds, but if it does, it might be worth calling with the argument and check if the toolchain this downloads triggers an assertion failure.
Probably not. The crash is in the compiler process, not in code it output, and presumably in C++ code at that, so there's no reason to expect there's anything going wrong with any safe Rust code. While it's the scary sort of crash that sounds like it could hypothetically also result in completely bogus machine code being generated, there's no evidence of this actually happening / being possible. I mean I guess we could decide to tag stuff I-unsound merely because "something's gone really wrong in the C++ and that could have arbitrarily bad consequences", but if we do that we should also blanket tag all LLVM assertion failures as I-unsound, but we don't currently do that nor do I think it would be useful.
Could not reproduce with or on on Zen 2000 based system using Linux GNU and by cross compiling to Windows GNU, I'll check native Windows GNU toolchain later.
Happens with both MSVC and GNU builds on Windows for me. How would I go about enabling backtraces on a Windows build of rustc? EDIT: Would rustc even produce a backtrace on segfault? I know I've seen proper backtraces with ICEs, but this is different. (just the alt version of the last nightly I had) does not appear to respond to
It's not panic but LLVM segfault so you should use with Windows GNU toolchain, no idea about MSVC.
On Windows exits with and GDB only prints: . There are no alternative builds for Windows GNU toolchain so I won't be able to do anything until I do debug build.
Make sure you running the real and not the wrapper from rustup. What you’re seeing here is a typical symptom of failing to account for the wrapper.
Hmm, I could swear I could debug crash on Linux without caring about the wrapper. Anyway the stack is corrupt: <details<summaryClick here to expand</summary</detailsOther threads are just waiting. I'll return with debug Rust build if I somehow manage to build LLVM in debug mode on PC with 16 GiB of RAM...
What led you to the conclusion of a corrupt stack? It looks fairly reasonable to me.
I think the wrapper switched to at some point (i.e. the process gets reused, without forking). AFAIK, that would allow debugging to continue to the real . I don't think anything like this is possible on Windows (without manually loading the executable you want to run in your address space, of course).
You don't need to, this is not in LLVM, it's in . You can probably reproduce with just (or / ).
Process gave me exit code for stack corruption before, took quick look at the trace and it didn't make any sense to me. On second look I noticed the crash happened inside proc macro... Yeah, it hit me later. Running on crate with changes from reliably reproduces it. In case you find it useful here is trace from debug build and disassembly: Assembly for and generic is the same so the problem is somewhere earlier. What is the best way to proceed here, compiler with assertions or tearing down to something more handy?
Since AFAICT the bug happens during the macro expansion in , you should be able to get rid of most of it. Not even names need to be resolvable, other than invoking 's macros. So, for example, you can remove all dependencies of , other than , because they're not needed in the reproduction. EDIT: wait, no, it must be code compiled with that's getting miscompiled, so it's all within /. Could you try to run with hardcoded somewhere? (presumably in )
I've been struggling few past 2 days to build Rust because of so I'm unable to progress on this issue.
Any luck with this? I'm still stuck on because of it.
Odd. On my personal project, this appears mostly fixed on the latest nightly (), but the issue still occurs on the example.
visiting for triage. Its not clear to me whom on our team can investigate this; has anyone from managed to reproduce this locally? Sounds like the answer to that is likely "no."
Just noticed it's fully broken again on .
triage: are you still looking at this? should I try to identify someone else with this hardware setup who might be able to help further?
I'll try to find time to setup environment with older C toolchain today (to workaround ).
Status update: I built current master for with and assertions for both rustc and LLVM but it does not reproduce the crash. I can still reproduce in on nightly though.
visiting for triage. , things certainly sound fishy (and frustrating) when a local build does not replicate the problem but the CI-produced nightly artifacts does. Would it be feasible for you to replicate something closer to what the CI setup is, e.g. via a docker image?
Finally reproduced with local build but experiment proposed by only broke 4 unrelated tests (cross compilation, simd detection...). Tested passing either or . Back to the drawing board I guess.
Also had this happen spuriously on the Windows GNU toolchain (latest nightly), but running again continued where it left off and finished just fine.
Is there anything I can do to help speed up fixing this issue? It's been nearly two months since I've been able to compile benchmarks, and it's interfering with my work.
I don't actually use directly, it was just one of the many crates that was previously crashing rustc. However, on my system almost always crashes rustc, or even somehow compiles but hangs during execution (). It's very unpredictable, and sometimes works on tiny one-crate projects, but for my primary work project workspace, it's totally unusable. Even compiling non-criterion benchmarks crashes rustc sometimes. I can't even compile Criterion as a regular crate binary (rather than using the profile). Amazingly my primary binary somehow started compiling again about a month ago, or else I'd be totally screwed, but everything else is broken on Zen 1. At this rate I'll have to roll my own benchmarking tools to continue my work.
Removed previous comment as it was just PEBCAK. not sure it it helps but I reduced to this: It does not reproduce directly but crashes when another crate depends on crate with this snippet. It appears it requires specific combination: derive (de)serialize, enum, doc comment inside, rustc argument and Windows as the OS. Otherwise it just works. Anybody got ideas? <details<summaryBacktrace</summary</details
You should use to get the expanded version of that and start reducing. I doubt you need serde, just a few traits from it that you could copy over etc. EDIT: disregard me, that makes no sense, the crash happens in the proc macro.
I tried while working on latest comment. It produced this snippet but it does not crash when using it from another crate: I'm available on both Discord and Zulip btw.
I encountered another odd behavior with these crashes when creating a crate specifically for custom benchmarks in my workspace. The file structure is simply the following, with only one binary, because I just started: Running the specific binary like appears to work fine right now, no rustc crashes. However, earlier I accidentally typed instead of , and it crashed on compiling one of the other crates in the workspace. One that was already compiled and cached, at that. Normally it would simply compile the only binary available, but instead it crashes, as if something about specifying the binary name prevents it from crashing. I thought that odd enough to mention.
Actually, I take that back. I was passing the binary name wrong, If I instead do , it also crashes, even with a single binary. This makes it impossible to have more than one binary, so I guess I'm screwed again. One binary magically works, but only if I pass its name incorrectly.
Apparently the presence of some crates can magically allow other (previously crashing) crates to successfully compile. Adding unbreaks things in same cases, probably because of some chain of features reaching all the way down into . Something is seriously broken in rustc, and I'm amazed this isn't getting more attention. I'm betting my personal future on Rust, and it can't even compile benchmarks on my system. Is AMD not a supported platform anymore? EDIT: I apologize. It's easy to become emotional with so much personal time invested in projects.
I works perfectly fine on Linux when targetting znver1. Something is broken in LLVM but it's very hard to narrow it.
I do not recommend doing this.
but it does not crash when using it from another crate: Oops, while suggesting I forgot that the crash is in the proc macro itself. <hr/So far, I think what's happening here is: proc macro () is compiled with LLVM 9 (only on Windows?) miscompiles the proc macro code later crashes when the miscompiled code is executed Ideally we should be able to get a reproduction without part 2, i.e. without the compiler being involved in executing the miscompiled code. This could be done with e.g. . I realized recently that we happen to have one laptop with a Ryzen CPU in the office, if I get access to it I'll post the results of trying to reproduce/reduce this on it.
Confirmed repro on that laptop with and:
If you need something simpler than , I learned this morning that also triggers the miscompilation sometimes: Running error: could not compile . Caused by: process didn't exit successfully: (exit code: 0xc0000374, STATUSHEAP_CORRUPTION)
So far reduced to with and: No more proc macros are involved, eliminating part 2 from However, this leaves the entirety of left to reduce.
I'm now left with , no dependencies other than left, not even on the builtin (which is good, because I'd rather not reduce that). As you might be able to tell from types such as , certain types appear to only matter for their size, and there's also a dance like at some point. Would be interesting to see if I can remove all of the code from the side, because until then it will remain a potential suspect, however unlikely that may be.
EDIT: No, I changed something and that caused different results... but the following is still true It looks as if the comparison of here is being optimized away: if you change it to , that fixes everything. So it could be that the creation of and is undefined behavior.
After lots of fiddling with it, and some false leads, the highest up I can fix it is by changing: to Seems like that is the root cause of the invalid optimization, which was even stated in your last comment . Suppose I should have tried that first.
As of Rust 1.39.0, this is now in Stable.
FWIW I hit this on a Ryzen 3000 processor as well. I couldn't repro the code in . My repro is a bit larger (sorry, I spent a couple hours minimizing as much as I could) but is essentially: On a different note, given tier 1 platforms are "guaranteed to work", I am surprised this was allowed into stable given is a tier 1 platform. I totally understand that resources are spread thin and people are busy, but I just wanted to point this out so that if it was undesired, some thought could go into ways of avoiding release blocker bugs creeping into stable. Thank you for your awesome programming language :)
Does it crash during compilation? If so, it's likely because uses . This does not affect all of , it hasn't been reported on any Intel CPU nor non-Ryzen AMD CPUs and . Part of the reason this sat for so long was that there simply aren't that many Rust developers with Ryzen hardware around (and it doesn't appear to affect CI builds either). Ironically, my coworker has had that Ryzen laptop for months but I didn't connect the dots until recently.
I've spent a bit more time on it and I've updated . There's no more pointer iteration or lifetime juggling from left, so I think that pretty much rules out UB as the culprit. I've also switched to and brought in instead, which seems to produce a consistent instead of . Probably the "best" news is that and I have managed to repro the crash under , on both WSL and native Linux and even on my (older) IvyBridge i7! It does appear to require compiling for Windows, still, though, in order to reproduce.
's reliable crashes let me get farther with the reduction, resulting in this version, which relies on to keep LLVM cooperating, and doesn't even need anymore: (EDIT: replaced the heap corruption crash with an outright assertion failure) (EDIT2: removed the heap allocations completely, without using any ) (EDIT3: made the padding smaller, and replace with a copy) (EDIT4: made an to reproduce on linux + AVX) (EDIT5: replaced with a simpler )
Does it crash during compilation? If so, it's likely because pyo3-derive-backend uses syn. Yes it does! I'm pretty sure that is the reason as well. This does not affect all of Right, I'm aware. But "guaranteed to work" reads to me that stable rustc on that platform should always work, perhaps that is a misunderstanding on my part? Part of the reason this sat for so long was that there simply aren't that many Rust developers with Ryzen hardware around I definitely understand. But my point is that this bug would not have entered stable if it had been marked a release blocker, and more attention would have been given to it. If I had seen people (say on my Twitter timeline) asking for someone to lend hardware with these specifications for debugging, I would gladly have volunteered. Again, I don't mean to sound like what happened was wrong, more that I am making suggestions so that regressions don't enter stable Rust (which I think we can all agree we don't want :smiley: ) Also I'm very glad to hear you've been able to minimize it further! Hopefully that can help in figuring out what the root issue is.
I think I'll stop editing my previous comment (). In the final version I've numbered the variables, because they're all copies with the same value, except I can't seem to be able to simplify their weird scoping without the bug going away. Initially I attributed the sensitivity to scoping to MIR drop order, but with not needing drop, I think it's the MIR , which end up in LLVM IR as . So this could be a stack layout overlap issue? Maybe there's something different about stack frames on Windows which can cause this? Not sure where to go from here. EDIT: looking at the assembly, the frame sizes are definitely different between Windows and Linux, but also the calling conventions and therefore the register usage. I likely won't spend more time on this myself, but if someone wants me to run some testcases or dump some data using the Ryzen laptop I have access to, I can help with that. cc
ping icebreakers-llvm
Hey LLVM ICE-breakers! This bug has been identified as a good "LLVM ICE-breaking candidate". In case it's useful, here are some [instructions] for tackling these sorts of bugs. Maybe take a look? Thanks! <3 [instructions]: https://rust- cc
I'm happy to take this one on. Although I lack the target CPU myself. Can I still progress using cross-compilation, or is that a requirement?
I'd suggest trying my reduced testcase with either on Windows, or on Linux via cross-compilation + Wine (IIRC, got that to work). If you can get the assertion failure and not a SIGILL or some other crash, then I assume that's enough to work with that testcase and dive into LLVM internals responsible for it etc.
If that doesn't work, I have a server with the needed hardware, and I could probably spin up a VM if that helps.
That would definitely do the trick. Pretty sure I should be able to get it to repeat on my hardware though. Let me try during the weekend and confirm on Monday.
I built the on an ASUS ROG with Ubuntu 18.04. The commands I used to build and run on wine were: What should I change to force the error? Somehow it compiles and runs fine for me.
Those testcases aren't useful for reproduction (especially outside of Windows on AMD Ryzen CPUs) as they crash from a proc macro, which is messy. You should only use which relies on an assertion failure instead of a crash (so if you get a crash that likely means your CPU doesn't support certain instructions).
That worked. I needed to switch to nightly and used the following to compile Running it in wine produces following error (which I believe is what would be expected)
I dropped the ball while initially , but then pointed out that was getting trashed (after looking at it in a debugger) so I took a closer look, ignoring (irrelevant) stack offsets. Both use followed by a later use of to refer to that value, but both the and the initialization point of differs: non-windows initializes right before using it windows initializes much earlier it also saves/restores before/after the body of the function (i.e. it's a "callee-saved" register?) there is a call to in between the initialization of and its use, suggesting LLVM has hoisted the initialization past a call because it's a callee-saved register however, there is also a write to in between, which trashes confirmed that is the first callee-saved ("non-volatile registers") register, so that fits, but why is LLVM taking advantage of that when is also in use? (EDIT: ) Also, this finally explains the relevancy of the size! One register is s, so you need at least () s for to be used. Out of those s, of them are in , and are in .
I've just updated (I know, I said I wouldn't) with a small change to make this reproduce on a Linux Intel IvyBridge i7 laptop (i.e. it has AVX). The only reason windows was relevant was the fact that it has callee-saved registers at all, which we can also get by making an .
It is good that you posted here as it reminds me to update my code on my next debugging session with this issue.
With I was able to get to reduce the LLVM IR, which I then cleaned up (mostly because likes to replace values with ) to get this: <details<summary(click to show assembly output as well)</summary</detailsare callee-saved on the calling convention (used by here) LLVM takes advantage of being callee-saved to hoist the constant across the call and out of the (fake) loop, keeping it around in the two copies ( -and -) result in enough AVX registers being used (-) to overlap with , corrupting it by the time of the actual to , whatever happened to be in 's lower half gets stored, instead of the hoisted constant EDIT: reported as EDIT2: and someone wrote a patch already!
Did you ever learn why this only popped up on AMD Zen targets? The linked LLVM patch doesn't seem to touch target-specific code (that I see). I'm mostly asking this out curiosity.
IIRC someone ( speculated a while back that the cost tables for were different enough to cause different instructions/registers to be used. I'm guessing that would be the 256-bit AVX () registers. AIUI, the bug relies on and being distinct registers in LLVM, but which overlap in hardware (and the patch adds overlap handling to one part of LLVM which was missing it). If LLVM uses only 128-bit () registers for s (or calls the C function), then there's no chance for the bug to occur.
Yeah, we talked about it on Discord. has single test and it uses . So the bug was there for long time but was exposed in LLVM 9. I think one of recent optimisations when paired with znver1 scheduler (znver2 uses the same scheduler right now) generated code that triggered the faulty optimisation.
has landed in llvm-project: , but not yet in .
thank you all for your hard work on fixing this!
The fix will be available in the next nightly.

Discussion:
<del> Subproject commit de1a7dbf6c6b34f56e65732d45970ff27a8e84bf </del> <ins> Subproject commit 2cb41005ed5c4747b10d2bf01d8779d3bb4ae32d </ins>