Change:
Update 2021-12-27: More minimized MCVE (also linked from .) <!-- Thank you for filing a bug report! Please provide a short summary of the bug, along with any information you feel relevant to replicating the bug. --I tried this code (note that the crate name needs to match its usage in the doc test to reproduce the bug properly): /// use a64_doctestfail; /// And then ran . I expected to see this happen: (Working behavior from 1.56.1, witnessed via nightly-2021-10-13) Instead, this happened: (Broken behavior from 1.57.0, witnessed via nightly-2021-10-14) : rustc 1.57.0-nightly ( 2021-10-13) binary: rustc commit-hash: commit-date: 2021-10-13 host: aarch64-apple-darwin release: 1.57.0-nightly LLVM version: 13.0.0 rustc 1.57.0 ( 2021-11-29) binary: rustc commit-hash: commit-date: 2021-11-29 host: aarch64-apple-darwin release: 1.57.0 LLVM version: 13.0.0
(by the way, the reason that I didn't include output above is that the tool was not working for me today for this target; some kind of failure in its attempts to download the pre-built binaries per-commit. I haven't looked in more detail into it; for all I know, maybe we don't keep per-commit builds for aarch64...)
Here are the relevant commits over the period where this bug was injected: : Auto merge of - oli-obk:nllmemberconstraint_diag, r=estebank Auto merge of - tmiasko:overflap-duplicates, r=cjgillot Auto merge of - JohnTitor:rollup-xfymeo4, r=JohnTitor Auto merge of - camelid:all-candidates, r=petrochenkov Auto merge of - ehuss:update-cargo, r=ehuss Auto merge of - the8472:inline-mut-iter-next, r=m-ou-se
(it would be good to further minimize this. if possible I'd like to rewrite it as a normal test rather than a rustdoc one, if possible.)
can you run and post the output? I think rustdoc is hiding the actual compile error.
It seems strange this only happens with doctests - does it also happen if you add a second crate that depends on (without a doctest)?
Bisecting worked fine for me. The regression pinpointed to the cargo update <details<summarybisected with <a href=''cargo-bisect-rustc</av0.6.1</summarysearched toolchains through Regression in searched nightlies: from nightly-2021-10-13 to nightly-2021-12-08 regressed nightly: nightly-2021-10-14 searched commit range: regressed commit: Host triple: aarch64-apple-darwin Reproduce with: </details
Adding alone does not seem to help here. I think you are right that is hiding the actual compiler error; but I also think that we need to go deeper to dig out the root cause. (I do wish that did resolve it, or some better still, some set of flags...)
I figured out how to use to feed in a wrapper script around that also emits the arguments that were passed into . That should hopefully be enough for me to narrow this down to a bug that doesn't rely on for us to replicate it.
Also, since I'm staring at a bunch of verbose command output, and the problem was already bisected to a cargo update ... maybe the real culprit a change to how is being invoked by here? In 2021-10-13, the invocation from cargo looks like this: In 2021-10-14, the invocation from cargo looks like this: Leveraging Emacs ediff-mode, these invocations have only three differences: 64doctestfail- is rewritten to liba64doctestfail-, and is rewritten to We can probably safely disregard these two differences, which are just artifacts of how we generate filenames. old version has new version has So... did start passing along into invocations and it is exposing some underlying bug from LTO?
Aha, I bet its fallout from this new cargo feature rust-lang/cargo
Running the 2021-10-14 invocation manually without does indeed pass the test. So this seems like it is probably exposing some LTO-related bug, one that may well predate nightly-2021-10-14. Its non-trivial to try to test this thesis against the 2021-10-13 build, though, because its not simply a matter of adding to the invocation at the end in that case. If you do try that, then you get the error: So I'm guessing more parts of the build recipe would need to be revised to pass (or at least ?) And I think before I put in that kind of effort, I'm first going to investigate whether we can reduce this to a smaller MCVE (preferably one that doesn't need to pull in ...)
The problem isn't tokio-specific; I was able to get the same result with an MCVE variant that used Rayon instead of Tokio, with code like this: but its does not duplicate with just any old crate either. (I tried using an ancient demo crate of mine, , and that was not sufficient to witness the problem here.
did you make progress with your idea to use a wrapper script to get the rustc args? That seems like an easy way to avoid making this reliant on rustdoc (i.e make it possible to reproduce with an older version of cargo).
I was indeed able to get wrapper script to print the rustc args. But since then, I've decided to focus on first reducing the MCVE further. E.g. at this point I've found that just an declaration suffices to reproduce the problem; no necessary at all, nor any imports from the crate. I want to try to minimize the set of crate dependencies first, and then I'll switch back to extracting the series of invocations (which will hopefully be teeny tiny by that point...)
label: +P-high -I-prioritize
Okay, I've minimized it to a tiny fragment of rayon with no further dependencies beyond. I still need two distinct crates (and the rustdoc invocation), but that's mostly because I wanted to get to this point before I looked more into reducing that aspect of things. (Well, that, and the fact that its an LTO bug pretty much implies that an MCVE is going to require multiple crates.) Here's the repo where I've checkpointed my work:
Okay, I was successful in getting (and ) entirely out of the picture. The result is that itself is segfaulting during the invocation on this line: I'm going to keep investigating. It seems very likely to be some sort of issue with LLVM.
The segfault appears to have been injected between nightly-2021-08-21 nightly-2021-08-22.
Skimming over that, I'm betting this is connected to the LLVM 13 upgrade in .
Sweet! I generalized my script to support cross-compilation, and now I can reproduce the seg fault on Linux x86_64! This means I get to use to debug the segmentation fault!
FYI. (I haven't made a local build with better debugging info yet. But the stack trace seems to solidly point to LLVM, as previously hypothesized.)
Rebuilt Rust with an LLVM Debug build (I always forget that its not enough to just edit the section of the , you need to explicitly force LLVM to be rebuilt). I hit the assertion: Here is a for the bug. I'm going to spend a little while poking around in the session. If I don't find an obvious bug that way, then I will see if I can make a standalone LLVM test case.
Part of the problem is that I could not readily tell if that Debug assert was related to the root cause of the bug in play here, or is an artifact of some other unrelated issue. So I made a quick hack and put this in: (this side-steps the assertion failure by not doing the call on the instruction at all when the two registers are the same. In theory this should have no effect on the semantics of the program output.) After doing this, I again get a segfault from the resulting build of itself. (Its still possible that the Assert does have the same root cause, or a related one. But I'm hoping I will have an easier time identifying the actual root cause of the segfault itself this way.) This is a link to a new session (still being built as of this writing) that uses a binary with the LLVM diff above:
Looks like from LLVM is returning nullptr, and that line immediately dereferences it. (if that link makes you log into github, do so, and then go back and follow the link again; from my experience, the much of specific context is lost during the github login process.)
Okay, newest update: The pernosco links above are not as useful as I would like, because they do not have any logging output from LLVM, so its hard to understand the intermediate states of things as you work through the code. I tried doing runs with an debug-enabled LLVM and passing the flag as an LLVM arg. This, on its face, caused my Rust-debug-noopt build time to go from 48 seconds to, um, five days or so. In parallel, I followed 's advice to try making a nocore version of the MCVE, in hopes that would reduce the build times sufficiently that LLVM would be worth while. Long story short: It took a day to make a nocore version. It takes less than 2 seconds to build! (An intermediate nostd version took 20 seconds to build.) Passing the LLVM arg to this variant does not blow up the build times the way it did on the original example. The new MCVE is at ; it is now two files, instead of three. And, here is the newest , now with LLVM debug output. The above trace is built with a slightly editted version of LLVM. Namely, I some assertions that correspond to checks that we end up failing later on in the control flow. The problematic instruction seems to be this: (I do not know what that syntax means, namely the part. Continuing to dig. update*: oh, the is probably a type-ascription note, where signifies its a signed 64-bit integer. I.e. I don't think its relevant to the core issue here.)
I was just debugging rayon-rs/rayon that also bisected to the cargo update in , which led me to this issue. This is another case of rustdoc + LTO failure, but instead of a compiler crash I've got bad codegen that crashes at runtime. I'm hoping it's the same root cause in LLVM that we're both looking for, so maybe that different angle will help.
(I'm going to count the MCVE from as good enough to remove the E-needs-mcve from this ticket.) label: -E-needs-mcve
Sweet, massively reduced the test case further (from LLVM's perspective) and has a candidate patch up; see progress over on llvm/llvm-project
On the LLVM side, this was fixed by i'm going to see about cherry-picking that into the rustc local fork of LLVM.
(I'm not sure there's much value in encoding this specific reproducer as a regression test for rustc, since its so tightly coupled to LLVM-IR details... so I didn't include a regression test in the PR . But I'm open to counter-arguments...)

Discussion:
<del> Subproject commit 221a195b62f4373a8893f4d022e4ce981b09a857 </del> <ins> Subproject commit b6b46f596a7d2523ee1acd1c00e699615849da60 </ins>