Code change:
<!-- Thank you for filing a bug report! Please provide a short summary of the bug, along with any information you feel relevant to replicating the bug. --We have this code in our project on commit from line 44 and beyond: I expected to see this happen: running the doctest with the following command should work (note that we modify the release profile to have lto = "fat" enabled): Instead, this happened: the program crashes, compiling the same code in a separate example and the same cargo configuration results in an executable that works. Turning LTO off also makes a doctest that compiles properly, indicating LTO is at fault or part of the problem when combined with doctests. It has been happening randomly for doctests on a lot of Rust versions but we could not identify what the issue was, looks like enabling LTO creates a miscompile where a value that is provably 0 (as it's never modified by the code) is asserted to be != 0 and crashes the program, sometimes different things error out, it looks like the program is reading at the wrong location on the stack. The value being asserted != 0 is in Unfortunately we are not able to minify this issue at the moment as it's not happening reliably across doctests. <!-- If you're using the stable version of the compiler, you should also check if the bug also exists in the beta or nightly versions. --: Unfortunately on nightly (used to recover the doctest binaries via RUSTDOCFLAGS="-Z unstable-options --persist-doctests doctestbins") only exhibits the crash for the parallel version of an encryption algorithm used with rayon (on current stable we can also get the crash with a serial algorithm but we don't seem to be able to recover the doctest binary). The archive contains the for the code compiled as an example (running fine) and the code compiled as a doctest exhibiting the miscompilation, if needed I can provide the binaries, but I would understand if nobody would want to run a binary coming from a bug report. Here is a snippet of a backtrace with two threads erroring on two different issues (while there is no problem having the same code compiled as an example). <!-- Include a backtrace in the code block by setting in your environment. E.g. . --<details<summaryBacktrace</summary<p</p</detailsWe have also seen some flaky doctests on x8664 and could not narrow down the issue, we have turned off LTO for all of our doctests for now and we will monitor how things evolve, the reason for the suspicion of an issue on x86 as well is that M1 builds have been running with LTO off for months and have never exhibited the flaky doctest we saw on x8664, though given the compiled code in that case is significantly different (intrinsics usage being one factor) we can't yet be sure a similar issue is happening on x8664. Cheers
have you run the tests with miri to make sure that no UB is happening?
Hello not yet, I guess it's the occasion to use it, I'll report back
Should I try it on the example reproducing the doctest or is there a way to run it on doctests ?
I forgot whether Miri runs doctests. But you can just extract the doctest into a binary and run that.
Hello no undefined behavior found by MIRI, it seems rayon does not terminate its threads so MIRI detects that but the doctest taken out as an example is UB free (I adapted our crypto parameters to be very small but still trigger the crashing doctest). Cheers
adding I-unsound as this looks like a miscompilation. The next step would be to minimize the issue and create a smaller reproducer.
agreed, for now it's a bit hard as we used to hit that a bit randomly until we identified it seemed linked to doctests + LTO, we will try to find a way to minify it, in the meantime the assembly in the original report should already contain the miscompiled code (though I understand it's likely way too big for a reasonable analysis) Cheers
FWIW, a pattern I've seen a few times with doctests + fat lto is that doctests are unoptimized, so you get optimized IR from LTO fed into the backend with disabled optimization. This can expose bugs in FastISel that we don't otherwise hit. The key to reproducing it outside doc tests may be to use an lto=fat opt-level=0 configuration.
Thanks will give it a shot, I'm surprised doctest don't honor opitmization/configuration from the Cargo profile though any reason for this? If the various parts are not supposed to have mismatched opt levels then yes I can see how some hypothesis done would break at the junction of the optimized code and unoptimized one
For the doctest as an example binary with the lto=fat opt-level=0 configuration I cannot get it to crash
setting the opt-level=0 in release allows the doctest to pass as well, so a mixing of LTO and non 0 opt level seems to trigger the issue for the doctest, trying to minimize the repro case
setting opt-level=1 in release with LTO=fat crashes the doc test
Hello again, some news, I have a state where I'm able to trigger the crash at will with a single line being uncommented. the repro branch is here MIRI is still OK on the doctest taken as an example. The interesting part is in here, the line below will cause the doctest to crash when not commented, i.e. that function call has the issue, when commented out and inlining the function (i.e. the rest of the doctest below) the execution goes well. I will attach the dump of the assembly for both. nokfunctioncall.S has both the call to the function and the inlined version okfunctioninlined.S only has the inlined version run MIRI Run doctest
still feels like the wrong part of the stack gets read in the function where a value should be a 0 and something else is read from who knows where edit: will try to minify some more
With some logging from the inlined non bugged version Edit: the "MIRI run done" was a check for the MIRI example, I share the code between both, both runs here are normal runs from the function call bugged version looks like the first iteration of a loop is fine but then the data gets corrupted here the native modulus 2^64 is encoded as 0 in a u128, so the 2^64 are valid, but the is random data and seems to be changing from run to run
looks similar in spirit to with the sensitivity to opt levels, though I'm not familiar with the mir opt level
Hello, minified the example to an iterator call returning corrupted data when a specific feature (on which the code does not depend) is enabled. Disabling said feature makes the code run properly, changing the iterator to be an immutable iter does not cause the issue () The iterator in question is a wrapping lending iterator that is defined in , immutable variant is here Available here Run the doctest and crash it : Run the doctest and does not crash : Tried to take the code out to a different repo, it did not repro Let me know if there is something more I can do, but here I can't seem to minify it anymore than that at the moment
I can reproduce the crash on aarch64-linux after also enabling the seeder_unix feature. I'm not sure how to debug though -- it doesn't seem like rustdoc has any support for that at all? does nothing and neither does . How do you even get the executable it generates? Passing to rustdoc does fix the issue, so this looks like another "passing O3 IR to O0 backend" issue.
I do the following (notice RUSTDOCFLAGS) : then and copy the only executable found in there Yes I agree with the O3 thing, I just don't quite get why rustdoc does not use the configuration from the cargo profile provided in the command line Edit: I'm guessing it's not an easy problem and there may be a reason for this
should there be a rustdoc specific bug report somewhere ? I have to say I posted here mainly because I found an old issue that looked similar
Thanks, using worked! I've used this LLVM patch () to identify as the miscompiled function. This is the extracted IR: Yes, I think that would be a good idea. I think this is probably a cargo bug, as doesn't get passed to .
Thanks a lot and great that you could find the faulty function! Alright then you advise opening an issue on to let them know of the opt level not being forwarded ?
Upstream issue: Yeah. It might be intentional, but it seems suspicious to forward but not .
WG-prioritization assigning priority (). label -I-prioritize +P-high

Related discussion:
text: <del> Subproject commit febc39711a7c91560eb0f0980916ae23c343b99d </del> <ins> Subproject commit fef3d7b14ede45d051dc688aae0bb8c8b02a0566 </ins>