jiebi/CodeConvo huggingface dataset:
Now that has been merged, only a few checks remain to be in order to stabilize this feature. First, we need to actually perform those checks in the compiler directly instead of rustdoc. You mentioned other checks too ? This issue is part of .
We should reject on items that don't actually appear in the search index such as blocks and s.
Ok, makes sense! Last question: I remember you telling that you'd prefer this check in rustc directly instead of rustdoc. I can move things there, just wanted to check one last time before doing it.
Yeah, rustc validates some attributes in and it might make sense to validate the attribute there as well.
Since has been merged, we can close this issue.
We're still not rejecting on associated s in trait impls like:
Ok! Putting on hold then!
I have no idea what's causing this, but on the current nightly, this fails: with "use of unstable library feature 'threadlocalinternals'." Enabling the feature works, but this should be a stable function. Interestingly, tweaking the works: I wonder if it's because there is a (private) sub-module called which has a module-wide marker. Version info:
This change was introduced just recently: I reproduced the problem without using anything thread-related: Commands: Output: If I remove the attribute from the module, the code gives me the error. I mark as stable, then the code compiles. If I instead remove the function, I get the bar error. cc
With merged, is this fixed?
The exact issue at hand has been resolved, but the underlying issue remains. Specifically a module and free-function of the same name but different stability level will cause problems.
Also hit by this; still affecting the nightly for osx right now, but hopefully the next build will include the fix.
Stability is an internal detail. This is unlikely to impact out-of-tree code.
liballoc's fails when run in Miri. The error occurs in the call: is called with a as argument. contains contains a shared reference to a slice; that slice is thus marked as "must not be mutated for the entire duration of this function call". calls calls calls calls , which eventually drops the . calls to re-arrange stuff (I have not fully understood this yet), and in some cases this will end up writing to memory that the slice in points to. I am not sure what the best way to fix this is. We have to fix holding (indirectly) a shared reference to something that it'll mutate during its . The mutation is likely there for a reason, so I guess the shared reference has to go. (FWIW, this shared reference already caused other trouble, but that was fixed by ) Cc <!-- TRIAGEBOTSTART --<!-- TRIAGEBOTASSIGNSTART --<!-- TRIAGEBOTASSIGNDATASTARTTRIAGEBOTASSIGNDATAEND --<!-- TRIAGEBOTASSIGNEND --<!-- TRIAGEBOTEND --
yeah I guess just change it to and toss in a PhantomData for the borrow. shrug
You mean, change this in as well? Sure, that should work. In principle this reduces the precision of compiler analyses on .
ahh misread. maybe would just not use Iter then. depends on what the impl looks like now
But the first thing it does is which should exhaust the such that its is an empty slice. I suppose its pointer will still be somewhere in the deque's memory, but with length 0 it can't actually read anything. Is that not good enough to appease Miri?
No. This code still matches the pattern There is no notion of "exhausting" or "consuming" a shared reference. When you create a shared reference with a certain lifetime, you promise that until that lifetime ends, no mutation happens. here is mutable so you can change what the slice points to, but that has no effect at all on the guarantees that have been made about the value that was initially passed to . Now, lifetimes are an entirely static concept, so for Stacked Borrows a "lifetime" generally is "until the pointer is used for the last time".
However, for the special case of a reference being passed to a function, Stacked Borrows asserts that the "lifetime" will last at least as long as the function the reference is pointed to. (This is enforced by the "barriers" explained .) This is extremely useful for optimizations.
forward to . I wouldn't want to copy those.
Oh, I also didn't realize that doesn't actually shrink down to an empty slice -- it only updates its indexes. It would otherwise have to use two slices for the wrapped head and tail. So that full shared reference does still exist after the iterator is exhausted. (This also raises the validity question of having references to uninitialized data.) Could we base on instead? We could arrange to work through that mutable reference too, so there's no question of mutable aliasing.
That might work. However, must not access in that case -- that would be in conflict with the unique reference in .
Hmm, does need to be covariant over ? I guess would break that.
Actually this idea of "consuming a shared ref" is far from unique to ... the following code also errors in Miri:
As I understand it, Miri and stacked borrows should be "more powerful" than the Rust borrow checker, right? So, is this an example where Miri does not match what Rust allows or is this a bug in Rust that could lead to undefined behaviour in safe code? ()
-- I believe is admitting that example as something that should be allowed. But this issue with involves raw pointers, which rustc doesn't check at all, so miri is in uncharted waters here.
This is up for us all to decide. I can't tell you. ;) Not necessary. I have argued that / are problematic long before Stacked Borrows. The lifetimes there are, strictly speaking, wrong. I think it would be reasonable to rule that they have to use raw pointers.
FWIW here is a stand-alone version of the testcase to reproduce this with Miri (but it needs access to some private internals):
See for the issue about whether Stacked Borrows should allow such code or not. For now, with , Miri accepts this code again. So I am going to close the issue here. It still represents an interesting data point to inform what Stacked Borrows rules should be.
nods wisely just gotta stand still long enough for the spec to bend around you
;-) I don't think the decision is made either way, but for now with several counterexamples in libstd, it is not helpful for Miri to complain about this. Plus, this matches what we formalized in Coq; the basic optimizations that we considered so far are all fine with this change.
See there actually is a bug here right now, because we do emit for passed to a function. So what Miri found is actually UB in the LLVM IR that we are generating right now. Reopening for now, but maybe it makes more sense to treat this as a duplicate of
Smaller reproduction that I found in (a now closed dupe of this issue) that doesn't rely on VecDeque internals, that fails now with
re-enables the part of Stacked Borrows that got disabled (in parts) to work around the issue, before I knew that we actually have here and are risking real miscompilations.
claim
In commit I had to disable one of the rope tests. The purpose of that commit was to fix up a bug in the way operator [] works on strings. In particular it used to let you index up to and including the trailing null; now it considers access to the null byte an index overrun and fails. I tried to figure out exactly what was going on in the rope code for quite a while, and tried adjusting a number of obvious-looking things, but to no avail. I'm not really good with boundary-case arithmetic (are any of us?) and the rope module is a maze of such arithmetic. Sharper minds requested to find the bug. Meanwhile I marked the rope test in question () as ignored. The others seem to work.
Paging
I'll take a look.
There is some discussion on IRC that maybe we should just pitch the rope code as it exists now.
agreed. The code is pretty outdated and the work required to modernize it would probably be similar to the work needed to just re-write it. When I was looking through it, it seemed to be doing some very strange things that I don't think are even close to being safe, so there's that too.
These include: The reasoning for all of these extensions is that they all have wide use not just in GitHub, but in many applications of Markdown.
I'm not sure if automatically creating links from bare domain names is a good idea, so if there's an option to disable that specifically I'd suggest it. The rest all seems fine to me.
Just like said for Autolinks. I'm also not sure to see the point of the task list one. However, adding strikethrough would be nice.
Tasklist would be useful for things like this: [x] Implemented thing [x] Some other implemented thing [ ] Unimplemented thing [ ] Some other unimplemented thing or possibly in the crate root [x] Webhooks [x] Websockets [ ] Websites [ ] Other
Another noteworthy extension to implement is . This would prevent the following tags from being used for safety reasons: - - - - -
We decided in not to add this. Instead rustdoc will now suggest to turn raw links into automatic links (). So I think the only thing left is adding task list parsing.
The following program causes UB: Miri says: The problem is this code: This calls unless the new code point is in , but what about surrogates in ? This code is unchanged since its introduction in I am not sure what the intended safety contract of is. That method is not marked but clearly should be -- it calls . So my guess is the safety precondition is that must not be part of a surrogate pair, but the thing is, calls it without actually ensuring that condition. The condition it does ensure is that the codepoint is not in , but that does not help.
is excluded from Unicode scalar values a.k.a. , but it’s fine as a Unicode code point. I think this bug is not in the contract of (which is private anyway) but in its implementation, namely going through . The fix is to either duplicate the logic of into , or (to avoid duplication) move that logic to a new function in libcore that takes a parameter and that is called by both and . This function would need to be public because is in a different crate, but it should be prema-unstable and .
I would classify the priority of this bug as low. Although this call to is UB, I expect Miri checking for it explicitly is the only case where that has any consequence in today’s implementation: behaves as expected (for the purpose of WTF-8) in the surrogate range and does not exploit this UB The "layout" of in rustc excludes values beyond but not surrogates:
By the way, the docs at feel insufficient: Which values are invalid? Similarly, and say that represents Unicode Scalar values, but I couldn’t find it documented anywhere that constructing an out of range is Undefined Behavior rather than "merely" a logic bug. Is this specified elsewhere? What is the closest we have to a written down normative resource that specifies what is or isn’t UB in the the Rust language? (For APIs I assume this is the responsibility of their respective doc-comments.)
I agree the impact is low, but it's not just Miri -- this blocks , which is how I discovered the problem. That would be
But then what is "unchecked" about it? Elsewhere in that file the code is careful not to construct a from a without checking for surrogates: That made me assume the author was aware that surrogates in are UB.
The type contains a (generalized-UTF-8-encoded) sequence of code point (including potentially surrogates) that doesn’t form a surrogate pair. doesn’t check for surrogate pairs. is a public method that is documented as returning a scalar value, therefore excluding all surrogates. Ok I looked into it, there are three different people involved. The original code in 2014 was YOLO Later in 2014, moved to duplicating logic instead. In 2015, PR which first imported WTF-8 support in libstd deduplicated that logic by adding a function that takes . (I knew that approach sounded familiar…) In 2016, PR changed the signature of and in passing removed . It made use + instead, presumably in the middle of updating many callers other.
Okay, so the fix would be to revert the part of that removed ?
That or copy into . I find them both not great, so meh.
Even with that done, there's still UB in another testcase: This is converting something to a before calling -- so probably the same issue as the above, just for a different encoding.
Interesingly, and shockingly If you remove and , this doesn't panic. <details<summaryWhats up with the diff? </summaryThe assetion format is differnt to usual as I've patch rustc with to make it readable, but this can also be repoduced with the latest nightly </details modify labels: +A-rustdoc-json +T-rustdoc
MCVE: One has while the other has - I bet one is going through HIR and the other is going through metadata, that could explain why the generic parameter matters.
I have a similar MCVE generated from , but the only item that differs in the assertion is the visibility: The left has a visibility of , and the right has a visibility of . A similar MCVE, with as a parameter, has as an input for the right side, but not for the left (and the / visibility issue):
Right now, the docs produced on the stable and beta channels include unstable items. The docs are already fairly complex and contain a lot of information to sort through, and finding documentation for items that are not usable for the majority of developers working on stable Rust does not help with this. In addition, unstable APIs have very attention-grabbing warnings associated with them, making them much more prominent than the APIs that are actually usable. <img width="1031" alt="Screen shot demonstrating eye-catching unstable warning" src="https://user-"This has been broken out of to provide smaller, actionable items that can be independently discussed and worked on. A review of rustdoc for accessibility, especially for dyslexia and attention disorders, finds that there are a number of visually distracting elements in rustdoc which can draw attention away from the main content. This should be considered along with the other issues mentioned in about the overall effect of a number of distracting elements on the page. This has been previously discussed and closed in and . However, when it was first discussed was very shortly after the Rust 1.0 release, when there was a large amount of API surface which had not been stabilized for the 1.0 release, and what could be done with stable Rust was fairly limited (for example, no stable proc macros so Serde was only usable on nightly Rust). At this point, most people can use stable Rust for most tasks. There is a lot less reason for the average Rust developer to need to switch to nightly to use unstable standard library features. There is also the open issue that the stable docs of unstable items may be out of date; because these items are unstable, they can change in nightly releases, so not only are they a distraction, but they may be misleading for people who actually try to use them. Between the extra information that most people don't need, distracting UI, potential for being out of date, and time since this was last discussed, I think that it is worth reconsidering this decision and hiding unstable items in the stable docs by default. This would require either providing an opt-in setting for seeing them, or make finding the nightly docs easier. An alternative discussed in the comments is to just make them less visually prominent; unstable items are already rendered at lower contrast and at the bottom on module index pages, but within struct or trait documentation they have big bright warning boxes calling attention to them.
Making them less visually prominent, such as by reducing their contrast and making the warning just plain unstyled text, perhaps in the same place that stability information goes (though see for discussion on that), would help out a lot.
I would prefer this option. One benefit of showing the unstable APIs is that it invites people to test them, so having an opt-in for the curious would be good.
I would prefer an opt-out to an opt-in, that is, keep what we do today, but provide the ability to hide it if desired.
After spending some more time browsing the docs with an eye out for things that are visually distracting or attention grabbing, another option would be to just use a less attention grabbing way to denote unstable and deprecated items. Right now, they each use an extra, brightly colored box that makes them take up more visual space on the screen than stable items. On module index pages, they are actually made less prominent; they are sorted to the bottom, and made lower contrast with the background. This helps make them less attention catching, rather than more attention catching. But when browsing the methods provided on a struct or trait, they come with the big bright warning boxes that make them stand out more. I think I would still have a slight preference for having them hidden by default and just making the nightly docs easier to discover, such as having a version chooser in the header or sidebar, because of the issue mentioned in about docs of unstable items potentially being out of date. But making them less visually prominent rather than more visually prominent would help a lot.
I agree that the better way to solve this is by (perhaps) reducing the prominence of the "nightly-only" box. I don't think we should hide unstable items in the stable rustdoc generated at If someone is reading code that uses an unstable method, there's a good chance they will go to look it up in the stable docs. It would be a frustrating and confusing experience to not be able to find the method in the docs. It's a much better experience to find the method and be informed it's nightly-only / unstable. We probably don't want to reduce the prominence too much, though. If someone's looking for a method to call, it's pretty important that they notice when it's nightly-only. If they're working on a stable crate, that may mean they're not able to use the method at all.
Candidate PR at
Here is the error Here is the code that generates the error My system is x86_64 archlinux. The rust compiler version is Let me know if you guys need anything else. (The code is really short so it shouldn't be hard to just type it in to duplicate the bug)
It only needs one line for me
Workaround (in the sense that the compiler no longer ICE's): I think there is some unfortunate interaction here with the way we attempt to infer which kind of numeric literals are in the array (, , etc) and other code not being prepared to handle the that results when you don't have that information available.
This is almost certainly a duplicate of (but maybe the integer inference codepath is slightly different?).
This also causes the same ICE -
Similarly:
Looks like it's been fixed.
Versions: Code to reproduce the ICE: Compilation output:
Seems to be fine now but potentially needs a test.
Nope, still failing. An updated example:
That code example fails to compile, but it's because the function signature for is wrong. It should be: When you make that change and make the struct fields public, the code compiles fine. Without the change I also didn't see any internal compiler errors. Rust Version: 0.12.0-pre-nightly, , Linux x8664
Yeah, I think this can be closed.
Flagging as needstest (not sure if one is checked in already)
After discussing it on the forums I bit I decided to look into what the optimizer spends time on, and inlining seems to be a significant part of it. There seems to be a lot of room for improvement here. Tested by compiling syntex in rustc-benchmarks (though the inlining time seem quite significant in other crates as well.). I counted which functions were inlined the most, and and to a lesser extent , which are when automatically deriving , and which simply wrap two other functions inside fmt::builder show up a lot. Maybe derive(debug) could output the wrapped functions instead, or the equivalent code, so llvm doesn't have to spend time inlining them (unless this causes problems for debug info). is also inlined a lot, and also plays a role in derive(Debug). Maybe something could be done about that one as well, as it's that creates a struct but doesn't do anything else. <details<summaryOutput from time-llvm-passes</summaryUsing in the syntex-0.42.2 directory in rustc-benchmarks. It's a bit hard to say which crate is which, as the output from llvm and cargo isn't synced, though these are the bottom three (excluding the benchmark dummy) in reverse order: </details<details<summaryMost inlined functions: </summaryRunning , and filtering the output using this: (Yeah it's a bit of a monstrosity, and takes a long time, maybe I could look into making some tool analyze this if it turns out to be useful.)
gives: <pre1865 ZN4core3ptr13dropinplace17hffb7ed55089717f8E inlined into 917 ZN50Tascore..convert..FromTGT$4from17h99c56b7d2123957eE inlined into <b677 ZN4core3fmt9Arguments6newv117h0a32116e8ade096aE inlined into</b549 ZN4core3ptr13dropinplace17h8905708a4bea3261E inlined into <b517 ZN4core3fmt9Formatter11debugtuple17h312acfb009988478E inlined into</b484 ZN50Tascore..convert..FromTGT$4from17hf6536c6dc214691bE inlined into 438 ZN5alloc9allocator6Layout4size17h278602cb6317171dE inlined into 402 ZN63alloc..allocator..LayoutasGT$5clone17hba86f6db001903cdE inlined into 381 ZN61u20u20GT$7dealloc17h3db9762b4fdfd21eE inlined into 344 ZN5alloc4heap15exchangemalloc17he9800f033a6330e4E inlined into 341 ZN4core3ptr13dropinplace17hb5325e51b98a161aE inlined into ... <b82 ZN4core3fmt9Formatter12debug_struct17h5cfb920e90696799E inlined into</b... </pre</details<details<summaryrustc --version</summary</details
We should just drop the from those functions - debug printing is something we want to be optimizing on code size/compile time rather than perf.
Yup, that's a good point. EDIT: Actually since they're just generic over lifetimes it looks like that's not an issue.
<!-- [31mICE[0m: Rustc '' 'thread 'rustc' panicked at 'assertion failed: !self.intercrate'', 'thread 'rustc' panicked at 'assertion failed: !self.intercrate'' File: --auto-reduced (treereduce-rust): original: Version information ` Command: <!-- Include a backtrace in the code block by setting in your environment. E.g. . --<details<summary<strongProgram output</strong</summary<p</p</details<!-- query stack: [specializationgraphof] building specialization graph of trait [coherent_trait] coherence checking all impls of trait -- label +F-Unsize +F-unsize
The bisection looks incorrect? is from 1.77, and this looks to be from 1.72 to me.
Too lazy to look at its PRs :
Due to . This assertion is correct, and we probably shouldn't drop candidates in intercrate mode, since this is incomplete. I would say this fix is low-pri because of the fact that is not expected to be referenced.
and are both broken on s390x since Rust 1.28. When is built without debuginfo, the tests hang with their backtracing processes stuck in a loop. With debuginfo, it fails with output like: I believe the problem is this part of : where doesn't identify that is a 64-bit platform. See also alexcrichton/backtrace-rs. I plan to update that here when it's published, and make a similar fix in .
fixes the backtrace when is built with debuginfo, but I'm still getting a hang without. In GDB, it looks like the unwinder gets to the that catches panics, then loops unwinding that same PC forever. This part of the regression is older -- happens on 1.27.2 as well, but 1.26.2 is fine. Binaries from the older version don't appear to have a distinct symbol at all, but maybe that's just inlined differently. I'm bisecting for that hang now.
Bisecting pointed me at (cc which attributes to all functions. For s390x the base features actually include a removal, . But it doesn't look like the features are applied to the synthesized (), and as a result it doesn't get inlined anymore since it has more features (w/ ) than the callee. We probably should apply target-features to as well. I don't know why the non-inlined can't be unwound -- maybe we need to force ? By default GDB backtrace stops at crate , but tries to go further: Maybe Rust's could have a similar defensive check, at least.
Nice find Definitely makes sense to me, and this should be fixed up (at least the business) in
With , , and , I believe we have this fixed.