jiebi/CodeConvo huggingface dataset:
Code change:
#![feature(unsafe_cell_raw_get)] #![feature(unwind_attributes)] #![feature(vec_into_raw_parts)] <ins> #![feature(vec_spare_capacity)] </ins> #![feature(wake_trait)] // NB: the above list is sorted to minimize merge conflicts. #![default_lib_allocator]

Related discussion:
text: I was using std::io::copy from an SSD to memory using , which reported an eye-raising 30 MB/s. Wrapping it in a BufReader (either as the outer or middle layer) made no real difference. So I googled around a bit and found the recommended size was 64KB according to some random Microsoft article, so I decided to copy-paste the std::io::copy implementation and increase the buffer size. This got up to 600 MB/s at about 256KB which sounded a lot more it continued going upwards from there past the point of believability (unless the files were being pulled from RAM as a consequence of my repeated accesses). With a 10MB Vec, I got 228.36 GB/s . Unrealistic values aside, has anybody checked the buffer value to make sure there isn't some huge low-hanging fruit for std::io::copy speeds?
You mention windows. This may be platform-specific behavior. On unix systems I would fadvise on the input file for copying. Googling tells me that the windows equivalent is . Have you tried that?
After a coarse overview, my intuition is that there is some funny business going on with the length determined by and the subsequent calls on the , that could explain the unrealistic results.
It’s possible. The actual times did increase dramatically from the std::io::copy default though - my suspicion is that there is probably some caching to RAM going on or something as cold runs were generally slower than hot runs. This was for some work code that I’m no longer actively developing. I’ll leave this email marked as unread for awhile in case I have some spare time to write a benchmark. I don’t think the other flag that was mentioned is accessible with the default open file api, but in any case it at least appears there’s some low hanging fruit wrt to std::io::copy perf
I tried benchmarking on Windoze before and after changing to (instead of the original ) and the results were slightly worse (up to ~3%) with the modified setup, both on HDD and SSD. However, I'm not sure if my benchmarks were good enough - I was having a hard time tweaking them so that I could see any difference between results for different file sizes.
Code change:
#![feature(unsafe_cell_raw_get)] #![feature(unwind_attributes)] #![feature(vec_into_raw_parts)] <ins> #![feature(vec_spare_capacity)] </ins> #![feature(wake_trait)] // NB: the above list is sorted to minimize merge conflicts. #![default_lib_allocator]

Related discussion:
text: FWIW, I'm providing the benchmark code below; it can easily be expanded to different file sizes:
4MB isn’t a whole lot. That’s well within the size of hard drive caches or possibly an OS memory-based cache. I was noticing the difference with files several gigabytes in size. As a sanity check, maybe look to see what the actual time is, not just the % difference between timings with different buffer sizes. The actual time may be wildly different than the actual steady state performance of the disk if it’s reading the data from a cache. It may also be possible at a low level to pass a flag to encourage the OS to disable or flush the cache, but I don’t know if there’s any way to instruct the hard drive to do the same.
I only pasted the benchmark for 4MB as a reference - I tested files from 4KB up to 16MB. I'm not an expert with low-level hard drive stuff, just hoping my attempt can help someone else.
Ideally you would test with a file approximately twice as big as you have RAM, so if you have 8GB of RAM you would want to test with a 16GB file.
extended open flags are accessible via FWIW lots of bennchmarks exist showing synchronous IO calls improve when matching IO operations to the underlying block size (e.g. 4MB for modern SSD's) across multiple OS's, so its very likely that there are improvements to gain here. However, 4MB is stack-breaking size by default, and heap allocations will destroy potential improvements (and further, if we're optimising, we probably want overlapped IO and a ring of buffers rather than just one buffer which will always leave one device idle).... so its complex;)
heap allocations will destroy potential improvements How come? would allocate once at the beginning. Heap memory isn't inherently slower and one memory allocation is basically free compared to the heavy IO that's happening afterwards.
But we also don't want to be allocating 4MB every time we do as someone might be doing a bunch of small copies.
Code change:
#![feature(unsafe_cell_raw_get)] #![feature(unwind_attributes)] #![feature(vec_into_raw_parts)] <ins> #![feature(vec_spare_capacity)] </ins> #![feature(wake_trait)] // NB: the above list is sorted to minimize merge conflicts. #![default_lib_allocator]

Related discussion:
text: possible approaches: thread local buffers or a buffer pool stat the input before deciding on the buffer size (this is what gnu cp does I think) heap-allocate a bigger buffer after a few iterations of the copy loop with an on-stack buffer specialize copy operations from file to memory to directly into the already allocated target, thus skipping the need for a buffer
specialize copy operations from file to memory to directly into the already allocated target, thus skipping the need for a buffer I think some kind of specialization for sources would be ideal. Because in this case, should not buffer at all and simply use the reader's buffer. Effectively this also gives you a customizable buffer size by wrapping your in a with whatever size you want. As of today, if your source is then is always the wrong answer. I got a substantial performance improvement using a simple loop instead:
a single heap allocation is entirely reasonable for a single invocation of copy of a lot of data. For many copies of a small amount of data, one ends up with a lot of allocations again (consider for instance copying data out of a tar, where we have views on the data and there's no actual synchronous IO etc happening. I like some of the suggested possibilities with specialisation.
It's a nice general optimization, but not necesarily ideal if you use a as . Adding a on the reader size does let you control the buffer size but it still introduces an unnecessary memcopy compared to . Perhaps a writer-side optimization would be more flexible. Then you can specialize either for or
That sounds rather uncommon to me though. After all, is provided by the trait so it's always available. The only case I can see where you would invoke with a as destination is some generic interfaces that takes any . But IMO such interfaces are ill-designed, as they could simply implement instead and offer a much more flexible API.
yes, generic code taking a write was the concern. E.g. tar-rs has a tar archive which takes a and is fed with items. I'm not saying that creating an in-memory archive would be a good idea, just pointing out that these patterns do exist.
I had same kind of issue but with different library (). The copy rate was around 70 MB/s (inside single NVMe SSD). The problem was actually with running the debug version of my program.
Code change:
#![feature(unsafe_cell_raw_get)] #![feature(unwind_attributes)] #![feature(vec_into_raw_parts)] <ins> #![feature(vec_spare_capacity)] </ins> #![feature(wake_trait)] // NB: the above list is sorted to minimize merge conflicts. #![default_lib_allocator]

Related discussion:
text: When I built and ran the release version, the speed jumped to over 1 GB/s.
With merged you can now (on next nightly) use to control the chunk size of . Since this relies on specialization it only works if you pass a directly, any other wrapper around it would disable this optimization.
Code change:
cx.span_lint(UNUSED_COMPARISONS, e.span, "comparison is useless due to type limits"); } <ins> if is_shift_binop(binop) { let opt_ty_bits = match ty::get(ty::expr_ty(cx.tcx, &**l)).sty { ty::ty_int(t) => Some(int_ty_bits(t)), ty::ty_uint(t) => Some(uint_ty_bits(t)), _ => None }; if let Some(bits) = opt_ty_bits { let exceeding = if let ast::ExprLit(ref lit) = r.node { if let ast::LitInt(shift, _) = lit.node { shift > bits } else { false } } else { match eval_const_expr_partial(cx.tcx, &**r) { Ok(const_int(shift)) => { shift as u64 > bits }, Ok(const_uint(shift)) => { shift > bits }, _ => { false } } }; if exceeding { cx.span_lint(EXCEEDING_BITSHIFTS, e.span, "bitshift exceeds the type's number of bits"); } }; } </ins> }, ast::ExprLit(ref lit) => { match ty::get(ty::expr_ty(cx.tcx, e)).sty {

Related discussion:
text: Currently overlong bit shifts are undefined behavior because that's how llvm rolls--. But even when they aren't, we should warn on something like where happens to be , since it's a useless operation that couldn't possibly return anything interesting. There's a and I imagine it'd be a common gotcha since other languages tend to read unadorned integral literals as 32bit s and then promote the in before doing the actual shift.
I think this can be closed, resolved in
Hooray!
Code change:
ty = it.type_(), path = relpath ); if parentlen == 0 { <del> // There is no sidebar-items.js beyond the crate root path // FIXME maybe dynamic crate loading can be merged here </del> <ins> write!( buffer, "<script defer src="{}sidebar-items{}.js"></script>", relpath, cx.shared.resource_suffix ); </ins> } else { <del> write!(buffer, "<script defer src="{path}sidebar-items.js"></script>", path = relpath); </del> <ins> write!(buffer, "<script defer src="{}sidebar-items.js"></script>", relpath); </ins> } // Closes sidebar-elems div. buffer.write_str("</div>"); }

Related discussion:
text: <!-- Thank you for filing a regression report! A regression is something that changed between versions of Rust but was not supposed to. Please provide a short summary of the regression, along with any information you feel is relevant to replicate it. --Since i don't know which version, when running on a project, the left sidebar of the docs no longer has a nav for all crates in the dependency graph. However, the documentation for all dependencies is still compiled. a Rust project with a few dependencies. : Left sidebar has a list of all dependencies, linking to each crate's local docs. : No such thing. <!-- Provide the most recent version this worked on, for example: It most recently worked on: Rust 1.47 --It most recently worked on: I honestly don't remember. I update my nightly before every dev session. Allegedly that bug is 27 days old. <!-- Provide the version you are using that has the regression. --: <!-- If you know when this regression occurred, please add a line like below, replacing with one of stable, beta, or nightly. modify labels: +regression-from-stable-to-{channel} -regression-untriaged --
Appears to be a regression from , the call to was moved into , but this function is not called on the crate-root doc page. I tested locally that it fails on and works on .
Checking looks like this has made it onto the 1.53 beta.
cc
I think the ideal fix is around We should emit a sidebar- for the top page that just calls . And remove that comment, since we now have dynamic crate loading. A quicker fix is to just move the call to addSidebarCrates so it gets called independently from initSidebarItems. I like that less since the crates really are a sidebar item and should be treated as such.
I can do that if no one else wants to do it!
Go for it! I'm happy to review.
Assigning .
It's actually something I fixed in (still waiting for approval/review).
I propose we plan on landing and backporting to fix the regression: it's much smaller and limited to the specific fix. Then we can work on landing in nightly.
No, the approach taken in is problematic. The information is already there and we don't need this new file.
Code change:
ty = it.type_(), path = relpath ); if parentlen == 0 { <del> // There is no sidebar-items.js beyond the crate root path // FIXME maybe dynamic crate loading can be merged here </del> <ins> write!( buffer, "<script defer src="{}sidebar-items{}.js"></script>", relpath, cx.shared.resource_suffix ); </ins> } else { <del> write!(buffer, "<script defer src="{path}sidebar-items.js"></script>", path = relpath); </del> <ins> write!(buffer, "<script defer src="{}sidebar-items.js"></script>", relpath); </ins> } // Closes sidebar-elems div. buffer.write_str("</div>"); }

Related discussion:
text: EDIT: also, is ready, there is a debate for the other part of the PR (the change on module's sidebar) which I can move to another PR.
See - after some discussion, and concluded to merge as our quick fix. To explain why I proposed to add a tiny sidebar- We have one that's loaded by a variety of different pages. It also potentially has many different entry points. For instance, the primary entry point is that when itself is loaded, it runs various functions, adding event handlers and so on. Another entry point is that when sidebar- loads, it calls . That's the root cause of this bug - code in expected would get called, and for a subset of pages it wasn't. It's a bit inelegant to have a small file whose purpose is only to call a known function. But I think that inelegance is balanced against the improved maintainability of having most doc pages follow closer to the same code path through The more code paths, the more chances for bugs and the more complexity in testing. Another way to address this longer-term would be to eliminate the entrypoint entirely. Instead, when runs, it would trigger the load of sidebar-items and run any callbacks based on that load. Yet another way to address this would be to separate out from , so that is called unconditionally for all pages.
Code change:
Ok(Some((d, path))) => (path.trim(), Some(d)), Ok(None) => (link.trim(), None), Err((err_msg, relative_range)) => { <del> let disambiguator_range = (no_backticks_range.start + relative_range.start) ..(no_backticks_range.start + relative_range.end); disambiguator_error(self.cx, &item, dox, disambiguator_range, &err_msg); </del> <ins> if !should_ignore_link_with_disambiguators(link) { // Only report error if we would not have ignored this link. // See issue #83859. let disambiguator_range = (no_backticks_range.start + relative_range.start) ..(no_backticks_range.start + relative_range.end); disambiguator_error(self.cx, &item, dox, disambiguator_range, &err_msg); } </ins> return None; } }; <del> if path_str.contains(|ch: char| !(ch.is_alphanumeric() || ":_<>, !*&;".contains(ch))) { </del> <ins> if should_ignore_link(path_str) { </ins> return None; }

Related discussion:
text: <!-- Thank you for filing a bug report! Please provide a short summary of the bug, along with any information you feel relevant to replicating the bug. --I tried this code: I expected to see this happen: rustdoc parses the link as an email address and renders it as a mailto: link. Instead, this happened: <!-- If you're using the stable version of the compiler, you should also check if the bug also exists in the beta or nightly versions. --(from a GitHub Actions run, the latest nightly won't install on my Arch box due to broken components)
This turned out to be a bug in my docs; I didn't intend for this to be rendered as a link, but I was hoping for a better error message.
I think this PR introduced this new behavior:
This is weird - rustdoc should be ignoring any link that has in it:
Ah I see the issue - should only be reported if the link isn't ignored. do you have time to fix it? It should be as simple as adding the same check here:
Assigning and removing as in the prioritization working group. label: +P-medium -I-prioritize
can part of the fix here be to improve the error message? "Unknown disambiguator" doesn't tell me much about what is wrong or how to fix it.
I'm confused. After there won't be an error message, because the link shouldn't have been treated as an intra-doc link in the first place. Are you asking for a lint that warns about docs that will be rendered as mailto: links? I'm not sure how useful that is, mailto: links are normal and reasonable to write.
based on the UI tests that are checked in, if I actually intend to use an intradoc link with a disambiguator and fat finger as , the error message is not super friendly.
Hmm, what would you like to see instead? There's quite a few valid disambiguators, I'm not sure it makes sense to list them all. Maybe we could add a header to the disambiguator section of and link there?
Ideally I'd like to see a suggestion to use a disambiguator that corresponds to an item that is currently in scope or otherwise resolvable via the path after the invalid disambiguator.
Code change:
Ok(Some((d, path))) => (path.trim(), Some(d)), Ok(None) => (link.trim(), None), Err((err_msg, relative_range)) => { <del> let disambiguator_range = (no_backticks_range.start + relative_range.start) ..(no_backticks_range.start + relative_range.end); disambiguator_error(self.cx, &item, dox, disambiguator_range, &err_msg); </del> <ins> if !should_ignore_link_with_disambiguators(link) { // Only report error if we would not have ignored this link. // See issue #83859. let disambiguator_range = (no_backticks_range.start + relative_range.start) ..(no_backticks_range.start + relative_range.end); disambiguator_error(self.cx, &item, dox, disambiguator_range, &err_msg); } </ins> return None; } }; <del> if path_str.contains(|ch: char| !(ch.is_alphanumeric() || ":_<>, !*&;".contains(ch))) { </del> <ins> if should_ignore_link(path_str) { </ins> return None; }

Related discussion:
text: Hmm, that seems useful, although I think it should be a separate issue. Maybe we could try resolving without a disambiguator and see if there are any results? That would need a pretty significant overhaul of how it works currently, right now all the errors are emitted eagerly which would end up with a lot more verbose output that might not be relevant. What would you expect to see if the link isn't valid even without the disambiguator?
I think an extra to more docs on intradoc link disambiguators would be helpful per your previous comment. The idea to attempt to resolve items for all invalid disambiguators feels similar to the Ruby CLI and REPL's "did you mean" feature, which I find quite useful.
The PR improving the diagnostics is being reverted in , reopening the issue.
I don't think this is still a regression; the diagnostics improvement was not part of the fix for the regression. (The regression was fixed in .)
Yeah, never had a tracking issue.
was relanded in
Code change:
if self.filename { match self.path.filename() { None => ~"", <del> Some(v) => from_utf8_with_replacement(v) </del> <ins> Some(v) => str::from_utf8_lossy(v) </ins> } } else { <del> from_utf8_with_replacement(self.path.as_vec()) </del> <ins> str::from_utf8_lossy(self.path.as_vec()) </ins> } } }

Related discussion:
text: We need some way to interpret a as UTF-8, using the replacement character for invalid sequences instead of conditions. This would ideally be provided in the form of an .
Could we remove the current condition API in favour of this? It doesn't really seem to offer anything but failing, and we already have an API returning an .
I would be quite pleased if we could do that. Using the condition is rather awkward, and I would rather just have replacement chars myself.
This is a duplicate of , no?
And also the e-mail thread here I think is relevant:
This is an implementation of a mostly complete → UTF-8 decoder
It's only a duplicate if the intended resolution of is to use the replacement char instead of conditions, but it's unclear from the discussion if that's the plan.
I guess my thinking was that a maximally expressive condition-based solution would let one express a replacement char approach using conditions. But I don't actually favor doing that at this point. Or at least, I don't favor making that the only way to accomplish this, since I suspect a more specialized approach will be much nicer to use (both in terms of programmer convenience and in terms of efficiency). So okay, this is not a duplicate of .
Rust needs to support both fatal and replacement error handling modes, it should follow the WHATWG spec simonsapin has said (edit by is WHATWG spec.)
the fatal mode is already handled by the API
well, the current functions in str don't handle decoding buffers in chunks like the proposed encodings API. But yes, it does handle one-off decoding with 'fatal' error handling.
I think we need to improve/extend our API along the lines suggested here for 1.0. Nominating for P-backcompat-lang. (Arguably we could avoid the backwards compatibility hazard by offering a fail-only method for 1.0 and then adding an alternative entry point that provides the more flexible API supporting replacement characters in post 1.0. But I think this case is important enough that we should try to get the primary API method to provide both choices up front. That, or put dynamic fluid support in for representing the state of that choice.)
I submitted a PR for this last night, although I forgot there was an open issue too so I didn't link them. The PR is
Code change:
/// and surrogates as `u` followed by four hexadecimal digits. /// Example: `"au{D800}"` for a slice with code points [U+0061, U+D800] impl fmt::Debug for Wtf8 { <del> fn fmt(&self, formatter: &mut fmt::Formatter) -> Result<(), fmt::Error> { </del> <ins> fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result { fn write_str_escaped(f: &mut fmt::Formatter, s: &str) -> fmt::Result { use fmt::Write; for c in s.chars().flat_map(|c| c.escape_default()) { try!(f.write_char(c)) } Ok(()) } </ins> try!(formatter.write_str(""")); let mut pos = 0; loop { match self.next_surrogate(pos) { None => break, Some((surrogate_pos, surrogate)) => { <del> try!(formatter.write_str(unsafe { // the data in this slice is valid UTF-8, transmute to &str mem::transmute(&self.bytes[pos .. surrogate_pos]) })); </del> <ins> try!(write_str_escaped( formatter, unsafe { str::from_utf8_unchecked( &self.bytes[pos .. surrogate_pos] )}, )); </ins> try!(write!(formatter, "u{{{:X}}}", surrogate)); pos = surrogate_pos + 3; } } } <del> try!(formatter.write_str(unsafe { // the data in this slice is valid UTF-8, transmute to &str mem::transmute(&self.bytes[pos..]) })); </del> <ins> try!(write_str_escaped( formatter, unsafe { str::from_utf8_unchecked(&self.bytes[pos..]) }, )); </ins> formatter.write_str(""") } }

Related discussion:
text: Formatting a string containing special characters with usually prints them escaped i.e.. This snippet runs as expected in the . However on Windows, using both and , the is printed unescaped : From what I can gather,  only concerns itself with faulty surrogates and completely ignores special characters.