Change:
The Unicode Windows API internally works with (potentially ill-formed) UTF-16 paths that must be converted to from Rust's UTF-8 strings. The conversion happens here: does the translation from WTF-8 to UTF-16, leaving a with . The following calling to will then allocate to reserve additional memory for the null-terminating character. This could be resolved in two ways: 1) Call beforehand and then call on EncodeWide. 2) Make a wrapper around that increases by one. Incorporating the final null byte into would add a performance penalty, therefore it would probably be desirable to just adjust the sizehint by one and adding the null-terminator manually. <!-- TRIAGEBOTSTART --<!-- TRIAGEBOTASSIGNSTART --<!-- TRIAGEBOTASSIGNDATASTARTTRIAGEBOTASSIGNDATAEND --<!-- TRIAGEBOTASSIGNEND --<!-- TRIAGEBOT_END --
In the interest of enumerating possibilities, there's also using (hopefully this will optimize): Or we could test the for nulls before converting.
That's a good idea! Chain actually gives an example for windows paths: Although I'm not sure what's more efficient here, std::iter::once or [0]? Allocations are quite expensive so I believe the additional check (if it's not optimized away) would be worth it. However I also discovered a deeper issue. The sizehint that originates from the WTF-8 iterator is in UTF-32 code points. The lower bound will be (assuming that all code points are U+ or higher), the upper bound is (everything is "ASCII"). When collecting into a vector, the lower bound is chosen as an initial capacity. Since you almost never encounter paths that consists of U+ characters only, the estimated lower bound will always be too low and 4 allocations are required if the path is ASCII only, 2 allocations are required if the path is UTF-16 only, and 1 allocation is required if the path is UTF-32 only. It might be worth a consideration whether to call , as that would be a sane default (on the assumption that the majority of paths are UTF-8). However, this might waste some memory, do we have any idea on whether paths created from this function are primarily used ephemerally?
I believe all uses of are currently ephemeral. We tend to use it only for as long as necessary to call a Windows API. There are a few places we keep wide strings around longer but those don't use as far as I can tell.
In light of that, I believe that the first solution would be the best. Solution number two would make naming very awkward (EncodeWideWithNul?). And while your solution is very elegant and correct, it would still mostly under-allocate given the interaction between size_hint's lower bound and collect. However, I find both solution 1 and solution 3 acceptable. It would be good to know how much reallocation actually costs and if it even matters at all in the context of interacting with expensive IO.
I do agree implementing solution 1 would be a good idea. Having some good performance numbers would be great but I do suspect reducing allocations would be a win here. I would also like us to use stack buffers more, similar to . and only use new allocations when necessary. But that's perhaps another issue.
Reducing via stack buffers is a good idea, but I think it's out of scope for this issue. For now let's just reduce regular allocations, that should already give us some improvement! I'll try to see if I can implement this one. claim
Fixed by

Discussion:
pub fn to_u16s<S: AsRef<OsStr>>(s: S) -> crate::io::Result<Vec<u16>> { fn inner(s: &OsStr) -> crate::io::Result<Vec<u16>> { <del> let mut maybe_result: Vec<u16> = s.encode_wide().collect(); </del> <ins> // Most paths are ASCII, so reserve capacity for as much as there are bytes // in the OsStr plus one for the null-terminating character. We are not // wasting bytes here as paths created by this function are primarily used // in an ephemeral fashion. let mut maybe_result = Vec::with_capacity(s.len() + 1); maybe_result.extend(s.encode_wide()); </ins> if unrolled_find_u16s(0, &maybe_result).is_some() { return Err(crate::io::const_io_error!( ErrorKind::InvalidInput,