Code change:
unsafe { Box::from_raw(Box::into_raw(v) as *mut str) } } <del> /// Converts the bytes while the bytes are still ascii. </del> <ins> /// Converts leading ascii bytes in `s` by calling the `convert` function. /// </ins> /// For better average performance, this happens in chunks of `2*size_of::<usize>()`. <del> /// Returns a vec with the converted bytes. </del> <ins> /// /// Returns a tuple of the converted prefix and the remainder starting from /// the first non-ascii character. /// /// This function is only public so that it can be verified in a codegen test, /// see `issue-123712-str-to-lower-autovectorization.rs`. #[unstable(feature = "str_internals", issue = "none")] #[doc(hidden)] </ins> #[inline] #[cfg(not(test))] #[cfg(not(no_global_oom_handling))] <del> fn convert_while_ascii(b: &[u8], convert: fn(&u8) -> u8) -> Vec<u8> { let mut out = Vec::with_capacity(b.len()); </del> <ins> pub fn convert_while_ascii(s: &str, convert: fn(&u8) -> u8) -> (String, &str) { // Process the input in chunks of 16 bytes to enable auto-vectorization. // Previously the chunk size depended on the size of `usize`, // but on 32-bit platforms with sse or neon is also the better choice. // The only downside on other platforms would be a bit more loop-unrolling. const N: usize = 16; let mut slice = s.as_bytes(); let mut out = Vec::with_capacity(slice.len()); let mut out_slice = out.spare_capacity_mut(); let mut ascii_prefix_len = 0_usize; let mut is_ascii = [false; N]; while slice.len() >= N { // SAFETY: checked in loop condition let chunk = unsafe { slice.get_unchecked(..N) }; // SAFETY: out_slice has at least same length as input slice and gets sliced with the same offsets let out_chunk = unsafe { out_slice.get_unchecked_mut(..N) }; for j in 0..N { is_ascii[j] = chunk[j] <= 127; } </ins> <del> const USIZE_SIZE: usize = mem::size_of::<usize>(); const MAGIC_UNROLL: usize = 2; const N: usize = USIZE_SIZE * MAGIC_UNROLL; const NONASCII_MASK: usize = usize::from_ne_bytes([0x80; USIZE_SIZE]); </del> <ins> // Auto-vectorization for this check is a bit fragile, sum and comparing against the chunk // size gives the best result, specifically a pmovmsk instruction on x86. // See https://github.com/llvm/llvm-project/issues/96395 for why llvm currently does not // currently recognize other similar idioms. if is_ascii.iter().map(|x| *x as u8).sum::<u8>() as usize != N { break; } </ins> <del> let mut i = 0; unsafe { while i + N <= b.len() { // Safety: we have checks the sizes `b` and `out` to know that our let in_chunk = b.get_unchecked(i..i + N); let out_chunk = out.spare_capacity_mut().get_unchecked_mut(i..i + N); let mut bits = 0; for j in 0..MAGIC_UNROLL { // read the bytes 1 usize at a time (unaligned since we haven't checked the alignment) // safety: in_chunk is valid bytes in the range bits |= in_chunk.as_ptr().cast::<usize>().add(j).read_unaligned(); } // if our chunks aren't ascii, then return only the prior bytes as init if bits & NONASCII_MASK != 0 { break; } </del> <ins> for j in 0..N { out_chunk[j] = MaybeUninit::new(convert(&chunk[j])); } </ins> <del> // perform the case conversions on N bytes (gets heavily autovec'd) for j in 0..N { // safety: in_chunk and out_chunk is valid bytes in the range let out = out_chunk.get_unchecked_mut(j); out.write(convert(in_chunk.get_unchecked(j))); } </del> <ins> ascii_prefix_len += N; slice = unsafe { slice.get_unchecked(N..) }; out_slice = unsafe { out_slice.get_unchecked_mut(N..) }; } </ins> <del> // mark these bytes as initialised i += N; </del> <ins> // handle the remainder as individual bytes while slice.len() > 0 { let byte = slice[0]; if byte > 127 { break; } // SAFETY: out_slice has at least same length as input slice unsafe { *out_slice.get_unchecked_mut(0) = MaybeUninit::new(convert(&byte)); </ins> } <del> out.set_len(i); </del> <ins> ascii_prefix_len += 1; slice = unsafe { slice.get_unchecked(1..) }; out_slice = unsafe { out_slice.get_unchecked_mut(1..) }; </ins> } <del> out </del> <ins> unsafe { // SAFETY: ascii_prefix_len bytes have been initialized above out.set_len(ascii_prefix_len); // SAFETY: We have written only valid ascii to the output vec let ascii_string = String::from_utf8_unchecked(out); // SAFETY: we know this is a valid char boundary // since we only skipped over leading ascii bytes let rest = core::str::from_utf8_unchecked(slice); (ascii_string, rest) } </ins> }

Related discussion:
text: I'm looking into the performance of / on mostly ascii strings, using a small microbenchmark to . Using linux perf tooling I see that the hot part of the code is the following large loop, which despite heavy use of sse2 instructions only seems to process 32 bytes per iteration. I don't see an easy way to improve the autovectorization of this code, but it should be relatively easy to explicitly vectorize it using , and I would like to prepare such a PR if there are no objections. As far as I know, is already in use inside , for example by .
The way the loop ors the bits for the is-ascii check into a single usize doesn't seem ideal for vectorization. keeping the lanes independent should yield better autovectorization results.
Thanks for the suggestion. I already started with a simd implementation, but now checked again if the autovectorization could be improved. With the following main loop The assembly and performance is indeed better, but there is still some weird shuffling and shifting going on: The explicit simd version looks a bit better, mostly because the ascii check directly translates to a : Do you have a preference here between autovectorization and explicit simd?
If the SIMD impl results in reasonable code across architectures, including some without SIMD then that should be fine. I think it would be the first time where we use portable simd unconditionally, so it could use some additional scrutiny. If it's only meant to target x86 with SSE2 then that'd mean having multiple paths and in that case we'd still have to tweak the generic path anyway. At that point we might as well just optimize the latter further. You can probably get rid of those too by keeping multiple bools in a small array. That way the optimizer will more easily see that it can shove them into independent simd lanes. Increasing the unroll count might help too to fit better into simd registers. We do get pretty decent autovectorization in other places in the standard library by massaging things into a state that basically looks like arrays-instead-of-SIMD-types. E.g.
It seems llvm is too "smart" for this trick, on its own that generates the exact same code. The trick does work, although it seems a bit fragile.