Change:
I'm looking into the performance of / on mostly ascii strings, using a small microbenchmark to . Using linux perf tooling I see that the hot part of the code is the following large loop, which despite heavy use of sse2 instructions only seems to process 32 bytes per iteration. I don't see an easy way to improve the autovectorization of this code, but it should be relatively easy to explicitly vectorize it using , and I would like to prepare such a PR if there are no objections. As far as I know, is already in use inside , for example by .
The way the loop ors the bits for the is-ascii check into a single usize doesn't seem ideal for vectorization. keeping the lanes independent should yield better autovectorization results.
Thanks for the suggestion. I already started with a simd implementation, but now checked again if the autovectorization could be improved. With the following main loop The assembly and performance is indeed better, but there is still some weird shuffling and shifting going on: The explicit simd version looks a bit better, mostly because the ascii check directly translates to a : Do you have a preference here between autovectorization and explicit simd?
If the SIMD impl results in reasonable code across architectures, including some without SIMD then that should be fine. I think it would be the first time where we use portable simd unconditionally, so it could use some additional scrutiny. If it's only meant to target x86 with SSE2 then that'd mean having multiple paths and in that case we'd still have to tweak the generic path anyway. At that point we might as well just optimize the latter further. You can probably get rid of those too by keeping multiple bools in a small array. That way the optimizer will more easily see that it can shove them into independent simd lanes. Increasing the unroll count might help too to fit better into simd registers. We do get pretty decent autovectorization in other places in the standard library by massaging things into a state that basically looks like arrays-instead-of-SIMD-types. E.g.
It seems llvm is too "smart" for this trick, on its own that generates the exact same code. The trick does work, although it seems a bit fragile. The following creates a instruction just like the simd version, but a slightly different alternative with a array and comparing against 0 does not. It only needs to be better than the autovectorized version on those platforms ;) My initial idea was to gate the simd code on either SSE2 or Neon (possibly also Altivec and RiscV). I'd also add a scalar loop for the non-multiple-of-N remainder, so all-ascii strings are fully handled by specialized code. Currently this remainder goes through the generic and code path. On other platforms, only the scalar loop would be used, since I assume the autovectorization would also generate suboptimal code. But I agree that if similar code quality can be achieved with autovectorization, that would be preferable. I'll open a PR after a little bit more polishing the code.

Discussion:
make_test!(split_space_str, s, s.split(" ").count()); make_test!(split_ad_str, s, s.split("ad").count()); <ins> make_test!(to_lowercase, s, s.to_lowercase()); </ins>

use core::borrow::{Borrow, BorrowMut}; use core::iter::FusedIterator; <ins> use core::mem::MaybeUninit; </ins> #[stable(feature = "encode_utf16", since = "1.8.0")] pub use core::str::EncodeUtf16; #[stable(feature = "split_ascii_whitespace", since = "1.34.0")]

without modifying the original"] #[stable(feature = "unicode_case_mapping", since = "1.2.0")] pub fn to_lowercase(&self) -> String { <del> let out = convert_while_ascii(self.as_bytes(), u8::to_ascii_lowercase); </del> <ins> let (mut s, rest) = convert_while_ascii(self, u8::to_ascii_lowercase); </ins> <del> // Safety: we know this is a valid char boundary since // out.len() is only progressed if ascii bytes are found let rest = unsafe { self.get_unchecked(out.len()..) }; // Safety: We have written only valid ASCII to our vec let mut s = unsafe { String::from_utf8_unchecked(out) }; </del> <ins> let prefix_len = s.len(); </ins> for (i, c) in rest.char_indices() { if c == 'Σ' {

// in `SpecialCasing.txt`, // so hard-code it rather than have a generic "condition" mechanism. // See https://github.com/rust-lang/rust/issues/26035 <del> let out_len = self.len() - rest.len(); let sigma_lowercase = map_uppercase_sigma(&self, i + out_len); </del> <ins> let sigma_lowercase = map_uppercase_sigma(self, prefix_len + i); </ins> s.push(sigma_lowercase); } else { match conversions::to_lower(c) {

without modifying the original"] #[stable(feature = "unicode_case_mapping", since = "1.2.0")] pub fn to_uppercase(&self) -> String { <del> let out = convert_while_ascii(self.as_bytes(), u8::to_ascii_uppercase); // Safety: we know this is a valid char boundary since // out.len() is only progressed if ascii bytes are found let rest = unsafe { self.get_unchecked(out.len()..) }; // Safety: We have written only valid ASCII to our vec let mut s = unsafe { String::from_utf8_unchecked(out) }; </del> <ins> let (mut s, rest) = convert_while_ascii(self, u8::to_ascii_uppercase); </ins> for c in rest.chars() { match conversions::to_upper(c) {

unsafe { Box::from_raw(Box::into_raw(v) as *mut str) } } <del> /// Converts the bytes while the bytes are still ascii. </del> <ins> /// Converts leading ascii bytes in `s` by calling the `convert` function. /// </ins> /// For better average performance, this happens in chunks of `2*size_of::<usize>()`. <del> /// Returns a vec with the converted bytes. </del> <ins> /// /// Returns a tuple of the converted prefix and the remainder starting from /// the first non-ascii character. /// /// This function is only public so that it can be verified in a codegen test, /// see `issue-123712-str-to-lower-autovectorization.rs`. #[unstable(feature = "str_internals", issue = "none")] #[doc(hidden)] </ins> #[inline] #[cfg(not(test))] #[cfg(not(no_global_oom_handling))] <del> fn convert_while_ascii(b: &[u8], convert: fn(&u8) -> u8) -> Vec<u8> { let mut out = Vec::with_capacity(b.len()); </del> <ins> pub fn convert_while_ascii(s: &str, convert: fn(&u8) -> u8) -> (String, &str) { // Process the input in chunks of 16 bytes to enable auto-vectorization. // Previously the chunk size depended on the size of `usize`, // but on 32-bit platforms with sse or neon is also the better choice. // The only downside on other platforms would be a bit more loop-unrolling. const N: usize = 16; let mut slice = s.as_bytes(); let mut out = Vec::with_capacity(slice.len()); let mut out_slice = out.spare_capacity_mut(); let mut ascii_prefix_len = 0_usize; let mut is_ascii = [false; N]; while slice.len() >= N { // SAFETY: checked in loop condition let chunk = unsafe { slice.get_unchecked(..N) }; // SAFETY: out_slice has at least same length as input slice and gets sliced with the same offsets let out_chunk = unsafe { out_slice.get_unchecked_mut(..N) }; for j in 0..N { is_ascii[j] = chunk[j] <= 127; } </ins> <del> const USIZE_SIZE: usize = mem::size_of::<usize>(); const MAGIC_UNROLL: usize = 2; const N: usize = USIZE_SIZE * MAGIC_UNROLL; const NONASCII_MASK: usize = usize::from_ne_bytes([0x80; USIZE_SIZE]); </del> <ins> // Auto-vectorization for this check is a bit fragile, sum and comparing against the chunk // size gives the best result, specifically a pmovmsk instruction on x86. // See https://github.com/llvm/llvm-project/issues/96395 for why llvm currently does not // currently recognize other similar idioms. if is_ascii.iter().map(|x| *x as u8).sum::<u8>() as usize != N { break; } </ins> <del> let mut i = 0; unsafe { while i + N <= b.len() { // Safety: we have checks the sizes `b` and `out` to know that our let in_chunk = b.get_unchecked(i..i + N); let out_chunk = out.spare_capacity_mut().get_unchecked_mut(i..i + N); let mut bits = 0; for j in 0..MAGIC_UNROLL { // read the bytes 1 usize at a time (unaligned since we haven't checked the alignment) // safety: in_chunk is valid bytes in the range bits |= in_chunk.as_ptr().cast::<usize>().add(j).read_unaligned(); } // if our chunks aren't ascii, then return only the prior bytes as init if bits & NONASCII_MASK != 0 { break; } </del> <ins> for j in 0..N { out_chunk[j] = MaybeUninit::new(convert(&chunk[j])); } </ins> <del> // perform the case conversions on N bytes (gets heavily autovec'd) for j in 0..N { // safety: in_chunk and out_chunk is valid bytes in the range let out = out_chunk.get_unchecked_mut(j); out.write(convert(in_chunk.get_unchecked(j))); } </del> <ins> ascii_prefix_len += N; slice = unsafe { slice.get_unchecked(N..) }; out_slice = unsafe { out_slice.get_unchecked_mut(N..) }; } </ins> <del> // mark these bytes as initialised i += N; </del> <ins> // handle the remainder as individual bytes while slice.len() > 0 { let byte = slice[0]; if byte > 127 { break; } // SAFETY: out_slice has at least same length as input slice unsafe { *out_slice.get_unchecked_mut(0) = MaybeUninit::new(convert(&byte)); </ins> } <del> out.set_len(i); </del> <ins> ascii_prefix_len += 1; slice = unsafe { slice.get_unchecked(1..) }; out_slice = unsafe { out_slice.get_unchecked_mut(1..) }; </ins> } <del> out </del> <ins> unsafe { // SAFETY: ascii_prefix_len bytes have been initialized above out.set_len(ascii_prefix_len); // SAFETY: We have written only valid ascii to the output vec let ascii_string = String::from_utf8_unchecked(out); // SAFETY: we know this is a valid char boundary // since we only skipped over leading ascii bytes let rest = core::str::from_utf8_unchecked(slice); (ascii_string, rest) } </ins> }

assert_eq!("ΑΣ''Α".to_lowercase(), "ασ''α"); // https://github.com/rust-lang/rust/issues/124714 <ins> // input lengths around the boundary of the chunk size used by the ascii prefix optimization assert_eq!("abcdefghijklmnoΣ".to_lowercase(), "abcdefghijklmnoς"); </ins> assert_eq!("abcdefghijklmnopΣ".to_lowercase(), "abcdefghijklmnopς"); <ins> assert_eq!("abcdefghijklmnopqΣ".to_lowercase(), "abcdefghijklmnopqς"); </ins> // a really long string that has it's lowercase form // even longer. this tests that implementations don't assume

<ins> //@ only-x86_64 //@ compile-flags: -C opt-level=3 #![crate_type = "lib"] #![no_std] #![feature(str_internals)] extern crate alloc; /// Ensure that the ascii-prefix loop for `str::to_lowercase` and `str::to_uppercase` uses vector /// instructions. /// /// The llvm ir should be the same for all targets that support some form of simd. Only targets /// without any simd instructions would see scalarized ir. /// Unfortunately, there is no `only-simd` directive to only run this test on only such platforms, /// and using test revisions would still require the core libraries for all platforms. // CHECK-LABEL: @lower_while_ascii // CHECK: [[A:%[0-9]]] = load <16 x i8> // CHECK-NEXT: [[B:%[0-9]]] = icmp slt <16 x i8> [[A]], zeroinitializer // CHECK-NEXT: [[C:%[0-9]]] = bitcast <16 x i1> [[B]] to i16 #[no_mangle] pub fn lower_while_ascii(s: &str) -> (alloc::string::String, &str) { alloc::str::convert_while_ascii(s, u8::to_ascii_lowercase) } </ins>