Code change:
(and similarly, should call ) Windows provides these nice functions which performantly implement utf-8 <-utf-16 conversions. We should probably call them in the standard library when possible.
Do we get measurable performance wins that make calling more unsafe C APIs worth it? Especially with the inlining barrier across languages. And I would assume console printing is bottlenecked on the console rendering part anyway, not on string conversions.
Do we have any good benchmarks for console I/O perf?
A simple main writing 100MB worth of lines to stdout and measuring the time it takes should do?
For sure. I was just wondering if we had something already. I mean, we'd probably want to be testing a few different languages as well to make sure we're not just optimizing for ASCII only.
I don't think we do because most benchmarks suppress console output.
I did do a benchmark, and interestingly, ASCII text was about twice as fast with as with rust-native code; however, Chinese text was slightly slower. Given that most of the actual cost of printing to the screen is in the actual printing, I no longer think it's a good idea to do this. See benchmarks at [strega-nil/rust-bench-cvt-utf8-utf16][]. [strega-nil/rust-bench-cvt-utf8-utf16]: Run on M1 chip: someone may want to check on an x64 chip, since that might be faster.
Just to check my understanding: The utf16 conversion only happens when one writes to an actually rendered console. There isn't some stuff like virtual consoles that act more like NUL where speeding up the conversion could help because the data will be discarded by the OS. And output to files is binary anyway.
There are now (since Windows 10, 1809) which are consoles without the rendering part. Though the intent would be for the rendering to be implemented by another program, it would also be possible to skip or delay (e.g. by writing a log) rendering.
In that case a speedup may be useful, but is the optimization potential of the built-in conversion already squeezed dry?
yeah, I think it's unlikely that the complexity is worth it given what the benchmarks show; I would really appreciate if someone ran the benchmark on x64, since it may be that Windows implements faster algorithms on that platform.
I don't think anyone has worked on optimizing this so I'm sure there's plenty of optimization potential. The results from I got from an AMD Ryzen 5 3600 machine are:
Mmh, thanks I'll also push some benchmarks for testing other cases later, since there are plenty of other cases than just "bulk-converting a bunch of text". Thanks! Nicole

Related discussion:
text: Ok(amount as usize) } <del> #[allow(unused)] </del> fn utf16_to_utf8(utf16: &[u16], utf8: &mut [u8]) -> io::Result<usize> { <del> let mut written = 0; for chr in char::decode_utf16(utf16.iter().cloned()) { match chr { Ok(chr) => { chr.encode_utf8(&mut utf8[written..]); written += chr.len_utf8(); } Err(_) => { // We can't really do any better than forget all data and return an error. return Err(io::const_io_error!( io::ErrorKind::InvalidData, "Windows stdin in console mode does not support non-UTF-16 input;  encountered unpaired surrogate", )); } } </del> <ins> debug_assert!(utf16.len() <= c::c_int::MAX as usize); debug_assert!(utf8.len() <= c::c_int::MAX as usize); let result = unsafe { c::WideCharToMultiByte( c::CP_UTF8,                    // CodePage c::WC_ERR_INVALID_CHARS,       // dwFlags utf16.as_ptr(),                // lpWideCharStr utf16.len() as c::c_int,       // cchWideChar utf8.as_mut_ptr() as c::LPSTR, // lpMultiByteStr utf8.len() as c::c_int,        // cbMultiByte ptr::null(),                   // lpDefaultChar ptr::null_mut(),               // lpUsedDefaultChar ) }; if result == 0 { // We can't really do any better than forget all data and return an error. Err(io::const_io_error!( io::ErrorKind::InvalidData, "Windows stdin in console mode does not support non-UTF-16 input;  encountered unpaired surrogate", )) } else { Ok(result as usize) </ins> } <del> Ok(written) </del> } impl IncompleteUtf8 {