Code change:
(and similarly, should call ) Windows provides these nice functions which performantly implement utf-8 <-utf-16 conversions. We should probably call them in the standard library when possible.
Do we get measurable performance wins that make calling more unsafe C APIs worth it? Especially with the inlining barrier across languages. And I would assume console printing is bottlenecked on the console rendering part anyway, not on string conversions.
Do we have any good benchmarks for console I/O perf?
A simple main writing 100MB worth of lines to stdout and measuring the time it takes should do?
For sure. I was just wondering if we had something already. I mean, we'd probably want to be testing a few different languages as well to make sure we're not just optimizing for ASCII only.
I don't think we do because most benchmarks suppress console output.
I did do a benchmark, and interestingly, ASCII text was about twice as fast with as with rust-native code; however, Chinese text was slightly slower. Given that most of the actual cost of printing to the screen is in the actual printing, I no longer think it's a good idea to do this. See benchmarks at [strega-nil/rust-bench-cvt-utf8-utf16][]. [strega-nil/rust-bench-cvt-utf8-utf16]: Run on M1 chip: someone may want to check on an x64 chip, since that might be faster.
Just to check my understanding: The utf16 conversion only happens when one writes to an actually rendered console. There isn't some stuff like virtual consoles that act more like NUL where speeding up the conversion could help because the data will be discarded by the OS. And output to files is binary anyway.
There are now (since Windows 10, 1809) which are consoles without the rendering part. Though the intent would be for the rendering to be implemented by another program, it would also be possible to skip or delay (e.g. by writing a log) rendering.
In that case a speedup may be useful, but is the optimization potential of the built-in conversion already squeezed dry?
yeah, I think it's unlikely that the complexity is worth it given what the benchmarks show; I would really appreciate if someone ran the benchmark on x64, since it may be that Windows implements faster algorithms on that platform.
I don't think anyone has worked on optimizing this so I'm sure there's plenty of optimization potential. The results from I got from an AMD Ryzen 5 3600 machine are:
Mmh, thanks I'll also push some benchmarks for testing other cases later, since there are plenty of other cases than just "bulk-converting a bunch of text". Thanks! Nicole

Related discussion:
text: pub type LPBOOL = *mut BOOL; pub type LPBYTE = *mut BYTE; <ins> pub type LPCCH = *const CHAR; </ins> pub type LPCSTR = *const CHAR; <ins> pub type LPCWCH = *const WCHAR; </ins> pub type LPCWSTR = *const WCHAR; <ins> pub type LPCVOID = *const c_void; </ins> pub type LPDWORD = *mut DWORD; pub type LPHANDLE = *mut HANDLE; pub type LPOVERLAPPED = *mut OVERLAPPED; pub type LPPROCESS_INFORMATION = *mut PROCESS_INFORMATION; pub type LPSECURITY_ATTRIBUTES = *mut SECURITY_ATTRIBUTES; pub type LPSTARTUPINFO = *mut STARTUPINFO; <ins> pub type LPSTR = *mut CHAR; </ins> pub type LPVOID = *mut c_void; <del> pub type LPCVOID = *const c_void; </del> pub type LPWCH = *mut WCHAR; pub type LPWIN32_FIND_DATAW = *mut WIN32_FIND_DATAW; pub type LPWSADATA = *mut WSADATA;