Code change:
} } } <ins> #[inline] fn len_utf8(code: u32) -> usize { if code < MAX_ONE_B { 1 } else if code < MAX_TWO_B { 2 } else if code < MAX_THREE_B { 3 } else { 4 } } /// Encodes a raw u32 value as UTF-8 into the provided byte buffer, /// and then returns the subslice of the buffer that contains the encoded character. /// /// Unlike `char::encode_utf8`, this method also handles codepoints in the surrogate range. /// (Creating a `char` in the surrogate range is UB.) /// The result is valid [generalized UTF-8] but not valid UTF-8. /// /// [generalized UTF-8]: https://simonsapin.github.io/wtf-8/#generalized-utf8 /// /// # Panics /// /// Panics if the buffer is not large enough. /// A buffer of length four is large enough to encode any `char`. #[unstable(feature = "char_internals", reason = "exposed only for libstd", issue = "none")] #[doc(hidden)] #[inline] pub fn encode_utf8_raw(code: u32, dst: &mut [u8]) -> &mut [u8] { let len = len_utf8(code); match (len, &mut dst[..]) { (1, [a, ..]) => { *a = code as u8; } (2, [a, b, ..]) => { *a = (code >> 6 & 0x1F) as u8 | TAG_TWO_B; *b = (code & 0x3F) as u8 | TAG_CONT; } (3, [a, b, c, ..]) => { *a = (code >> 12 & 0x0F) as u8 | TAG_THREE_B; *b = (code >> 6 & 0x3F) as u8 | TAG_CONT; *c = (code & 0x3F) as u8 | TAG_CONT; } (4, [a, b, c, d, ..]) => { *a = (code >> 18 & 0x07) as u8 | TAG_FOUR_B; *b = (code >> 12 & 0x3F) as u8 | TAG_CONT; *c = (code >> 6 & 0x3F) as u8 | TAG_CONT; *d = (code & 0x3F) as u8 | TAG_CONT; } _ => panic!( "encode_utf8: need {} bytes to encode U+{:X}, but the buffer has {}", len, code, dst.len(), ), }; &mut dst[..len] } /// Encodes a raw u32 value as UTF-16 into the provided `u16` buffer, /// and then returns the subslice of the buffer that contains the encoded character. /// /// Unlike `char::encode_utf16`, this method also handles codepoints in the surrogate range. /// (Creating a `char` in the surrogate range is UB.) /// /// # Panics /// /// Panics if the buffer is not large enough. /// A buffer of length 2 is large enough to encode any `char`. #[unstable(feature = "char_internals", reason = "exposed only for libstd", issue = "none")] #[doc(hidden)] #[inline] pub fn encode_utf16_raw(mut code: u32, dst: &mut [u16]) -> &mut [u16] { // SAFETY: each arm checks whether there are enough bits to write into unsafe { if (code & 0xFFFF) == code && !dst.is_empty() { // The BMP falls through *dst.get_unchecked_mut(0) = code as u16; slice::from_raw_parts_mut(dst.as_mut_ptr(), 1) } else if dst.len() >= 2 { // Supplementary planes break into surrogates. code -= 0x1_0000; *dst.get_unchecked_mut(0) = 0xD800 | ((code >> 10) as u16); *dst.get_unchecked_mut(1) = 0xDC00 | ((code as u16) & 0x3FF); slice::from_raw_parts_mut(dst.as_mut_ptr(), 2) } else { panic!( "encode_utf16: need {} units to encode U+{:X}, but the buffer has {}", from_u32_unchecked(code).len_utf16(), code, dst.len(), ) } } } </ins>

Related discussion:
text: The following program causes UB: Miri says: The problem is this code: This calls unless the new code point is in , but what about surrogates in ? This code is unchanged since its introduction in I am not sure what the intended safety contract of is. That method is not marked but clearly should be -- it calls . So my guess is the safety precondition is that must not be part of a surrogate pair, but the thing is, calls it without actually ensuring that condition. The condition it does ensure is that the codepoint is not in , but that does not help.
is excluded from Unicode scalar values a.k.a. , but it’s fine as a Unicode code point. I think this bug is not in the contract of (which is private anyway) but in its implementation, namely going through . The fix is to either duplicate the logic of into , or (to avoid duplication) move that logic to a new function in libcore that takes a parameter and that is called by both and . This function would need to be public because is in a different crate, but it should be prema-unstable and .
I would classify the priority of this bug as low. Although this call to is UB, I expect Miri checking for it explicitly is the only case where that has any consequence in today’s implementation: behaves as expected (for the purpose of WTF-8) in the surrogate range and does not exploit this UB The "layout" of in rustc excludes values beyond but not surrogates:
By the way, the docs at feel insufficient: Which values are invalid? Similarly, and say that represents Unicode Scalar values, but I couldn’t find it documented anywhere that constructing an out of range is Undefined Behavior rather than "merely" a logic bug. Is this specified elsewhere? What is the closest we have to a written down normative resource that specifies what is or isn’t UB in the the Rust language? (For APIs I assume this is the responsibility of their respective doc-comments.)
I agree the impact is low, but it's not just Miri -- this blocks , which is how I discovered the problem. That would be
But then what is "unchecked" about it? Elsewhere in that file the code is careful not to construct a from a without checking for surrogates: That made me assume the author was aware that surrogates in are UB.