Change:
The following program causes UB: Miri says: The problem is this code: This calls unless the new code point is in , but what about surrogates in ? This code is unchanged since its introduction in I am not sure what the intended safety contract of is. That method is not marked but clearly should be -- it calls . So my guess is the safety precondition is that must not be part of a surrogate pair, but the thing is, calls it without actually ensuring that condition. The condition it does ensure is that the codepoint is not in , but that does not help.
is excluded from Unicode scalar values a.k.a. , but it’s fine as a Unicode code point. I think this bug is not in the contract of (which is private anyway) but in its implementation, namely going through . The fix is to either duplicate the logic of into , or (to avoid duplication) move that logic to a new function in libcore that takes a parameter and that is called by both and . This function would need to be public because is in a different crate, but it should be prema-unstable and .
I would classify the priority of this bug as low. Although this call to is UB, I expect Miri checking for it explicitly is the only case where that has any consequence in today’s implementation: behaves as expected (for the purpose of WTF-8) in the surrogate range and does not exploit this UB The "layout" of in rustc excludes values beyond but not surrogates:
By the way, the docs at feel insufficient: Which values are invalid? Similarly, and say that represents Unicode Scalar values, but I couldn’t find it documented anywhere that constructing an out of range is Undefined Behavior rather than "merely" a logic bug. Is this specified elsewhere? What is the closest we have to a written down normative resource that specifies what is or isn’t UB in the the Rust language? (For APIs I assume this is the responsibility of their respective doc-comments.)
I agree the impact is low, but it's not just Miri -- this blocks , which is how I discovered the problem. That would be
But then what is "unchecked" about it? Elsewhere in that file the code is careful not to construct a from a without checking for surrogates: That made me assume the author was aware that surrogates in are UB.
The type contains a (generalized-UTF-8-encoded) sequence of code point (including potentially surrogates) that doesn’t form a surrogate pair. doesn’t check for surrogate pairs. is a public method that is documented as returning a scalar value, therefore excluding all surrogates. Ok I looked into it, there are three different people involved. The original code in 2014 was YOLO Later in 2014, moved to duplicating logic instead. In 2015, PR which first imported WTF-8 support in libstd deduplicated that logic by adding a function that takes . (I knew that approach sounded familiar…) In 2016, PR changed the signature of and in passing removed . It made use + instead, presumably in the middle of updating many callers other.
Okay, so the fix would be to revert the part of that removed ?
That or copy into . I find them both not great, so meh.
Even with that done, there's still UB in another testcase: This is converting something to a before calling -- so probably the same issue as the above, just for a different encoding.

Discussion:
#[inline] #[stable(feature = "char_from_unchecked", since = "1.5.0")] pub unsafe fn from_u32_unchecked(i: u32) -> char { <del> transmute(i) </del> <ins> if cfg!(debug_assertions) { char::from_u32(i).unwrap() } else { transmute(i) } </ins> } #[stable(feature = "char_convert", since = "1.13.0")]

Err(CharTryFromError(())) } else { // SAFETY: checked that it's a legal unicode value <del> Ok(unsafe { from_u32_unchecked(i) }) </del> <ins> Ok(unsafe { transmute(i) }) </ins> } } }

#[stable(feature = "rust1", since = "1.0.0")] #[inline] pub fn len_utf8(self) -> usize { <del> let code = self as u32; if code < MAX_ONE_B { 1 } else if code < MAX_TWO_B { 2 } else if code < MAX_THREE_B { 3 } else { 4 } </del> <ins> len_utf8(self as u32) </ins> } /// Returns the number of 16-bit code units this `char` would need if

#[stable(feature = "unicode_encode_char", since = "1.15.0")] #[inline] pub fn encode_utf8(self, dst: &mut [u8]) -> &mut str { <del> let code = self as u32; let len = self.len_utf8(); match (len, &mut dst[..]) { (1, [a, ..]) => { *a = code as u8; } (2, [a, b, ..]) => { *a = (code >> 6 & 0x1F) as u8 | TAG_TWO_B; *b = (code & 0x3F) as u8 | TAG_CONT; } (3, [a, b, c, ..]) => { *a = (code >> 12 & 0x0F) as u8 | TAG_THREE_B; *b = (code >> 6 & 0x3F) as u8 | TAG_CONT; *c = (code & 0x3F) as u8 | TAG_CONT; } (4, [a, b, c, d, ..]) => { *a = (code >> 18 & 0x07) as u8 | TAG_FOUR_B; *b = (code >> 12 & 0x3F) as u8 | TAG_CONT; *c = (code >> 6 & 0x3F) as u8 | TAG_CONT; *d = (code & 0x3F) as u8 | TAG_CONT; } _ => panic!( "encode_utf8: need {} bytes to encode U+{:X}, but the buffer has {}", len, code, dst.len(), ), }; // SAFETY: We just wrote UTF-8 content in, so converting to str is fine. unsafe { from_utf8_unchecked_mut(&mut dst[..len]) } </del> <ins> // SAFETY: `char` is not a surrogate, so this is valid UTF-8. unsafe { from_utf8_unchecked_mut(encode_utf8_raw(self as u32, dst)) } </ins> } /// Encodes this character as UTF-16 into the provided `u16` buffer,

#[stable(feature = "unicode_encode_char", since = "1.15.0")] #[inline] pub fn encode_utf16(self, dst: &mut [u16]) -> &mut [u16] { <del> let mut code = self as u32; // SAFETY: each arm checks whether there are enough bits to write into unsafe { if (code & 0xFFFF) == code && !dst.is_empty() { // The BMP falls through (assuming non-surrogate, as it should) *dst.get_unchecked_mut(0) = code as u16; slice::from_raw_parts_mut(dst.as_mut_ptr(), 1) } else if dst.len() >= 2 { // Supplementary planes break into surrogates. code -= 0x1_0000; *dst.get_unchecked_mut(0) = 0xD800 | ((code >> 10) as u16); *dst.get_unchecked_mut(1) = 0xDC00 | ((code as u16) & 0x3FF); slice::from_raw_parts_mut(dst.as_mut_ptr(), 2) } else { panic!( "encode_utf16: need {} units to encode U+{:X}, but the buffer has {}", from_u32_unchecked(code).len_utf16(), code, dst.len(), ) } } </del> <ins> encode_utf16_raw(self as u32, dst) </ins> } /// Returns `true` if this `char` has the `Alphabetic` property.

} } } <ins> #[inline] fn len_utf8(code: u32) -> usize { if code < MAX_ONE_B { 1 } else if code < MAX_TWO_B { 2 } else if code < MAX_THREE_B { 3 } else { 4 } } /// Encodes a raw u32 value as UTF-8 into the provided byte buffer, /// and then returns the subslice of the buffer that contains the encoded character. /// /// Unlike `char::encode_utf8`, this method also handles codepoints in the surrogate range. /// (Creating a `char` in the surrogate range is UB.) /// The result is valid [generalized UTF-8] but not valid UTF-8. /// /// [generalized UTF-8]: https://simonsapin.github.io/wtf-8/#generalized-utf8 /// /// # Panics /// /// Panics if the buffer is not large enough. /// A buffer of length four is large enough to encode any `char`. #[unstable(feature = "char_internals", reason = "exposed only for libstd", issue = "none")] #[doc(hidden)] #[inline] pub fn encode_utf8_raw(code: u32, dst: &mut [u8]) -> &mut [u8] { let len = len_utf8(code); match (len, &mut dst[..]) { (1, [a, ..]) => { *a = code as u8; } (2, [a, b, ..]) => { *a = (code >> 6 & 0x1F) as u8 | TAG_TWO_B; *b = (code & 0x3F) as u8 | TAG_CONT; } (3, [a, b, c, ..]) => { *a = (code >> 12 & 0x0F) as u8 | TAG_THREE_B; *b = (code >> 6 & 0x3F) as u8 | TAG_CONT; *c = (code & 0x3F) as u8 | TAG_CONT; } (4, [a, b, c, d, ..]) => { *a = (code >> 18 & 0x07) as u8 | TAG_FOUR_B; *b = (code >> 12 & 0x3F) as u8 | TAG_CONT; *c = (code >> 6 & 0x3F) as u8 | TAG_CONT; *d = (code & 0x3F) as u8 | TAG_CONT; } _ => panic!( "encode_utf8: need {} bytes to encode U+{:X}, but the buffer has {}", len, code, dst.len(), ), }; &mut dst[..len] } /// Encodes a raw u32 value as UTF-16 into the provided `u16` buffer, /// and then returns the subslice of the buffer that contains the encoded character. /// /// Unlike `char::encode_utf16`, this method also handles codepoints in the surrogate range. /// (Creating a `char` in the surrogate range is UB.) /// /// # Panics /// /// Panics if the buffer is not large enough. /// A buffer of length 2 is large enough to encode any `char`. #[unstable(feature = "char_internals", reason = "exposed only for libstd", issue = "none")] #[doc(hidden)] #[inline] pub fn encode_utf16_raw(mut code: u32, dst: &mut [u16]) -> &mut [u16] { // SAFETY: each arm checks whether there are enough bits to write into unsafe { if (code & 0xFFFF) == code && !dst.is_empty() { // The BMP falls through *dst.get_unchecked_mut(0) = code as u16; slice::from_raw_parts_mut(dst.as_mut_ptr(), 1) } else if dst.len() >= 2 { // Supplementary planes break into surrogates. code -= 0x1_0000; *dst.get_unchecked_mut(0) = 0xD800 | ((code >> 10) as u16); *dst.get_unchecked_mut(1) = 0xDC00 | ((code as u16) & 0x3FF); slice::from_raw_parts_mut(dst.as_mut_ptr(), 2) } else { panic!( "encode_utf16: need {} units to encode U+{:X}, but the buffer has {}", from_u32_unchecked(code).len_utf16(), code, dst.len(), ) } } } </ins>

#[stable(feature = "unicode_version", since = "1.45.0")] pub use crate::unicode::UNICODE_VERSION; <ins> // perma-unstable re-exports #[unstable(feature = "char_internals", reason = "exposed only for libstd", issue = "none")] pub use self::methods::encode_utf16_raw; #[unstable(feature = "char_internals", reason = "exposed only for libstd", issue = "none")] pub use self::methods::encode_utf8_raw; </ins> use crate::fmt::{self, Write}; use crate::iter::FusedIterator;

#![feature(cfg_target_has_atomic)] #![feature(cfg_target_thread_local)] #![feature(char_error_internals)] <ins> #![feature(char_internals)] </ins> #![feature(clamp)] #![feature(concat_idents)] #![feature(const_cstr_unchecked)]

/// Copied from String::push /// This does **not** include the WTF-8 concatenation check. fn push_code_point_unchecked(&mut self, code_point: CodePoint) { <del> let c = unsafe { char::from_u32_unchecked(code_point.value) }; </del> let mut bytes = [0; 4]; <del> let bytes = c.encode_utf8(&mut bytes).as_bytes(); </del> <ins> let bytes = char::encode_utf8_raw(code_point.value, &mut bytes); </ins> self.bytes.extend_from_slice(bytes) }

let mut buf = [0; 2]; self.code_points.next().map(|code_point| { <del> let c = unsafe { char::from_u32_unchecked(code_point.value) }; let n = c.encode_utf16(&mut buf).len(); </del> <ins> let n = char::encode_utf16_raw(code_point.value, &mut buf).len(); </ins> if n == 2 { self.extra = buf[1]; }