Change:
We need some way to interpret a as UTF-8, using the replacement character for invalid sequences instead of conditions. This would ideally be provided in the form of an .
Could we remove the current condition API in favour of this? It doesn't really seem to offer anything but failing, and we already have an API returning an .
I would be quite pleased if we could do that. Using the condition is rather awkward, and I would rather just have replacement chars myself.
This is a duplicate of , no?
And also the e-mail thread here I think is relevant:
This is an implementation of a mostly complete → UTF-8 decoder
It's only a duplicate if the intended resolution of is to use the replacement char instead of conditions, but it's unclear from the discussion if that's the plan.
I guess my thinking was that a maximally expressive condition-based solution would let one express a replacement char approach using conditions. But I don't actually favor doing that at this point. Or at least, I don't favor making that the only way to accomplish this, since I suspect a more specialized approach will be much nicer to use (both in terms of programmer convenience and in terms of efficiency). So okay, this is not a duplicate of .
Rust needs to support both fatal and replacement error handling modes, it should follow the WHATWG spec simonsapin has said (edit by is WHATWG spec.)
the fatal mode is already handled by the API
well, the current functions in str don't handle decoding buffers in chunks like the proposed encodings API. But yes, it does handle one-off decoding with 'fatal' error handling.
I think we need to improve/extend our API along the lines suggested here for 1.0. Nominating for P-backcompat-lang. (Arguably we could avoid the backwards compatibility hazard by offering a fail-only method for 1.0 and then adding an alternative entry point that provides the more flexible API supporting replacement characters in post 1.0. But I think this case is important enough that we should try to get the primary API method to provide both choices up front. That, or put dynamic fluid support in for representing the state of that choice.)
I submitted a PR for this last night, although I forgot there was an open issue too so I didn't link them. The PR is

Discussion:
if self.filename { match self.path.filename() { None => ~"", <del> Some(v) => from_utf8_with_replacement(v) </del> <ins> Some(v) => str::from_utf8_lossy(v) </ins> } } else { <del> from_utf8_with_replacement(self.path.as_vec()) </del> <ins> str::from_utf8_lossy(self.path.as_vec()) </ins> } } }

v.iter().any(|&x| x == 0) } <del> #[inline(always)] fn from_utf8_with_replacement(mut v: &[u8]) -> ~str { // FIXME (#9516): Don't decode utf-8 manually here once we have a good way to do it in str // This is a truly horrifically bad implementation, done as a functionality stopgap until // we have a proper utf-8 decoder. I don't really want to write one here. static REPLACEMENT_CHAR: char = 'uFFFD'; let mut s = str::with_capacity(v.len()); while !v.is_empty() { let w = str::utf8_char_width(v[0]); if w == 0u { s.push_char(REPLACEMENT_CHAR); v = v.slice_from(1); } else if v.len() < w || !str::is_utf8(v.slice_to(w)) { s.push_char(REPLACEMENT_CHAR); v = v.slice_from(1); } else { s.push_str(unsafe { ::cast::transmute(v.slice_to(w)) }); v = v.slice_from(w); } } s } </del> #[cfg(test)] mod tests { use prelude::*;

// The first byte is special, only want bottom 5 bits for width 2, 4 bits // for width 3, and 3 bits for width 4 macro_rules! utf8_first_byte( <del> ($byte:expr, $width:expr) => (($byte & (0x7F >> $width)) as uint) </del> <ins> ($byte:expr, $width:expr) => (($byte & (0x7F >> $width)) as u32) </ins> ) // return the value of $ch updated with continuation byte $byte macro_rules! utf8_acc_cont_byte( <del> ($ch:expr, $byte:expr) => (($ch << 6) | ($byte & 63u8) as uint) </del> <ins> ($ch:expr, $byte:expr) => (($ch << 6) | ($byte & 63u8) as u32) </ins> ) static TAG_CONT_U8: u8 = 128u8; <ins> /// Converts a vector of bytes to a new utf-8 string. /// Any invalid utf-8 sequences are replaced with U+FFFD REPLACEMENT CHARACTER. /// /// # Example /// /// ```rust /// let input = bytes!("Hello ", 0xF0, 0x90, 0x80, "World"); /// let output = std::str::from_utf8_lossy(input); /// assert_eq!(output, ~"Hello uFFFDWorld"); /// ``` pub fn from_utf8_lossy(v: &[u8]) -> ~str { static REPLACEMENT: &'static [u8] = bytes!(0xEF, 0xBF, 0xBD); // U+FFFD in UTF-8 let mut i = 0u; let mut lastgood = 0u; let total = v.len(); fn unsafe_get(xs: &[u8], i: uint) -> u8 { unsafe { *xs.unsafe_ref(i) } } fn safe_get(xs: &[u8], i: uint, total: uint) -> u8 { if i >= total { 0 } else { unsafe_get(xs, i) } } let mut res = with_capacity(total); while i < total { let i_ = i; let byte = unsafe_get(v, i); i += 1; macro_rules! error(() => { unsafe { if lastgood != i_ { raw::push_bytes(&mut res, v.slice(lastgood, i_)); } lastgood = i; raw::push_bytes(&mut res, REPLACEMENT); } }) if byte < 128u8 { // lastgood handles this } else { let w = utf8_char_width(byte); match w { 2 => { if safe_get(v, i, total) & 192u8 != TAG_CONT_U8 { error!(); continue; } i += 1; } 3 => { match (byte, safe_get(v, i, total)) { (0xE0        , 0xA0 .. 0xBF) => (), (0xE1 .. 0xEC, 0x80 .. 0xBF) => (), (0xED        , 0x80 .. 0x9F) => (), (0xEE .. 0xEF, 0x80 .. 0xBF) => (), _ => { error!(); continue; } } i += 1; if safe_get(v, i, total) & 192u8 != TAG_CONT_U8 { error!(); continue; } i += 1; } 4 => { match (byte, safe_get(v, i, total)) { (0xF0        , 0x90 .. 0xBF) => (), (0xF1 .. 0xF3, 0x80 .. 0xBF) => (), (0xF4        , 0x80 .. 0x8F) => (), _ => { error!(); continue; } } i += 1; if safe_get(v, i, total) & 192u8 != TAG_CONT_U8 { error!(); continue; } i += 1; if safe_get(v, i, total) & 192u8 != TAG_CONT_U8 { error!(); continue; } i += 1; } _ => { error!(); continue; } } } } unsafe { raw::push_bytes(&mut res, v.slice(lastgood, total)) }; res } </ins> /// Unsafe operations pub mod raw { use cast;

// Multibyte case is a fn to allow char_range_at to inline cleanly fn multibyte_char_range_at(s: &str, i: uint) -> CharRange { <del> let mut val = s[i] as uint; </del> <ins> let mut val = s[i] as u32; </ins> let w = UTF8_CHAR_WIDTH[val] as uint; assert!((w != 0));

if w > 2 { val = utf8_acc_cont_byte!(val, s[i + 2]); } if w > 3 { val = utf8_acc_cont_byte!(val, s[i + 3]); } <del> return CharRange {ch: unsafe { transmute(val as u32) }, next: i + w}; </del> <ins> return CharRange {ch: unsafe { transmute(val) }, next: i + w}; </ins> } return multibyte_char_range_at(*self, i);

i -= 1u; } <del> let mut val = s[i] as uint; </del> <ins> let mut val = s[i] as u32; </ins> let w = UTF8_CHAR_WIDTH[val] as uint; assert!((w != 0));

if w > 2 { val = utf8_acc_cont_byte!(val, s[i + 2]); } if w > 3 { val = utf8_acc_cont_byte!(val, s[i + 3]); } <del> return CharRange {ch: unsafe { transmute(val as u32) }, next: i}; </del> <ins> return CharRange {ch: unsafe { transmute(val) }, next: i}; </ins> } return multibyte_char_range_at_reverse(*self, prev);

} #[test] <ins> fn test_str_from_utf8_lossy() { let xs = bytes!("hello"); assert_eq!(from_utf8_lossy(xs), ~"hello"); let xs = bytes!("ศไทย中华Việt Nam"); assert_eq!(from_utf8_lossy(xs), ~"ศไทย中华Việt Nam"); let xs = bytes!("Hello", 0xC2, " There", 0xFF, " Goodbye"); assert_eq!(from_utf8_lossy(xs), ~"HellouFFFD ThereuFFFD Goodbye"); let xs = bytes!("Hello", 0xC0, 0x80, " There", 0xE6, 0x83, " Goodbye"); assert_eq!(from_utf8_lossy(xs), ~"HellouFFFDuFFFD ThereuFFFD Goodbye"); let xs = bytes!(0xF5, "foo", 0xF5, 0x80, "bar"); assert_eq!(from_utf8_lossy(xs), ~"uFFFDfoouFFFDuFFFDbar"); let xs = bytes!(0xF1, "foo", 0xF1, 0x80, "bar", 0xF1, 0x80, 0x80, "baz"); assert_eq!(from_utf8_lossy(xs), ~"uFFFDfoouFFFDbaruFFFDbaz"); let xs = bytes!(0xF4, "foo", 0xF4, 0x80, "bar", 0xF4, 0xBF, "baz"); assert_eq!(from_utf8_lossy(xs), ~"uFFFDfoouFFFDbaruFFFDuFFFDbaz"); let xs = bytes!(0xF0, 0x80, 0x80, 0x80, "foo", 0xF0, 0x90, 0x80, 0x80, "bar"); assert_eq!(from_utf8_lossy(xs), ~"uFFFDuFFFDuFFFDuFFFDfooU00010000bar"); // surrogates let xs = bytes!(0xED, 0xA0, 0x80, "foo", 0xED, 0xBF, 0xBF, "bar"); assert_eq!(from_utf8_lossy(xs), ~"uFFFDuFFFDuFFFDfoouFFFDuFFFDuFFFDbar"); } #[test] </ins> fn test_to_send_str() { assert_eq!("abcde".to_send_str(), SendStrStatic("abcde")); assert_eq!("abcde".to_send_str(), SendStrOwned(~"abcde"));

} #[bench] <ins> fn from_utf8_lossy_100_ascii(bh: &mut BenchHarness) { let s = bytes!("Hello there, the quick brown fox jumped over the lazy dog!  Lorem ipsum dolor sit amet, consectetur. "); assert_eq!(100, s.len()); bh.iter(|| { let _ = from_utf8_lossy(s); }); } #[bench] fn from_utf8_lossy_100_multibyte(bh: &mut BenchHarness) { let s = bytes!("𐌀𐌖𐌋𐌄𐌑𐌉ปรدولة الكويتทศไทย中华𐍅𐌿𐌻𐍆𐌹𐌻𐌰"); assert_eq!(100, s.len()); bh.iter(|| { let _ = from_utf8_lossy(s); }); } #[bench] fn from_utf8_lossy_invalid(bh: &mut BenchHarness) { let s = bytes!("Hello", 0xC0, 0x80, " There", 0xE6, 0x83, " Goodbye"); bh.iter(|| { let _ = from_utf8_lossy(s); }); } #[bench] fn from_utf8_lossy_100_invalid(bh: &mut BenchHarness) { let s = ::vec::from_elem(100, 0xF5u8); bh.iter(|| { let _ = from_utf8_lossy(s); }); } #[bench] </ins> fn bench_with_capacity(bh: &mut BenchHarness) { bh.iter(|| { let _ = with_capacity(100);