Talk:Comparison of Unicode encodings

message: This article appears to be a sub-page of Unicode, which is ok; but it should have an encyclopedic name that reflects its importance (that of an article on Unicode encodings, rather than some evaluative comparison).  —donhalcon╤ 16:26, 7 March 2006 (UTC)
replies: text: It should be moved to Unicode encodings. Once that's done, the opening sentences should be redone to inform readers on the basic who/what/why. --Apantomimehorse 10:11, 10 July 2006 (UTC)
thread_title: Requested move

message: See this page  which describes the encoding. Olivier Mengué | ⇄ 23:19, 22 May 2007 (UTC)
thread_title:  UTF-7,5 ? 

message: This seems to be a bit out of date. I just searched the reference library and can not come up with anything in the current version of Mac OS regarding UTF-16. Since the cited material is two revisions (10.3 vs. the current 10.5) AND since Mac OS has understands UTF-8, the fact that it uses UTF-16 in a previous version for INTERNAL system files, is irrelevant. I suggest this be removed. Lloydsargent (talk) 14:08, 24 April 2008 (UTC)
thread_title:  Mac OS Reference 

message: "A UTF-8 file that contains only ASCII characters is identical to an ASCII file"—Only with the strictest (too strict) reading is this true. A UTF-8 file could have a BOM, which then would not "[contain] only ASCII characters."  Can someone re-word this without making it require such a strict reading yet still be simple?  —Preceding unsigned comment added by 72.86.168.59 (talk) 18:07, 3 September 2009 (UTC)
replies: text: If it has a BOM then it does not consist only of ASCII characters. The BOM must not be required for the file to be handled as UTF-8, this sort of short-sightedness is stopping I18N from being implemented as too much software cannot handle garbage bytes at the start of the file but would have no problem with these bytes inside the file (such as in a quoted string constant).Spitzak (talk) 21:36, 9 November 2009 (UTC)
replies: text: If the file doesn't start with a BOM, the encoding cannot be sniffed in a way that is reliable and not too wasteful. I wish we could just always assume UTF-8 by default, but there are too many files in crazy legacy encodings out there. Software that cannot handle (or even tolerate) Unicode will have to go eventually.--88.73.0.195 (talk) 23:26, 20 May 2011 (UTC)
replies: text: The encoding *can* be sniffed quite well if you assume UTF-8 first, and assume legacy encodings only if it fails the UTF-8 test. For any legacy encoding that uses bytes with the high bit set, the chances of it forming valid UTF-8 are miniscule, like 2% for a 3-character file, and rapidly dropping as the file gets longer. (the only legacy encoding that does not use the high bit set that is at all popular today is ASCII, which is already identical to UTF-8). One reason I really dislike the BOM in UTF-8 files is that it discourages programmers from using this method to determine encoding. The result is that it *discourages* I18N, rather than helping it.Spitzak (talk) 03:09, 21 May 2011 (UTC)
replies: text: STD 63 = RFC 3629 agrees with you, but UTF-8 processors still must be ready to accept (and ignore) a signature (formerly known as BOM, but it clearly is no BOM in UTF-8). Others, notably the XML and Unicode standards, don't agree with you, and a Wikipedia talk page anyway isn't the place to change standards. Sniffing is no good option, UTF-8 as well as windows-1252 can be plain ASCII for the first Megabytes, and end with a line containing ™ - some W3C pages do this (of course UTF-8 without signature, but still an example why sniffing is not easy). –89.204.137.230 (talk) 21:14, 10 June 2011 (UTC)
replies: text: Your example of a file that is plain ASCII for the first Megabytes works perfectly with assumptions that it is UTF-8. It will be drawn correctly for all those megabytes. Eventually it will hit that character that is invalid UTF-8 and it can then decide that the encoding should be something other than UTF-8. Sorry, you have not given an example that does anything other than prove the assumption of UTF-8 is correct. I agree that software should ignore BOM bytes (not just at the start, but imbedded in the file), or treat it as whitespace.Spitzak (talk) 01:27, 11 June 2011 (UTC)

text: Older charsets will go away, but not this year. The IETF policy on charsets predicted 40 years, still more than 25 years until UTF-8 will have gained world dominance. At least all new Internet protocols must support UTF-8. The HTML5 folks will in essence decree that "Latin-1" (ISO 8859-1) is an alias for windows-1252, and the next HTTP RFC also addresses the problem, so for now you can at least assume that Latin-1 is almost always windows-1252. Billions of old Web pages will stay as is until their servers are shut down. –89.204.137.230 (talk) 21:25, 10 June 2011 (UTC)
replies: text: I believe about 90% of the reason older character sets are not going away is due to people not assuming UTF-8 in their detectors, and that the existence of the UTF-8 BOM is the primary reason these incorrect detectors are being written (since programmers then think they should check for it). Therefore the UTF-8 BOM is the primary reason we still have legacy character sets. Web pages in windows-1252 would display correctly if browsers displayed invalid UTF-8 by translating the individual bytes to the windows-1252 characters, and there would be NO reason for any charset and the default could still be UTF-8.Spitzak (talk) 01:27, 11 June 2011 (UTC)
thread_title:  UTF-8 with BOM!!! 

message: Spitzak, your summary in  is wrong. UTF-8 has 8-bit bytes (i.e. coded message alphabet has ≤ 256 symbols, more exactly, 243), but its code points are just Unicode ones. Surrogates are code points, they are not characters indeed. Please, recall the terminology. The word "character" is inappropriate. Incnis Mrsi (talk) 19:47, 23 March 2010 (UTC)
replies: text: I believe you are right. I was confusing this with the units used to make the encoding.Spitzak (talk) 01:00, 24 March 2010 (UTC)
thread_title:  Code points are not bytes, but just opposite thing 

message: I propose to delete this section. "UTF-5" and "UTF-6" are unimplemented vaporware; they were early entries in an IDNA competition which Punycode ultimately won. Doug Ewell 20:14, 20 September 2010 (UTC)
replies: text: IMO nothing is wrong with mentioning encodings in obsolete Internet Drafts, after all Martin Dürst is one of the top ten i18n developers I could name, and helped to develop UTF-5 before they decided to use PunyCode for their IDNA purposes. IIRC I added this section, and refrained from adding my "original research" UTF-4 ;-) –89.204.153.200 (talk) 20:35, 10 June 2011 (UTC)
thread_title:  Historical: UTF-5 and UTF-6 

message: Found this in the UTF-16 RFC: "Characters with values greater than 0x10FFFF cannot be encoded in UTF-16."  (http://www.ietf.org/rfc/rfc2781.txt)  I'm wondering if this is a specific limit of UTF-16?  Can characters above 0x10FFFF be encoding in UTF-8, for example?  (Are there any such characters?) 86.186.87.85 (talk) 10:08, 30 April 2012 (UTC)
replies: text: Yes this is a limit of UTF-16. UTF-8 and initially Unicode was designed to encode up to 0x7FFFFFFF. The Unicode definition was changed to match the limits of UTF-16 by basically saying any encoding of a value greater than 0x10FFFF, no matter how obvious it is, is invalid. No characters are or will ever be assigned codes above 0x10FFFF (personally I feel this is bogus, people believed 128, 2048, and 65536 were limits to character set sizes and these were all broken).Spitzak (talk) 18:10, 30 April 2012 (UTC)
replies: text:  UTF-32 code-point algorithm can encode up to 0xFFFFFFFF, though, making it possible to encode even higher invalid code-points. 112.134.149.92 (talk) 06:57, 6 November 2015 (UTC)
thread_title:  0x10FFFF limit in UTF-16 

message: UTF-16 theoretically can encode codes up to U+FFFFFFFFFF. I'm not sure if UTF-1 examples are correct, but i replaced 5-byte sequences to four.
modified UTF-16:
 0000000000-000000FFFF=        xxxx
 0000010000-0000FFFFFF=    D8xxxxxx
 0001000000-0001FFFFFF=    D9xxxxxx
 0002000000-0002FFFFFF=    DAxxxxxx
 0003000000-0003FFFFFF=    DBxxxxxx
 0004000000-0004FFFFFF=    DCxxxxxx
 0005000000-0005FFFFFF=    DDxxxxxx
 0006000000-0006FFFFFF=    DExxxxxx
 0007000000-FFFFFFFFFF=DFxxxxxxxxxx
The UTF-24:
 00000000-00FFFFFF=      xxxxxx
 01000000-FFFFFFFF=00D8xxxxxxxx
Comparison:
                   UTF-8 UTF-16 UTF-32 UTF-EBCDIC UTF-1 UTF-24
 00000000-0000007F   1     2      4        1        1     3
 00000080-0000009F   2     2      4        1        1     3
 000000A0-000003FF   2     2      4        2        2     3
 00000400-000007FF   2     2      4        3        2     3
 00000800-00003FFF   3     2      4        3        2     3
 00004000-00004015   3     2      4        4        2     3
 00004016-0000FFFF   3     2      4        4        3     3
 00010000-00038E2D   4     4      4        4        3     3
 00038E2E-0003FFFF   4     4      4        4        4     3
 00040000-001FFFFF   4     4      4        5        4     3
 00200000-003FFFFF   5     4      4        5        4     3
 00400000-006C3725   5     4      4        6        4     3
 006C3726-00FFFFFF   5     4      4        6        5     3
 01000000-03FFFFFF   5     4      4        6        5     6
 04000000-06FFFFFF   6     4      4        7        5     6
 07000000-3FFFFFFF   6     6      4        7        5     6
 40000000-4E199F35   6     6      4        8        5     6
 4E199F36-FFFFFFFF   6     6      4        8        6     6
164.127.155.17 (talk) 20:40, 6 March 2015 (UTC)
replies: text: In your "modified UTF-16" you re-use 16-bit codewords. e.g. the 16-bit codeword 002Fhex can represent U+002F (the "slash" /) or can be part of a longer sequence. This has a lot of drawbacks, hence UTF-8 and UTF-16 are designed in a way, that such "double usage" of codewords never happened. --RokerHRO (talk) 08:10, 11 March 2016 (UTC)
thread_title:  Extended Table 

message: "Default" implies that "this is what you get if you don't declare the encoding." But the cited section doesn't say that. It seems to say that you should assume UTF-8 if there's no BOM. (I'm not clear what it should assume if there is a BOM.) Even if "UTF-8 is the default" is true in some technical sense, the paragraph would be clearer like this:
All XML processors must support both UTF-8 and UTF-16. If there is no encoding declaration and no byte order mark, the processor can safely assume the file is UTF-8. Thus a plain ASCII file is seen as UTF-8. Isaac Rabinovitch (talk) 19:27, 16 April 2022 (UTC)
replies: text: That is exactly what "UTF-8 default" means. If there is no indication as to what the encoding is, use UTF-8 by default.Spitzak (talk) 22:02, 16 April 2022 (UTC)
thread_title:  UTF-8 Default?