Talk:UTF-16

message: Concerning the text explaining UTF-16LE and UTF-16BE, would it not be better that instead of saying,
:A BOM at the beginning of UTF-16LE or UTF-16BE encoded data is not considered to be a BOM; it is part of the text itself.
we say something like,
:No BOM is required at the beginning of UTF-16LE or UTF-16BE encoded data and, if present, it would not be understood as such, but instead be mistaken as part of the text itself.
--Chris 17:27, 12 January 2006 (UTC)
thread_title: UTF-16LE BOMs Away!

message: The example: 119070 (hex 1D11E) / musical G clef / D834 DD1E:
the surrogate pair should be D874 DD1E for 1D11E.
Can somebody verify that and change the example? —Preceding unsigned comment added by 85.216.46.173 (talk) 
:NO the surrogate pair in the article is correct (and btw this incorrect correction has come up many times in the articles history before)
:0x1D11E-0x10000=0x0D11E
:0x0D11E=0b00001101000100011110
:split the 20 bit number in half
:0b0000110100 =0x0034
:0b0100011110 =0x011E
:add the surrogate bases
:0x0034+0xD800=0xD834
:0x011E+0xDC00=0xDD1E
: -- Plugwash 18:39, 8 November 2006 (UTC)
thread_title:  Surrogate Pair Example Wrong? 

message: Could there be an example for decoding the surrogate pairs, similar in format to the encoding example procedure? Neilmsheldon 15:29, 27 December 2006 (UTC)
thread_title:  Decoding example 

message: The first reference to the apache xalan bug  seems broken to me. I'm very interested in that particular problem, could someone fix the link? What was it all about? What was the problem? It is really unclear how it is right now.  —Preceding unsigned comment added by 160.85.232.196 (talk) 12:52, 20 April 2010 (UTC)
thread_title:  First reference to apache xalan UTF-16 surrogate pair bug 

message: The article says 'the UTF-8 decoder to "Unicode" produces correct UTF-16' - that seems to be either wrong or something that could use a longer explanation. Was 'the UTF-16 encoder from "Unicode" produces correct UTF-16' meant? Kiilerix (talk) 15:05, 30 July 2011 (UTC)
replies: text: When compiled for 16-bit strings, a "Unicode" string cannot possibly contain actual Unicode code points, since there are more than 2^16 of them. Experiments with the converter from UTF-8 to "Unicode" reveals that a 4-byte UTF-8 sequence representing a character > U+FFFF turns into two "characters" in the "Unicode" string that correctly match UTF-16 code units for this character. However I believe in most cases Python "Unicode" strings are really UCS-2, in particular the simple encode api means that encoders cannot look at more than one code unit at a time. On Unix where Python is compiled for 32-bit strings, the encoder from "Unicode" to UTF-16 does in fact produce surrogate pairs.Spitzak (talk) 20:41, 1 August 2011 (UTC)
thread_title: Python UTF-8 decoder?

message: I removed a sentence that was added saying "all the code that handles text has to be changed to handle variable-length characters".
This is not true, and this misconception is far more common when discussing UTF-8, where it is equally untrue and has led to a great deal of wasted effort converting UTF-8 to other forms to "avoid" the actually non-existent problem.
As far as I can tell the chief misconception is that somehow strlen() or indexing will somehow "not work" if the index is not "number of characters". And for some reason the obvious solution of indexing using the fixed-size code units is dismissed as though it violates the laws of physics.
There seems to be the idea that somehow having a number that could point into the "middle" of a letter is some horrendous problem, perhaps causing your computer to explode or maybe the universe to end. Most programmers would consider it pretty trivial and obvious to write code to "find the next word" or "find the Nth word" or "find the number of words in this string" while using an indexing scheme that allows them to point into the "middle" of a word. But for some reason they turn into complete morons when presented with UTF-8 or UTF-16 and literally believe that it is impossible.
The other misconception is that somehow this "number of characters" is so very useful and important that it makes sense to rewrite everything so that strlen returns this value. This is horribly misguided. This value is ambiguous (when you start working with combining characters, invisible ones, and ambiguous ideas about characters in various languages) and is really useless for anything. It certainly does not determine the length of a displayed string, unless you restrict the character set so much that you are certainly not using non-BMP characters anyway. The number of code units, however, is very useful, as it is directly translated into the amount of memory needed to store the string.
In fact on Windows all equivalents of strlen return the number of code units. The changes were limited to code for rendering strings, and fixes to text editor widgets. The Windows file system was updated to store UTF-16 from UCS-2 with *NO* changes, and it is quite possible to create files with invalid UTF-16 names and use them. 
This same observation also applies to use of UTF-8. Here the misconception that "everything has to be rewritten" is even more pervasive. Don't know how to stop it but me and obviously several others have to continuously edit to keep this misinformation out of Wikipedia.
Spitzak (talk) 23:25, 23 September 2012 (UTC)
thread_title:  Difficulty of converting from UCS-2 to UTF-16 

message: The article says, "It is possible to unambiguously encode them [i.e. code points D800-DFFF] in UTF-16, as long as they are not in a lead + trail pair, by using a code unit equal to the code point."
This doesn't make sense to me. It seems to be saying that you just drop in a code unit like D800 or DC00 and software should understand that you intend it to be a standalone code point rather than part of a surrogate pair. I suppose if this results in an invalid code pair being formed (such as a lead unit without a valid trail unit following), then the decoder could fall back to treating it as a standalone code unit, but what if you want to encode point D800 followed immediately by point DC00? Wouldn't any decoder treat the sequence D800 DC00 as a valid surrogate pair rather than as a pair of illegal code points? If so, then the statement that UTF-16 can "unambiguously encode them" (i.e. all values in the range D800-DFFF) is not true.
If I have misunderstood the point here, please clarify it in the article, because it isn't making sense to me the way it's written.  — Preceding unsigned comment added by 69.25.143.32 (talk) 18:09, 4 October 2012 (UTC)
thread_title:  Encoding of D800-DFFF code points in UTF-16 

message: The 1st reference (math) is not actually a reference, it should be a note instead.  — Preceding unsigned comment added by Tharos (talk • contribs) 14:23, 23 August 2013 (UTC)
thread_title:  1st reference 

message: In the section title "Code points U+0000 to U+D7FF and U+E000 to U+FFFF" and in the text therein, example: "The first plane (code points U+0000 to U+FFFF) contains...", a "U+" notation is used. This notation is also encountered in many other places, both inside and outside Wikipedia. I've never seen a meaning assigned. What does "U+" mean? - MarkFilipak (talk) 16:30, 22 July 2014 (UTC)
replies: text: "U+" is the standard prefix to indicate that what follows is the hexadecimal representation of a Unicode codepoint, i.e., the hex number of that "character" in Unicode.  See Unicode. -- Elphion (talk) 17:02, 22 July 2014 (UTC)
thread_title:  What does "U+" mean? 

message: The lede talks mostly about UCS-2. UCS-2 is OBSOLETE! Perhaps the article means UCS as described in the latest ISO 10646 revision ??? I do NOT have a solid understanding of UTF-16, but if my math is correct:
0x0000 to 0xD7FF plus 0xE000 to 0xFFFF is 55,296 + 8,192 = 63,488 -doesn't this mean that there are 2047 code points from the BMP which are NOT addressable in UTF-16 ? (65535-63488 = 2047). In the description it says:"The code points in each plane have the hexadecimal values xx0000 to xxFFFF, where xx is a hex value from 00 to 10, signifying the plane to which the values belong." Code points have "values" different from their code point values?!?!?! Wow, somebody really is inarticulate. Lets agree that any binary representation of a "character" is dependent on some system of rules, ie a standard. We should also (maybe? I hope!) be able to agree that "code page" is ambiguous, and IMPLEMENTATION DEPENDENT (I can't speak to whether the term exists in any of the ISO/Unicode standards, but it is clear that different standards (ANSI, Microsoft, DBCS, etc.) define 'code page' differently.) So, using a common (but poorly defined) term in this article needs to be done with much more caution. I really don't understand why UCS is used as the basis of comparison for this article on UTF-16. Shouldn't Unicode 6.x be THE basis?  I would suggest that if there is need for an article on UCS, then there should be an article on UCS, or at least a comparison article between it and Unicode. Afaics, this article lacks any mention of non-character Unicodes as well as text directionality, composition, glyphs, and importantly, graphemes. If UTF-16 is similar to UCS then this is a serious deficiency. It also seems to me that Microsoft is a huge part of the reason UTF-16 persists, and how it is implemented in various MS products & platforms should be addressed here. Also, I see virtually NOTHING on UTF-8, which is (it seems) much more common on the internet. All-in-all, this article does not present a clear picture of what "standard" UTF-16 is, and what the difference is between that and what its common implementations do.  — Preceding unsigned comment added by 72.172.1.40 (talk) 20:15, 28 September 2014 (UTC)
replies: text: I have already tried to delete any mention of "planes" but it got reverted, so I gave up. I agree that planes are irrelevant to describing UTF-16. You are correct that there are 2048 points that can't be put in UTF-16 and this causes more trouble than some people are willing to admit because they don't want to admit the design is stupid.Spitzak (talk) 21:45, 28 September 2014 (UTC)
thread_title:  This article is incoherent. It needs a complete revision. 

message: The last example seems to be wrong. It lists the symbol "軔", which is the codepoint U+8ED4http://unicode-table.com/en/search/?q=%E8%BB%94http://www.fileformat.info/info/unicode/char/8ed4/index.htm (JavaScript: "軔".charCodeAt(0).toString(16) and Python 3: hex(ord("軔")) agree), but the example says it is the codepoint U+2F9DE.http://www.fileformat.info/info/unicode/char/2f9de/index.htm It looks similar, but it isn't the same codepoint.
panzi (talk) 23:48, 25 October 2014 (UTC)
replies: text: You're quite right. U+2F9DE is a compatibility ideograph that decomposes to U+8ED4 軔 on normalization.  As any software might (and presumably has in this case) silently apply normalization and therefore convert U+2F9DE to U+8ED4 it would be best not to use a compatibility ideograph for the example, but use a unified ideograph. BabelStone (talk) 12:32, 26 October 2014 (UTC)
replies: text: OK. I just grabbed something from our Wikibooks UNICODE pages in the range of bit patterns I wanted to demonstrate. Thanks for catching the wrong bit 0xD bit pattern, too! —[AlanM1(talk)]— 02:40, 29 October 2014 (UTC)
thread_title:  Example for 軔 is wrong 

message: At present, the article refers to the halves of a surrogate pair as "lead" and "trail", and says that "high" and "low" was the terminology in "previous versions of the Unicode Standard". This appears to be incorrect; Unicode v7.0 calls them "high" and "low", and so does v6, which was (specifically v6.1) the current version when the change was made, in April 2012. What the Standard does say is:
: Sometimes high-surrogate code units are referred to as leading surrogates. Low-surrogate code units are then referred to as trailing surrogates. This is analogous to usage in UTF-8, which has leading bytes and trailing bytes.
Now, an earlier edit (October 2011) suggests that the change was made (both in Unicode and here) to avoid confusion over "high" surrogates having lower code-point numbers than "low" surrogates. That's a laudable goal, but the Unicode Standard doesn't say that was the intention, and so we need a citation. And in any case, Unicode still plainly uses "high" and "low" as the primary terms. (Note also that the alternatives are given as "leading" and "trailing".) I'm going to go change the article accordingly, but I'd be very interested to see if sources can be found arguing for the alternative names (or maybe even use of them in ISO 10646?). -- Perey (talk) 12:49, 3 November 2014 (UTC)
thread_title:  Surrogate terminology 

message: Why not use D8-DF start bytes as surrogates?
 0000000000-000000FFFF=        xxxx
 0000010000-0000FFFFFF=    D8xxxxxx
 0001000000-0001FFFFFF=    D9xxxxxx
 0002000000-0002FFFFFF=    DAxxxxxx
 0003000000-0003FFFFFF=    DBxxxxxx
 0004000000-0004FFFFFF=    DCxxxxxx
 0005000000-0005FFFFFF=    DDxxxxxx
 0006000000-0006FFFFFF=    DExxxxxx
 0007000000-FFFFFFFFFF=DFxxxxxxxxxx
PiotrGrochowski000 (talk) 08:02, 7 March 2015 (UTC)
replies: text: On Wikipedia we describe what is, not fantasize about what could have been.  As such, your comment is out of scope here (see WP:TALKNO). BabelStone (talk) 11:33, 7 March 2015 (UTC)
thread_title:  Single byte surrogates 

message: The History section mentions IEEE, but I’m not aware of IEEE ever being involved in Unicode or UTF-16. Could the author have been confused with ISO? ISO/IEC 10646 is related to Unicode.
I also think this is a mistake. The claim should be supported by a reference, or removed. Jsbien (talk) 08:43, 22 May 2019 (UTC)
replies: text: Yes, IEEE is a mistake. I have replaced IEEE with ISO/IEC JTC 1/SC 2 which is the ISO/IEC subcommittee responsible for the development of the Universal Coded Character Set. BabelStone (talk) 11:05, 22 May 2019 (UTC)
thread_title:  IEEE v.s. ISO confusion? 

message: The article says
"They neglected to define a method of encoding these code points, thus leading to a "hole" in the set of possible code points, a source of some difficulty when dealing with Unicode. This mistake was not made with UTF-8."
This implies that the hole was a "mistake". The alternative to a hole would have been making it so UCS-2 and UTF-16 encoded the same code point in different ways which would have made things even more confusing given that the purpose of UTF-16 was clearly to be a successor to UCS-2. 130.88.154.105 (talk) 14:55, 23 June 2015 (UTC)
replies: text: UTF-8 uses a different method of encoding the code points 128-255 than 8-bit encodings did, and I don't think anybody is "confused" and I'm pretty certain if those codes were impossible it would be considered a mistake. There is absolutely no reason UTF-16 could not have been designed to handle a continuous sequence of code points except that the programmers were lazy. Since they had to ditch simple sorting order anyway, I think it would have been ok to put these 2048 code points at the very end of the possible code sequences.Spitzak (talk) 02:01, 24 June 2015 (UTC)
replies: text: But several of those were already assigned characters.  They chose a range that wasn't already occupied. -- Elphion (talk) 02:29, 24 June 2015 (UTC)
replies: text: Of course they chose a range unoccupied by characters. That does not mean that you should be unable to encode those code points. If that was allowed they could have chosen *every* unassigned code point and said they all were to encode code points greater than 0xFFFF.Spitzak (talk) 21:20, 24 June 2015 (UTC)
replies: text: I wasn't arguing whether the surrogates should be encodable, just pointing out why the use of a non-terminal range was not a "mistake".  It was in fact critically important that they chose a range that didn't alter existing UCS-2 assignments, since otherwise several important manufacturers would never have bought into UTF-16 (and therefore into any form of Unicode for codepoints above U+FFFF).  At this point the only way I can think of to make the surrogates encodable in a standard way would be to use one of the reserved codepoints below U+FFFF as an escape character, signifying that the next codepoint is to be taken literally, even if it is a surrogate or otherwise reserved.  The standards people don't seem to be convinced that this is important (i.e., they feel that one shouldn't be encoding such codepoints anyway), anymore than, say, overlong UTF-8 encodings should ever be preserved when those are found in the input message.  This isn't strictly a UTF-16 issue; it boils down to how to retain strictly illegal encodings in any of the UTF schemes in a way that is portable and acceptable to the standard (so that standards-conforming decoders don't fail on that input). -- Elphion (talk) 16:28, 25 June 2015 (UTC)
thread_title:  biased wording about the hole 

message: When, under the section entitled Byte order encoding schemes, the article says that
"If the endian architecture of the decoder matches that of the encoder, the decoder detects the 0xFEFF value, but an opposite-endian decoder interprets the BOM as the non-character value U+FFFE reserved for this purpose. This incorrect result provides a hint to perform byte-swapping for the remaining values. If the BOM is missing, RFC 2781 says that big-endian encoding should be assumed."
and then follows with
(In practice, due to Windows using little-endian order by default, many applications also assume little-endian encoding by default.)
isn't the article contradicting itself?  An application can't assume two defaults.  Shouldn't the word 'also' above be omitted? 
-- SourceMath (talk) 13:30, 25 June 2015 (UTC)
replies: text: The article means "many applications similarly assume little-endian encoding by default."  You're right, the "also" construction can be misinterpreted. -- Elphion (talk) 16:27, 25 June 2015 (UTC)
thread_title:  Seemingly contradictory statement 

message: Which languages exist in utf-16 that don't exist in ucs-2?
It is implied by the article that there are some missing, but it isn't clearly stated anywhere.
84.94.119.60 (talk) 12:01, 23 August 2015 (UTC)
replies: text: I wrote a short note about this in the article, and a link to the main article answering this question, Plane (Unicode). --BIL (talk) 18:07, 23 August 2015 (UTC)
replies: text:  UCS-2 is obsolete. It was one written in 1990, 25 years ago. Comparing it to UTF-16 SHOULD be an academic question (in a perfect world). Or perhaps I'm wrong? WHY is there so much included here about it? Anything added to Unicode since then will not have any UCS-2 representation, and most Unicode code points have been created since then. But if there is a good reason that so much space is used to describe this ancient standard here, it should be explained. I'd go so far as to remove all but one mention of UCS-2 from this, leaving one in the lede since some people seem to think UCS-2 is a synonym for UTF-16 and using it to link to an UCS-2 article.216.96.79.108 (talk) 13:30, 24 September 2015 (UTC)
replies: text: Why is there "so much" about USC-2 here?  (There's really not very much.) Because UCS-2 redirects here, and this is the natural place to describe it, since it is the basis on which UTF-16 was built.  It's fairly clear from the article that UCS-2 is obsolete.  We can say that explicitly, say by adding the following at the end of the first paragraph of the History section:  "UCS-2 represents an early, incomplete, and now obsolete version of UTF-16 that is limited to the codepoints that can be represented in 2 bytes."  -- Elphion (talk) 18:08, 24 September 2015 (UTC)
thread_title:  languages missing from UCS-2 

message: There should be some clear explanation of the system used to designate the code points. As this article is now, the reader is assumed to be familiar with hexadecimal as well as common computer science representations of hexadecimal numbers. U+ indicates that the number following will be a hexadecimally encoded Unicode code point.
U+hhhh and U+hhhhh where h represents 1 hexadecimal digit (0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F) and is in the range of 0 to 1 114 111 (decimal) or 0 to 10FFFF (hexadecimal). In several computer languages, the symbols 0x are prepended to a hexadecimal number to distinguish it from a decimal number. Thus, 0 = 0x0000 but 10 = 0x000A or sometimes 0xA (the leading zeroes are often removed). This is useful to distinguish a number such as 100 and 0x100 since 0x100 is the hexadecimal representation for 256. It is possibly outside the scope of this article to also mention that Unicode code points are not necessarily unique (representations) and that there may be several ways to specify a character or glyph which have identical meaning. Parts of the specification originated by historical accident and is based on various pre-existing ("pre electronic-age") national and international standards.(Examples include ANSI, ASCII, and Simplified Chinese.) So, character (code point) order is not always the same as alphabetical order in the language which uses those code points. Code points may also represent symbols as well as meta-data (without a glyph representation, an example is U+0082 which means that a line break is allowed at that location in the text; U+0082 has no glyph associated with it.)216.96.79.108 (talk) 13:15, 24 September 2015 (UTC)
replies: text: We should mention at the beginning of the Description section that U+hex is the standard way of referring to Unicode codepoints, and provide a link to Unicode#Architecture and terminology, where this is described in more detail (and perhaps to Code point and Hexadecimal as well).  Most of the rest of what you mention is discussed elsewhere, and is not appropriate for this article, which is specifically about the encoding. -- Elphion (talk) 18:17, 24 September 2015 (UTC)
thread_title:  Jargon 

message: Can the surrogate halves (0xD800 to 0xDFFF) be encoded in UTF-32? 108.66.235.240 (talk) 18:22, 25 September 2016 (UTC)
replies: text: No. Surrogate code points are only used in UTF-16, and are invalid in UTF-32 an UTF-8. BabelStone (talk) 18:39, 25 September 2016 (UTC)
replies: text: They can technically be encoded in UTF-32 and UTF-8, but that's not valid according to Unicode standard. Single surrogate halves are also not really valid. Read the UTF-16#U+D800 to U+DFFF chapter in the article which says single surrogate halves are often tolerated by decoders. --BIL (talk) 20:51, 25 September 2016 (UTC)
thread_title: Surrogate halves

message: Nobody writes hexadecimal 0 as just 0. 108.65.81.120 (talk) 00:54, 3 October 2016 (UTC)
thread_title: 0 vs 0x0000

message: please make ucs-2 article. i think making them 1 article is wrong. 1) ucs-2 is constant length encoding while utf-16 is variable length encoding which makes utf-16 complicated and longer calculated. for example cutting string after letter number 100 or getting letter number 100 should be nearly 100 times longer for utf-16 string than for ucs-2 string! 2) maybe there are some additional letters in 2 bytes of ucs-2 compared to 2 bytes of utf-16 string which is wrongly simply divided after every 2 bytes (constantly instead of variably 2 or 4 bytes), because some character(s) in first 2 bytes of utf-16 character should be used to show that there are additional 2 bytes of current character , while that special code(s) in first 2 bytes of utf-16 could be used for additional letter(s) in ucs-2. --Qdinar (talk) 18:04, 13 July 2017 (UTC)
replies: text: that special codes that show presence of additional 2 bytes are d800-dfff and they are reserved by unicode for that purpose, so they do not encode characters and cannot be used for private use. but, i think, technically they can be used by programmers for private use if they use constant length encoding. --Qdinar (talk) 19:14, 13 July 2017 (UTC)
thread_title:  please make ucs-2 article 

message: "It is possible to unambiguously encode an unpaired surrogate..." This paragraph has been mentioned two times already on this talk page and it still doesn't make sense to me. I pondered it several times and I tried to find a way to actually do this, but I failed. Either (A) I'm just too stupid to get it, or (B) this paragraph confuses "encode" and "decode" or "code units" and "code points" or uses the same term in different meanings, or (C) this is  actually wrong. Sure, you can interpret one half of a surrogate pair as a UTF-8 character, but it would be wrong. Is this what is meant? See also https://unicode.org/faq/utf_bom.html#utf8-4 . The way it is written now sounds like it is somehow magically possible to squeeze a value >= 0x10000 into 16 bit. Please clarify or remove this paragraph. --213.68.42.195 (talk) 12:04, 9 March 2020 (UTC)
replies: text: You didn't quote the sentence in its entirety. The paragraph as written in the article is indeed factually correct and uses its terms properly. The encoding is indeed unambiguous when the surrogate isn't followed/preceded by a matching surrogate. E.g. the sequence of UTF-16 *code units* '0020 D800 0020' can be unambiguously interpreted as the sequence of *codepoints* U+0200, U+D800, U+0020. Granted, however, that such encoding is deemed invalid UTF-16 by the Unicode standard. This is not a theoretical exercise though. A lot of software that uses UCS-2 or "UTF-16" internally will happily accept, store, and transmit, such sequences. 2604:6000:9981:7600:F0E0:89C2:B327:B147 (talk) 13:37, 9 March 2020 (UTC)
replies: text: I can see the point about perhaps "decode" is a better word than "encode". Imagine a really bad encoder that turns every character into the same code unit (1, for instance). I guess you could say that encoder is not unambiguous, it knows exactly what to do with every character (turn it into 1). But the *decoder* for this encoding has a serious problem and is ambiguous (it could turn a 1 into any character). In any case the point was that a high surrogate not followed by a low, or a low not preceded by a high, can be turned into a UTF-16 code unit in a way that a decoder cannot possibly think it is some other character (it is also an obvious encoding, there are an infinite number of un-obvious encodings such as swapping the high and low values when writing code units).Spitzak (talk) 19:18, 9 March 2020 (UTC)
replies: text: Then I guess I over-interpreted the term "unambiguous". The actual meaning being that the output of such an encoding is wrong, but it is unambiguously wrong? But how do you encode an unpaired surrogate in UTF16? A surrogate already is the encoding, which only makes sense in UTF16. I still think this is confusing and deserves a clarification. How about a full round-trip example (input -> encoding -> interpretation by decoder)? 193.16.224.9 (talk) 10:20, 5 June 2020 (UTC)
thread_title:  Unpaired surrogates 

message: I edited the History section to attempt to answer some of the issues above.  But the last paragraph in that section is still pretty impenetrable, and refers to "surrogates", which have not been introduced yet.  How much of that paragraph needs to be preserved?  How could it be worded better? -- Elphion (talk) 17:13, 3 November 2020 (UTC)
replies: text: I know that most articles have the history up front, but I suggest that there is a strong case in this article to have it last. In most other cases, it is difficult to explain where we are today without first explaining how we got here, but in this case I believe that the history casts more shadow than it does light. Having it at the end means that readers have more of helicopter view and can better understand it. It also solves the surrogate problem!  --John Maynard Friedman (talk) 17:28, 3 November 2020 (UTC)

text: Of course it would also mean that the Description section couldn't just dive straight in with this sort of strange statement: When reading the first 16 bits of a character encoding, a direct 16-bit character would occupy the lower (0x0000 to 0xD7FF) range or the upper (0xE000 to 0xFFFF) range. If it's a 32-bit character encoding (with room for 20-bit character codes), the unused interval in between is used to signal that another 2 bytes should be read. What??? Wait up!! Let's get back to first principles: it should say something like "a 16 bit string is capable of encoding 216 (65,536) code-points and no more, but the number of world letters, symbols, signs and emojis that need to be encoded is many times more than that. UTF-16 resolves this issue by using just ten bits for characters encoding and reserves the remaining six bits to indicate where those characters are to be found". Or something like that? --John Maynard Friedman (talk) 17:45, 3 November 2020 (UTC)
thread_title:  The History section 

message: Just a private question: Anyone here who might be able to help me with my question at WP:RD/C#Mojibake of surrogate characters? ◅ Sebastian 12:19, 21 June 2023 (UTC)
replies: text: This question, while unanswered, has been archived at WP:Reference desk/Archives/Computing/2023 June 15#Mojibake of surrogate characters. ◅ Sebastian 12:14, 11 August 2023 (UTC)
thread_title:  Mojibake of surrogate characters 

message: Currently, surrogate pairs redirects here, while surrogate pair redirects to Universal Character Set characters. I feel the topic fits better here, but the latter redirect has the advantage that it goes to a dedicated section. Where should surrogates best be covered? A dedicated section here would encompass more than half of this article; that would probably be too big. IMHO, they are important enough to have an article on their own, in which case much of this article would be moved there. Any opinions? ◅ Sebastian 12:19, 21 June 2023 (UTC)
replies: text: Makes sense to me. The UCS article is pretty bloated with more than one section about surrogates. Spitzak (talk) 14:52, 21 June 2023 (UTC)
replies: text:  Thanks, Spitzak. Would you or anyone else help me with the creation of the new article? ◅ Sebastian 10:54, 11 August 2023 (UTC)
thread_title:  Coverage of surrogates 

message: Currently, the formula for the supplementary code conversion at  contains two arithmetic operators: + and -. While the latter is used in the correct arithmetic sense, the former is apparently used in the sense of a text concatenation. (Or in the sense of some preschool kids who think that 1+2=12.) Should we use a dedicated operator from Concatenation#Syntax, or use plain text? (Presumably, the intension might have been to emulate overloading, but that's very confusing here, where all operands can be seen as integers. Possibly there was some intention to express this through selective use of 0x, but that's quite idiosyncratic.) ◅ Sebastian 13:39, 11 August 2023 (UTC)
replies: text: No, just ordinary addition.  0xD800 = 0b1101100000000000 and 0xDC00 = 0b1101110000000000, so the values to the left of the comment are the result of adding those two hex values to yyyyyyyyyy and xxxxxxxxxx, respectively. Guy Harris (talk) 20:10, 11 August 2023 (UTC)
replies: text:  You're right, thanks! I missed that the values without 0x are meant to be 0b. That's already so in the source, but still it may not be the best presentation for an encyclopedic article, as it's not immediately obvious to all readers. ◅ Sebastian 14:58, 12 August 2023 (UTC)
thread_title:  What operator to use for the supplementary code conversion? 

message: Headline inserted 12 August 2023
Oh, and what's the point of the double slashes? Presumably, they stand for comments, but - with the rest of the formulas correct - the appropriate sign would be simply =. Note that the // are rather idiosyncratic, as well, as the're not in the source. ◅ Sebastian 17:01, 11 August 2023 (UTC)
replies: text: Yes, the point of the double slashes is to introduce comments, C++/C99-and-later-style.  The comments are there to indicate the formula used to calculate the values to the left of the comments; the items to the left of the comments and the right of the equal signs are the bit representations of U', W1, and W2. And, yes, showing them as {name} = {bit representation} = {formula} might also work. Guy Harris (talk) 20:14, 11 August 2023 (UTC)
replies: text:  Of course double slashes are used by some languages for comments. But that's not the question here. My question is: What's the point? Why use an unintroduced syntax (which is probably not known to all readers) to “illustrate” something that can be expressed with a generally known sign like the equal sign? And even if every reader knew the meaning of //: If an explanation itself needs a (meta-) comment, then it's not a good explanation. ◅ Sebastian 14:14, 12 August 2023 (UTC)
thread_title:  Why comments?