Talk:UTF-32

message: Yes, it's a unicode encoding. Yes if you want to know the details you can go read the standards. But this encoding ought to be simple enough to explain on this page. Specifically I came here to see whether the code points should be encoded little endian or big endian, i.e. should the least significant byte or most significant byte go first. Another aspect of UTF-32 is the possibility of a byte order mark indicating whether the following data used little endian or big endian.
Currently unicode defines 3 32-bit encodings. Their names all start with UTF-32. This page has no mention of them or how they differ.
Unicode standards have evolved since the old confusing days. The language, definitions, and FAQs have done a good job of clearing up a lot of confusion and I highly recommend checking that out for anyone who is interested. Good luck Wikipedia!  2600:1700:BA69:10:C50D:52CC:4048:FCA7 (talk) 06:16, 7 July 2023 (UTC)
thread_title:  Doesn't explain the details of the encoding 

message: This is not an encoding, but simply Unicode itself. If a character's Unicode code is 42, then the 32 bit integer which holds
42 is not "UTF-32". It's just the code of that character. UTF-32 is merely a synonym for Unicode.  UTF-32('A') is 65,
and Unicode('A') is 65. Same thing.
When we have a string of ASCII characters encoded as 8 bit values, do we call that ATF-1? No, it is just an ASCII string.
Another problem is that if this is an encoding, what is the byte order? Is the character 'A' stored as 00 00 00 41 or is it stored as 41 00 00 00? If this is an encoding, we should be able to answer such a basic question. Encoding means settling all issues of bitwise representation for the purposes of transport.
24.85.131.247 (talk) 22:49, 29 October 2011 (UTC)
replies: text: No, it's a encoding (big-endian, by default on networks), just your example is a hypothetical encoding: "When we have a string of ASCII characters encoded as 8 bit values, do we call that ATF-1?" you might, it would be an 8-bit encoding of ASCII; note ASCII characters aren't defined as 8-bit, they are seven-bit. Everything in computers is an encoding.. coding something, say colors, like letter (or even integer numbers), that do not have intrinsic "number"/bit pattern. comp.arch (talk) 22:26, 24 September 2016 (UTC)
thread_title:  This is not an encoding 

message: This notation seems more important, as it suggests that the number is 32 bit. 108.66.232.241 (talk) 20:57, 5 November 2016 (UTC)
thread_title:  0 -> 0x00000000 

message: For these reasons UTF-32 is little used in practice with UTF-8 and UTF-16 being the normal ways of encoding Unicode text
i disagree with this statement. wchar_t in unix/linux C applications is in UTF-32 format. This makes it pretty often used. Vid512 17:28, 19 March 2007 (UTC)
replies: text: Also UTF-32 is used as the internal format for strings in the Python programming language—the C-based reference implementation at any rate.  (Actually it uses UCS-4, as Python does not impose the restriction against lone high or low surrogates from being encoded).  As there are no referenced facts to support this statement and there are clear uses of UTF-32/UCS-4 in systems today, I'm removing the claim that it's not used. - Dmeranda 04:44, 5 August 2007 (UTC)
replies: text: I noticed this and also disagreed.  But when I read that Dmeranda removed it, well, it was still there!  Checking, it appears he removed another statement instead.  I've restored that and removed the above claim.  mdf (talk) 15:38, 27 November 2007 (UTC)

text: CPython can be compiled to use either UCS-4 or UCS-2 internally.  It defaults to UCS-2, but many Linux distributions compile it to use UCS-4. Agthorr (talk) 15:44, 9 May 2010 (UTC)
thread_title:  UTF-32 not used? 

message: Well first, the edit comment is a bit inaccurate. I quickly looked through the history and I thought I saw the cleanup message there but upon further examination it seems I was wrong. So it should instead say that the cleanup tag is wrong in that it was most certainly not there since September 2007, and the article is in fine shape (though not the best it could be), so it shouldn't be there.
As for the dubious tags, the claim that it's more space efficient is well justified by the following sentence, which notes that non-BMP characters are rare. This is by design; the BMP is intended to contain pretty much every character in major (and most minor) modern languages, as the standard notes . The BMP takes 2 bytes in UTF-16 and 1 to 3 in UTF-8, so for text consisting of BMP characters, UTF-32 obviously takes more space. For a real world example, with a file consisting of large amounts of Japanese and ASCII text (all in the BMP), it is 10MB with UTF-8, 14MB with UTF-16, and 28MB with UTF-32.
For the claim that it's rarely used, Unixy systems use UTF-8, Windows uses UTF-16, various programming languages mostly use either (though I know Python can use UCS-4 if you compile it so). Another message on this page talks about wchar_t, but that's implementation-specific, and the Unicode standard even advises against it for code that's supposed to be portable for this reason . In my experience it doesn't seem to be used nearly as much as others, though I admit my experience in this area isn't quite vast. Regardless, a completely implementation-specific data type in a single language hardly changes matters.
By those reasons I've removed those tags. The article might do with a few citations, but there's no dubious information in it, and though it could be improved, it's written well enough that it does not require a cleanup. 24.76.174.152 (talk) 07:15, 19 November 2009 (UTC)
thread_title:  Removing cleanup and dubious tags 

message: The History section reads: "UCS-4, in which each encoded character in the Universal Character Set (UCS) is represented by a 32-bit friendly code value in the code space of integers between 0 and hexadecimal 7FFFFFFF." Shouldn't this be "each encoded code point"? A 32-bit value doesn't necessarily represent one character, come characters are composed of several values. Tigrisek (talk) 19:45, 16 January 2011 (UTC)
replies: text: I agree, this fix was already done for UTF-8/16 pages.Spitzak (talk) 03:41, 18 January 2011 (UTC)
thread_title:  Character vs. code point 

message: :However there are few (if any) useful algorithms that examine the n'th code point without first examining the preceding n-1 code points, therefore a somewhat more difficult code replacement will almost always negate this advantage
This is bull. I'm a senior software developer, and I can't even count how often I need to examine the last few characters at the *end* of a string (examples: Check if a path is a directory or a file; check a file's extension)
EDIT: Seeing how others have already had almost exactly the same complaint, I'll now remove / rephrase the mentioned section.
:You can pattern-match the end of a string by going *backwards* from the end, which does not require fixed-sized code units. The current text is correct, do not change it.Spitzak (talk) 00:34, 30 September 2016 (UTC)
replies: text: All you are saying is that an offset (such as the length) should be in code units, not "characters". You can find the end of a UTF8 or 16 string instantly if the length is in code units, negating any advantage of UTF32. Anyway there is more text about this below, so this is ok. You should be warned however that thinking offsets must be in "characters" does not match claiming you are some kind of expert.Spitzak (talk) 15:50, 23 April 2012 (UTC)
thread_title:  Checking the end of a string not useful?? 

message: There are two citation needed tags for statements saying something is rare to non-existent. In both cases, I don't see how anyone would find a reference. For one, you need a definition of rare to actually know, but also you don't know how many documents people have written and stored on their own computers. Unless someone decides to do a random survey of all documents, it isn't likely we will ever know. I think the tags should be removed. Gah4 (talk) 21:25, 5 October 2015 (UTC)
thread_title:  Citation needed 

message: Numbers above 0x0010FFFF can be stored in UTF-32. In fact, numbers above 0x7FFFFFFF can be stored in UTF-32. If you click this link, you can enter a number 0x00000000 to 0xFFFFFFFF, despite there only being 1114112 unicode characters. 108.66.232.44 (talk) 01:13, 21 September 2016 (UTC)
replies: text: No, this seems to be the misunderstanding that leads to the above edit war. The 32-bit storage locations can store numbers up to 0xFFFFFFFF. However UTF-32 is defined as only using the numbers up to 0x10FFFF. If there is a larger numbers stored there then it is not valid UTF-32. The fact that it is physically possible for those numbers to appear is important. In fact the ignorance of the fact that there can be invalid UTF-8 and that this must be handled is probably the biggest current impediment to the adoption of Unicode. Similar rules of being prepared for invalid UTF-32 also apply. But the result is *not* UTF-32, just like invalid UTF-8 is not really UTF-8.Spitzak (talk) 17:35, 21 September 2016 (UTC)
replies: text: No, those characters got converted to a string of replacement characters. This just means that they are unassigned. 108.71.121.170 (talk) 21:48, 24 September 2016 (UTC)
replies: text: No, that means that the values you entered are illegal in UTF-32. If they were just unassigned they would not have been converted to replacement characters. Anyway, as everyone has repeatedly told you, values greater than U+10FFFF and surrogate ranges are not valid UTF-32 values, so can you just stop this pointless argument! BabelStone (talk) 21:59, 24 September 2016 (UTC)
replies: text: No, that means that those values are undefined. The "names" of those characters are "Undefined" followed by the code point! 108.71.121.170 (talk) 22:15, 24 September 2016 (UTC)
replies: text:  Also, U+ notation violates MOS:HEX, which specifically recomments using the 0x notation. 108.71.120.43 (talk) 01:23, 9 October 2016 (UTC)
replies: text:  It does not "violate" MOS:HEX, which is a guideline.  MOS clearly indicates that some situations warrant deviation from the guidelines.  In this case, Unicode code points have their own standard notation, which is widespread and recognized as a standard format.  A similar case would arise, say, if you were talking about hex numbers in the context of a computer language that does not use the 0x notation. -- Elphion (talk) 13:27, 9 October 2016 (UTC)
thread_title:  Maximum character? 

message: There are private use areas 0x00E00000 to 0x00FFFFFF and 0x60000000 to 0x7FFFFFFF. 108.66.232.212 (talk) 23:18, 1 October 2016 (UTC)
replies: text: These are no longer part of Unicode, which now stops at U+10FFFF. -- Elphion (talk) 23:50, 2 October 2016 (UTC)
replies: text:  They are still in the Universal Character Set. 108.66.234.86 (talk) 12:43, 3 October 2016 (UTC)
replies: text:  Which no one uses -- UCS-4 is obsolete. -- Elphion (talk) 13:01, 3 October 2016 (UTC)
replies: text:  Yes, but Unicode still uses them. 108.66.234.86 (talk) 14:56, 3 October 2016 (UTC)
replies: text:  There are plenty of Wikipedia articles on things that are obsolete, and that no-one uses. That doesn't mean we ignore the differences. It might mean that there should be a History section explaining the differences, and why they aren't used. Gah4 (talk) 15:50, 3 October 2016 (UTC)
thread_title:  Private Use Planes/Groups 

message: Why is there no UTF-21? It would go 0x000000 to 0x1FFFFF, more than enough to cover Unicode's limit of 0x0010FFFF. 108.71.120.246 (talk) 16:05, 2 October 2016 (UTC)
replies: text: There's no point, since Unicode stops at U+10FFFF. -- Elphion (talk) 23:51, 2 October 2016 (UTC)

text: To amplify:  How would "UTF-21" differ from UTF-32?  Any encoding would require at least 3 words bytes, and 3 words bytes is not as convenient as 4, which is already standard and used routinely on some platforms.  Where's the advantage?  UTF-32 works fine if you're not concerned about space; UTF-8 and UTF-16 work if you are. -- Elphion (talk) 00:00, 3 October 2016 (UTC)
replies: text:  UTF-32 has 4,294,967,296 possible characters (because it is a 32 bit encoding), and UTF-21 would have 2,097,152 possible characters. (because it would be a 21 bit encoding). 108.66.234.86 (talk) 12:49, 3 October 2016 (UTC)
replies: text: No it doesn't, UTF-32 could have 2^32 or 2^31 characters if definded as such, but it isn't; what you do outside of the standard doesn't matter. comp.arch (talk) 12:03, 5 October 2016 (UTC)
replies: text: Why would UTF-32 have 2^31 characters? 108.71.123.25 (talk) 14:17, 5 October 2016 (UTC)
replies: text: See "65,536 plane restriction" below -- Elphion (talk) 15:09, 5 October 2016 (UTC)
thread_title:  Why not UTF-21? 

message: If UCS-4 is a 31 bit encoding, then why is UCS-2 a 16 bit encoding? 108.71.120.246 (talk) 16:05, 2 October 2016 (UTC)
replies: text: Because it was clear from the beginning that 15 bits would not be sufficient.  (16 bits aren't either, but that wasn't clear from the beginning.) -- Elphion (talk) 23:53, 2 October 2016 (UTC)
replies: text:  But why is UCS-4 a 31 bit encoding? (because 4 * 8 ≠ 31). 108.66.234.86 (talk) 12:50, 3 October 2016 (UTC)
replies: text: I don't know, but my guess is that the design sought to avoid problems comparing signed integers (just as people still trip over comparing extended ASCII values as chars, which are signed, often leading to unintended results). -- Elphion (talk) 12:59, 3 October 2016 (UTC)
thread_title:  UCS-2 

message: This is a probably futile attempt to clarify the situation regarding the 17-plane restriction of the Unicode and ISO/IEC 10646 standards, for the benefit of the IP editor who insists that this restriction does not exit.
ISO/IEC 10646 ("Universal Coded Character Set") was not originally restricted to seventeen planes, whereas the Unicode Standard was. This lack of synchronization between the two standards was problematic, and so in 2005 the US national body (essentially representing the Unicode Consortium) requested ISO/IEC JTC1/SC2/WG2 to change the wording of ISO/IEC 10646 to limit the code space to 17 planes (see WG2 N2920). This was discussed at the SC2/WG2 meeting held in Xiamen in January 2005 (which I attended, incidentally), and the US proposal was accepted, with Japan abstaining (see WG2 M46 Minutes pp. 50–51 and Resolution M46.12 (17-plane restriction). The relevent changes were made in ISO/IEC 10646:2003 Amendment 2, which was published in 2006, and since that time ISO/IEC 10646 has been limited to 17 planes, and only code points in the range 0 through 10FFFF excluding D800–DF00 have been valid (see ISO/IEC 10646:2014 clauses 4.57, 4.58, 9.4 and elsewhere). Therefore, prior to 2006 UCS-4 was not equivalent to UTF-32, but since 2006 UCS-4 is identical to UTF-32 as it is restricted to the code space defined in ISO/IEC 10646. We should mention this historical discrepancy in the article, but there is no need to give it undue weight. BabelStone (talk) 16:09, 3 October 2016 (UTC)
replies: text: Unicode is a 32 bit code space. 108.71.121.129 (talk) 16:58, 3 October 2016 (UTC)
replies: text: See "65,536 plane restriction" below -- Elphion (talk) 15:09, 5 October 2016 (UTC)
thread_title:  17-plane restriction 

message: The reason why Unicode has 65536 planes, is because some operating systems use the extra 11 bits for a secondary purpose. 108.66.233.160 (talk) 17:47, 8 February 2017 (UTC)
replies: text: Not sure what you are saying, but Unicode has 17 planes. The reason is due to the limitations of UTF-16 which cannot encode any values higher than the top of the 17th plane (as well as having an unfortunate inability to encode a block of intermediate values, the "surrogate halves"). UTF-8 was designed to support 32768 planes, and (though it is a bit vague) UCS-4 appeared to be designed for 32768 planes (they wanted to avoid negative signed 32-bit numbers). The fact that limiting to 17 planes means there are 11 bits that are always zero in UTF-32 is not the reason, but certainly a lot of software has taken advantage of this to stuff extra information (such as font selection) into these bits.Spitzak (talk) 18:28, 8 February 2017 (UTC)
replies: text:  That secondary purpose is, for example, bold, italic, and narrow forms of letters. 108.65.82.146 (talk) 21:19, 11 February 2017 (UTC)
replies: text:  Also, why was UTF-16 the base standard for Unicode? 108.65.82.146 (talk) 21:31, 11 February 2017 (UTC)
replies: text:  They also forgot that some languages have characters without Unicode. 108.65.82.146 (talk) 16:59, 13 February 2017 (UTC)
thread_title:  Just to be clear... 

message: I know it's been said before, but UTF-32/UCS-4 actually does have those huge private use areas. In fact, I have a flip phone with them. 108.65.81.240 (talk) 01:05, 6 October 2016 (UTC)
thread_title: UTF-32 or UCS-4?

message: Which notation should be used in this page? 108.65.81.159 (talk) 16:41, 13 October 2016 (UTC)
replies: text: To begin with, it should be left at 0 until there is consensus to change it to anything else.  Stop edit warring.  If you read the section just above there is already a discussion going and there is no consensus at this point to change anything.  -- GB fan 16:43, 13 October 2016 (UTC)
replies: text: Why 0? 108.65.81.159 (talk) 16:49, 13 October 2016 (UTC)
replies: text: It should be left at 0 until consensus has been formed to change it because that was the status quo before you started changing it.  When there is a disagreement the article is left as it was until there is a consensus to change.  It isn't changed and then get a consensus to change it back.  -- GB fan 17:46, 13 October 2016 (UTC)
replies: text: Why should it be left at 0? 108.65.83.206 (talk) 21:23, 15 October 2016 (UTC)
replies: text:  Answered right above. Please read. -- GB fan 21:40, 15 October 2016 (UTC)

text: I prefer the 0x00000000 form, as it shows that the values are 32 bits, that being an important part of UTF-32.  In the case of a range, I prefer both sides of a range to have the same width, even if it isn't the width of the value in question. Then again, I also do VHDL and Verilog programming, where it is more usual to pad out values, as they match up with actual wires. I don't prefer it enough to start an edit war over it, though.  I asked about range width on WT:MOS, but there is no interest there. Gah4 (talk) 16:58, 13 October 2016 (UTC)
replies: text: Seems to be weeks now, and pretty much all the arguments are from me.  Does anyone else want to say anything about this? Gah4 (talk) 19:22, 8 November 2016 (UTC)
replies: text:  0x00000000 is more important, as it shows that it is a 32 bit number. (4×8=32) 108.65.80.10 (talk) 13:30, 14 November 2016 (UTC)

text: Everyone said above what they wanted, but mostly not why.  It seems that I was wrong about the U+ notation: from The Unicode Standard Appendix A, that U+xxxx is for four to six digit hex values, and U-xxxxxxxx for eight digit values. (For the U+ cases, one must use at least four hex digits, even with leading zeros.) It also seems that code points are represented as \uxxxxx and \Uxxxxxxxx, while the U+ and  U- forms represent character literals, with the latter for 32 bit values.  In the UTF-32 case, the character encoding and binary value are the same, except that endianness isn't specified. Once the article has explained that the binary code and the character encoding are the same, (except for endianness) then the use of U- values is not inconsistent, at least for parts of the article that are representing ranges of code points, even if no character has been assigned to that code point yet. If it is good enough for Unicode, Inc., it should be good enough for us. We could even ask Unicode Inc. Gah4 (talk) 02:39, 15 November 2016 (UTC)
replies: text: The Extended BNF notation given in Table A-1 allows for "U+HHHH" and "U-HHHHHHHH" to represent 16 bit and 32 bit values respectively. I don't know exactly where in the Unicode Standard this notational syntax is used, but it is not the same as the U+n notation for Unicode code points described in Appendix A under the heading "Code Points". The U+n notation allows for four to six hexadecimal digits, and is applicable to all code points in the Unicode code space. On the other hand, the U+HHHH and U-HHHHHHHH syntax in Table A-1 allows for exactly four hexadecimal digits and exactly eight decimal digits respectively. I think that the U+HHHH and U-HHHHHHHH syntax must be used is one or more of the standard annexes to the Unicode Standard where there is a need to distinguish between 16 bit and 32 bit values, but it is not used for general discussion of Unicode code points, and we should stick with the standard U+n (4-6 hex digits) notation when referring to particular Unicode code points, and use the standard 0xhhhhhhhh notation when talking about 32 bit values. BabelStone (talk) 09:48, 15 November 2016 (UTC)
replies: text: The U+xxxx shows for hex digits, but the text says 4 to 6. The first two, 0x0 and 0x7fffffff, are more obviously binary values.  (Unicode seems to like to represent them as bit strings, 0101010102.)  The rest are either actual code points, or actual non-code points, both of which Unicode.org represents with U+ or U- notation.)  The first two could be described as all non-negative 32 bit binary integers, and avoid numerical values. (Just wondering, why not all unsigned 32 bit values?) Gah4 (talk) 10:17, 15 November 2016 (UTC)
replies: text:  Is there a special exception for 0? 108.66.234.235 (talk) 17:09, 25 November 2016 (UTC)
replies: text: That is always the question.  Should 0 have units?  Since 0cm = 0m = 0in, you don't need a (length) unit on 0, but usually you put it in, anyway. This came up many years ago, as TeX requires a unit on lengths. Seems to me that zero shouldn't be special, but sometimes it is. Gah4 (talk) 21:25, 25 November 2016 (UTC)
thread_title: 0 or 0x0000 or 0x000000 or 0x00000000?

message: The statement “This removes any speed advantage of UTF-32” has been flagged as needing a reference for three months now. But even worse, it doesn't make sense, at least not to me. Unless someone can at least explain what it means, we should therefore just remove it. ◅ Sebastian 04:43, 11 August 2023 (UTC)
replies: text: It means an integer offset measured in code units into a variable-width encoding such as UTF-8 is just as fast as an integer offset measured in code points into a UTF-32 string. There are numerous attempts to explain this in the preceeding sentences. Spitzak (talk) 14:32, 21 August 2023 (UTC)
thread_title:  “This removes any speed advantage of UTF-32”