Source: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0482r0.html
Timestamp: 2019-04-25 22:50:02+00:00

Document:
Should UTF-8 literals continue to be referred to as narrow literals?
What should be the underlying type of char8_t?
C++11 introduced support for UTF-8, UTF-16, and UTF-32 encoded string literals via N2249 [N2249]. New char16_t and char32_t types were added to hold values of code units for the UTF-16 and UTF-32 variants, but a new type was not added for the UTF-8 variants. Instead, UTF-8 character literals (added in C++17 via N4197 [N4197]) and string literals were defined in terms of the char type used for the code unit type of ordinary character and string literals. UTF-8 is the only text encoding mandated to be supported by the C++ standard for which there is no distinct code unit type. Lack of a distinct type for UTF-8 encoded character and string literals prevents the use of overloading and template specialization in interfaces designed for interoperability with encoded text. The inability to infer an encoding for narrow characters and strings limits design possibilities and hinders the production of elegant interfaces that work seemlessly in generic code. Library authors must choose to limit encoding support, design interfaces that require users to explicitly specify encodings, or provide distinct interfaces for, at least, the implementation defined execution and UTF-8 encodings.
Whether char is a signed or unsigned type is implementation defined and implementations that use an 8-bit signed char are at a disadvantage with respect to working with UTF-8 encoded text due to the necessity of having to rely on conversions to unsigned types in order to correctly process leading and continuation code units of multi-byte encoded code points.
The lack of a distinct type and the use of a code unit type with a range that does not portably include the full unsigned range of UTF-8 code units presents challenges for working with UTF-8 encoded text that are not present when working with UTF-16 or UTF-32 encoded text. Enclosed is a proposal for a new char8_t fundamental type and related library enhancements intended to remove barriers to working with UTF-8 encoded text and to enable generic interfaces that work with all five of the standard mandated text encodings in a consistent manner.
This proposal is incomplete as the author ran out of time preparing it for the Issaquah mailing deadline. The following are known deficiencies that are expected to be addressed in a future revision of this proposal.
Backward compatibility is not adequately addressed. There is some discussion in the design considerations section, but no provisions addressing backward compatibility are currently present in the wording. The proposed changes effectively bring the standard to the state the author feels it would likely be in had char8_t been added at the same time as char16_t and char32_t were.
An implementation of the proposed changes is not yet available for assessing the impact to backward compatibility.
The claim that a new type may allow compilers to better optimize code that works with UTF-8 strings is unsubstantiated.
Wording updates for clauses C and D have not yet been provided.
Impact to other proposals such as P0353R0 [P0353R0] is not discussed.
"\u0123" // ???:    const char:     ???
L"\u0123" // ???:    const wchar_t:  ???
make_text_view("text")   // defaults to execution_character_encoding.
make_text_view(L"text")  // defaults to execution_wide_character_encoding.
make_text_view(u8"text") // defaults to utf8_encoding.
make_text_view(u"text")  // defaults to utf16_encoding.
make_text_view(U"text")  // defaults to utf32_encoding.
The inability to infer an encoding for narrow strings doesn't just limit the interfaces of new features under consideration. Compromised interfaces are already present in the standard library.
Consider the design of the codecvt class template. The standard specifies the following specializations of codecvt be provided to enable transcoding text from one encoding to another.
#1 performs no conversions. #2 converts between strings encoded in the implementation defined wide and narrow encodings. #3 and #4 convert between either the UTF-16 or UTF-32 encoding and the UTF-8 encoding. Specializations are not currently specified for conversion between the implementation defined narrow and wide encodings and any of the UTF-8, UTF-16, or UTF-32 encodings. However, if support for such conversions were to be added, the desired interfaces are already taken by #1, #3 and #4.
The file system interface adopted for C++17 via P0218R1 [P0218R1] provides an example of a feature that supports all five of the standard mandated encodings, but does so with an asymetric interface due to the inability to overload functions for UTF-8 encoded strings. Class std::filesystem::path provides the following constructors to initialize a path object based on a range of code unit values where the encoding is inferred based on the value type of the range.
§ 27.10.8.2.2 [path.type.cvt] describes how the source encoding is determined based on whether the source range value type is char, wchar_t, char16_t, or char32_t. A range with value type char is interpreted using the implementation defined narrow execution encoding. It is not possible to construct a path object from UTF-8 encoded text using these constructors.
To accommodate UTF-8 encoded text, the file system library specifies the following factory functions. Matching factory functions are not provided for other encodings.
The requirement to construct path objects using one interface for UTF-8 strings vs another interface for all other supported encodings creates unnecessary difficulties for portable code. Consider an application that uses UTF-8 as its internal encoding on POSIX systems, but uses UTF-16 on Windows. Conditional compilation or other abstractions must be implemented and used in otherwise platform neutral code to construct path objects.
The inability to infer an encoding based on string type is not the only challenge posed by use of char as the UTF-8 code unit type. The following code exhibits implementation defined behavior.
UTF-8 leading and continuation code units have values in the range 128 (0x80) to 255 (0xFF). In the common case where char is implemented as a signed 8-bit type with a two's complement representation and a range of -128 (-0x80) to 127 (0x7F), these values exceed the unsigned range of the char type. Such implementations typically encode such code units as unsigned values which are then reinterpreted as signed values when read. In the code above, integral promotion rules result in c being promoted to type int for comparison to the 0x80 operand. if c holds a value corresponding to a leading or continuation code unit value, then its value will be interpreted as negative and the promoted value of type int will likewise be negative. The result is that the comparison is always false for these implementations.
Finally, processing of UTF-8 strings is currently subject to an optimization pessimization due to glvalue expressions of type char potentially aliasing objects of other types. Use of a distinct type that does not share this aliasing behavior may allow for further compiler optimizations.
This proposal does not specify any backward compatibility features other than to retain interfaces that it deprecates. The lack of such features is not due to a belief that backward compatibility features are not necessary. The author believes such features are necessary, but time constraints prevented adequately researching what issues must be addressed, to what degree they must be addressed, and how those features should be specified. The author intends to address these concerns in a future revision of this document. In the meantime, the following sections discuss some of the backward compatibility impact and possible solution directions.
f(u8"text");                    // Ok, calls f(const char*).
char u8a = u8"text";          // Ok.
const char (&u8r) = u8"text"; // Ok.
const char *u8s = u8"text";     // Ok.
const auto *u8s = u8"text"; // C++14: Ok, type deduced to const char*.
// This proposal: Ok, type deduced to const char8_t*.
const char *s = u8s;        // C++14: Ok, u8s has type const char*.
If such implicit conversions are found to be necessary, specifying them may present a small challenge. The standard conversion sequence might have to be modified to allow a data representation conversion prior to an lvalue transformation in order for an argument of, for example, array of char8_t to match a parameter of type char*. However, the standard conversion sequence, as described in § 13.3.3.1.1 [over.ics.scs], states that lvalue transformations, including the array-to-pointer conversion, are performed before promotions and conversions that might change the data representation. It may be feasible to avoid such a change by stating that a candidate function that involves such an implicit conversion is only a viable function if no other viable non-template functions are identified, but the author has not yet convinced himself of this possibility.
If such implicit conversions are found to be necessary, providing them as deprecated features would enable a transition period and eventual removal.
This proposal includes a new specialization of std::basic_string for the new char8_t type, the associated typedef std::u8string, and changes to several functions to now return std::u8string instead of std::string. This change renders ill-formed the following code that is currently well-formed.
std::string s = p.u8string(); // C++14: Ok.
Implicit conversions from std::u8string to std::string would be undesirable in general. If they are found to be necessary, providing them as a deprecated feature seems warranted.
Under this proposal, UTF-8 string and character literals have type const char8_t and char8_t respectively. This affects the types deduced for placeholder types and template parameter types.
ft(u8"text", u8'c'); // C++14: T1 deduced to const char*, T2 deduced to char.
// This proposal: T1 deduced to const char8_t*, T2 deduced to char8_t.
auto u8s = u8"text"; // C++14: Type deduced to const char*.
// This proposal: Type deduced to const char8_t*.
auto u8c = u8'c';    // C++14: Type deduced to char.
// This proposal: Type deduced to char8_t.
ft(u8"text"); // C++14: Returns 1.
// This proposal: Returns 0.
UTF-8 literals are maintained as narrow literals in this proposal.
There are several choices for the underlying type of char8_t. Use of unsigned char closely aligns with historical use. Use of uint_least8_t would maintain consistency with how the underlying types of char16_t and char32_t are specified.
This proposal specifies unsigned char as the underlying type as noted in the changes to § 3.9.1 [basic.fundamental] paragraph 5.
This proposal introduces new codecvt and codecvt_byname specializations that use char8_t for conversion to and from UTF-8 and deprecates the existing ones specified in terms of char. The new specializations are functionally identical to the deprecated ones.
Filesystem path objects may now be constructed with UTF-8 strings using the existing path constructors used for construction with other encodings as specified in § 27.10.8.2.2 [path.type.cvt] and § 27.10.8.4.1 [path.construct]. This proposal deprecates the existing u8path path factory functions specified in § 27.10.8.6.2 [path.factory].
None yet, but the author intends to prototype an implementation in gcc/libstdc++ and/or Clang/libc++.
Add char8_t to the list of keywords in table 3 in 2.11 [lex.key] paragraph 1.
Ordinary string literals and UTF-8 string literals are also referred to as narrow string literals. A narrow string literal has type “array of n const char”, where n is the size of the string as defined below, and has static storage duration (3.7).
An ordinary string literal has type "array of n const char", where n is the size of the string as defined below, and has static storage duration (3.7).
For a UTF-8 string literal, each successive element of the object representation (3.9) has the value of the corresponding code unit of the UTF-8 encoding of the string. A string-literal that begins with u8, such as u8"asdf", is a UTF-8 string literal, also referred to as a char8_t string literal. A char8_t string literal has type "array of n const char8_t", where n is the size of the string as defined below; each successive element of the object representation (3.9) has the value of the corresponding code unit of the UTF-8 encoding of the s-char-sequence. A single s-char may produce more than one char8_t code unit.
Objects declared as characterswith type (char) shall be large enough to store any member of the implementation’s basic character set. If a character from this set is stored in a character object, the integral value of that character object is equal to the value of the single character literal form of that character. It is implementation-defined whether a char object can hold negative values. Characters declared with type char can be explicitly declared unsigned or signed. Plain char, signed char, and unsigned char are three distinct types, collectively called narrowordinary character types. The ordinary character types and char8_t are collectively called narrow character types. A char, a signed char, and an unsigned char, and a char8_t occupy the same amount of storage and have the same alignment requirements (3.11); that is, they have the same object representation. For narrow character types, all bits of the object representation participate in the value representation. [ Note: A bit-field of narrow character type whose length is larger than the number of bits in the object representation of that type has padding bits; see 9.2.4. — end note ] For unsigned narrow character types, including char8_t, each possible bit pattern of the value representation represents a distinct number. These requirements do not hold for other types. In any particular implementation, a plain char object can shall take on either the same values as a signed char or an unsigned char; which one is implementation-defined. For each value i of type unsigned char, or char8_t in the range 0 to 255 inclusive, there exists a value j of type char such that the result of an integral conversion (4.8) from i to char is j, and the result of an integral conversion from j to unsigned char or char8_t is i.
[…] Type wchar_t shall have the same size, signedness, and alignment requirements (3.11) as one of the other integral types, called its underlying type. Type char8_t denotes a distinct type with the same size, signedness, and alignment as unsigned char, called its underlying type. Types char16_t and char32_t denote distinct types with the same size, signedness, and alignment as uint_least16_t and uint_least32_t, respectively, in <cstdint>, called the underlying types.
Types bool, char, char8_t, char16_t, char32_t, wchar_t, and the signed and unsigned integer types are collectively called integral types.
(1.8) — The ranks of char8_t, char16_t, char32_t, and wchar_t shall equal the ranks of their underlying types (3.9.1).
As a consequence, operands of type bool, char8_t, char16_t, char32_t, wchar_t, or an enumerated type are converted to some integral type.
(4.5) — otherwise, decltype(e) is the type of e.
(17.3) — If the destination type is an array of characters, an array of char8_t, an array of char16_t, an array of char32_t, or an array of wchar_t, and the initializer is a string literal, see 8.6.2.
Drafting note: It is intentional that an array of ordinary character type can be initialized by a narrow string literal, including UTF-8 string literals. This is a backward compatibility feature.
The strings library (Clause 21) provides support for manipulating text represented as sequences of type char, sequences of type char8_t, sequences of type char16_t, sequences of type char32_t, sequences of type wchar_t, and sequences of any other character-like type.
This subclause defines requirements on classes representing character traits, and defines a class template char_traits<charT>, along with fourfive specializations, char_traits<char>, char_traits<char8_t>, char_traits<char16_t>, char_traits<char32_t>, and char_traits<wchar_t>, that satisfy those requirements.
This subclause specifies a class template, char_traits<charT>, and fourfive explicit specializations of it, char_traits<char>, char_traits<char8_t>, char_traits<char16_t>, char_traits<char32_t>, and char_traits<wchar_t>, all of which appear in the header <string> and satisfy the requirements below.
Drafting note: 21.2p4 appears to unnecessarily duplicate information previously presented in 21.2p1.
The header <string> shall define fourfive specializations of the class template char_traits: char_traits<char>, char_traits<char8_t>, char_traits<char16_t>, char_traits<char32_t>, and char_traits<wchar_t>.
The type u8streampos shall be an implementation-defined type that satisfies the requirements for pos_type in 27.2.2 and 27.3.
The two-argument members assign, eq, and lt shall be defined identically to the built-in operators =, ==, and < respectively.
The member eof() shall return an implementation-defined constant that cannot appear as a valid UTF-8 code unit.
The header <string> defines the basic_string class template for manipulating varying-length sequences of char-like objects and fourfive typedef-names, string, u8string, u16string, u32string, and wstring, that name the specializations basic_string<char>, basic_string<char8_t>, basic_string<char16_t>, basic_string<char32_t>, and basic_string<wchar_t>, respectively.
The specializations required in Table 65 (22.3.1.1.1) convert the implementation-defined native character set. codecvt<char, char, mbstate_t> implements a degenerate conversion; it does not convert at all. The specializations codecvt<char16_t, char, mbstate_t> (deprecated) and codecvt<char16_t, char8_t, mbstate_t> converts between the UTF-16 and UTF-8 encoding forms, and the specializations codecvt<char32_t, char, mbstate_t> (deprecated) and codecvt<char32_t, char8_t, mbstate_t> converts between the UTF-32 and UTF-8 encoding forms. codecvt<wchar_t,char,mbstate_t> converts between the native character sets for narrowordinary and wide character. Specializations on mbstate_t perform conversion between encodings known to the library implementer. Other encodings can be converted by specializing on a user-defined stateT type. Objects of type stateT can contain any state that is useful to communicate to or from the specialized do_in or do_out members.
For narrowordinary character strings, the operating system dependent current encoding for pathnames (27.10.4.18).
For wide character strings, the implementation defined execution wide-character set encoding (2.3).
Throughout this sub-clause, char, wchar_t, char8_t, char16_t, and char32_t are collectively called encoded character types.
— char8_t: The encoding is UTF-8. The method of conversion method is unspecified.
Remarks: Conversion, if any, is performed as specified by 27.10.8.2. The encoding of the strings returned by u8string(), u16string(), and u32string isare always UTF-8, UTF-16, and UTF-32 respectively.
Returns: pathname, reformatted according to the generic pathname format (27.10.8.1).
Remarks: Conversion, if any, is performed as specified by 27.10.8.2. The encoding of the strings returned by generic_u8string(), generic_u16string(), and generic_u16string isare always UTF-8, UTF-16, and UTF-32 respectively.
Requires: The source and [first, last) sequences are UTF-8 encoded. The value type of Source and InputIterator is char or char8_t.
Drafting note: It is intentional that the deprecated factory functions accept ranges with value types of either char or char8_t. This is a backward compatibility feature.
Drafting note: The u8path factory functions are deprecated.
Michael Spencer and Davide C. C. Italiano first proposed adding a new char8_t fundamental type in P0372R0 [P0372R0].
[N4197] Richard Smith, "Adding u8 character literals", N4197, 2014.
[N4606] "Working Draft, Standard for Programming Language C++", N4606, 2016.
[P0353R0] Beman Dawes, "Unicode Encoding Conversions for the Standard Library", P0353R0, 2016.
[P0372R0] Michael Spencer and Davide C. C. Italiano, "A type for utf-8 data", P0372R0, 2016.
[P0244R1] Tom Honermann, "Text_view: A C++ concepts and range based character encoding and code point enumeration library", P0244R1, 2016.
[P0218R1] Beman Dawes, "Adopt the File System TS for C++17", P0218R1, 2016.

References: § 27
 § 13
 § 3
 § 27
 § 27
 § 27