Unexpected printf behavior - Page 1
Forum section: electronics / general

author: MikeK
author_rank: Super Contributor
timestamp: July 08, 2023, 04:00:32 pm
content: Here's my program:Code:#include <stdio.h>int main(){printf("%6s %3s\n", "\u00B0C", "\u00B0F");printf("%6s %3s\n", "C", "F");printf("%6.1f %3.0f\n", 100.0, 212.0);}The bottom two lines appear right-justified, but the first line not. I'm not understanding why.Code:°C °FC F100.0 212

author: bpiphany
author_rank: Regular Contributor
timestamp: July 08, 2023, 04:37:32 pm
content: Wild guess.. The unicode characters are "2 chars wide"(?) Add an extra space in front of them to trick printf(?)Yes, seems like ithttps://www.google.com/search?q=printf+width+unicode

author: MikeK
author_rank: Super Contributor
timestamp: July 08, 2023, 04:54:49 pm
content: Adding a space does make C and F consistent, but not fully right-justifying them. Shouldn't they get forced all the way over?Code:°C °FC F100.0 212

author: DavidAlfa
author_rank: Super Contributor
timestamp: July 08, 2023, 05:17:59 pm
content: Try using \260 for the degree symbol(Still 0xB0 char but now in octal)Code:printf("%6s %3s\n", "\260C", "\260F");My guessing is printf is counting the input chars, not the effective ones.Two Unicode bytes generate 1 output char, but printf thought it found 2.Just increase the tabs when using Unicode:Code:printf("%7s %4s\n", "\u00b0C", "\u00B0F");

author: Perkele
author_rank: Regular Contributor
timestamp: July 08, 2023, 05:22:54 pm
content: You did not specify which OS is used.Unicode and printf don't mix really well, because printf expects pointer to a simple null-terminated string.If you need Unicode, set your locale to LC_ALL, and whatever Unicode flavour you plan to use.And then use Unicode-aware functions.But as already answered, there is a character that you need in standard 7/8-bit character set.

author: Perkele
author_rank: Regular Contributor
timestamp: July 08, 2023, 05:26:54 pm
content: Quote from: bpiphany on July 08, 2023, 04:37:32 pmDepending of "flavour", Unicode characters width can be from 1 to 4 bytes.On top of that, don't count on tricks working with all operating systems.

author: MikeK
author_rank: Super Contributor
timestamp: July 08, 2023, 05:45:45 pm
content: Quote from: DavidAlfa on July 08, 2023, 05:17:59 pmThe reason I went with unicode is that \260 shows up as the "undefined" inverse question mark character. It does format it correctly, though.(I'm using Linux Mint 21.1 and gcc.)

author: Veteran68
author_rank: Frequent Contributor
timestamp: July 08, 2023, 05:47:08 pm
content: Quote from: Perkele on July 08, 2023, 05:26:54 pmIt can get even more complicated than OS. Different implementations of stdio across compiler implementations, and even versions within the same implementation, can behave differently. If the ANSI/ISO standard doesn't specify an exact behavior, then it becomes undefined yet would still meet the standard. Although one would expect any reasonably recent implementation to be consistent here, given how long printf and Unicode/MBCS have been around.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 08, 2023, 09:21:29 pm
content: I've tried various locales and using wide-char literals instead (L prefix for literals, %ls for the printf format), and it doesn't seem to do what you'd expect either.Not sure there's really any way, at least portable, apart from writing your own formatting function.

author: MikeK
author_rank: Super Contributor
timestamp: July 08, 2023, 09:33:02 pm
content: Thanks for doing that.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 08, 2023, 09:36:17 pm
content: I haven't been able to figure out if this is a "known" limitation of printf formatting ("known" as in: according to the C standard), or if it's a limitation of glibc. Could be interesting to try with an alternative libc (/on a platform that doesn't use glibc.)

author: DiTBho
author_rank: Super Contributor
timestamp: July 08, 2023, 10:53:51 pm
content: for me, "printfs" is all wrong approach and does not even deserve to be implemented.A valid alternative was discussed here on this forum, ... too sad nobody has followed suggestions and replaced printfs with something better.

author: Nominal Animal
author_rank: Super Contributor
timestamp: July 08, 2023, 11:54:39 pm
content: This is the correct implementation that works in all standard C implementations that support wide output and the users' current locale.Code:#include <stdlib.h>#include <stdio.h>#include <wchar.h>#include <locale.h>int main(void){if (!setlocale(LC_ALL, ""))fprintf(stderr, "Warning: Current locale is not supported by the OS!\n");if (fwide(stdout, 1) < 1)fprintf(stderr, "Warning: Standard C library does not support wide character output!\n");wprintf(L"%6ls %3ls\n", L"\u00B0C", L"\u00B0F");wprintf(L"%6ls %3ls\n", L"C", L"F");wprintf(L"%6.1f %3.0f\n", 100.0, 212.0);return 0;}in theC.UTF-8locale (default UTF-8 locale) it outputs°C °FC F100.0 212which I assume was the point. I know Windows needs some kind of internal windows-only call between the two if clauses, to set some standard output stream internal flag, but I forget the details.Note that you can specify the degree symbol as a string macro,L"\u00B0", if you find it annoying to work with Unicode constants.With#define WC_DEGREES L"\u00B0"you can usewprintf(L"%6ls %3ls\n", WC_DEGREES L"C", WC_DEGREES L"F");.Note that it isnotsufficient to use%3sand"\u00B0C"for°C,nor%3lswith printf(), because only wide character strings' Unicode characters can be correctly counted. A wide character print function, a wide character formatting string, and a wide character constant (char if%lc, string if%ls) must be used for correct results.Edited: TheL"..."notation denotes a wide string constant, and must be used. (I forgot them above in the WC_DEGREES suggestion.)

author: MikeK
author_rank: Super Contributor
timestamp: July 09, 2023, 05:30:07 pm
content: Quote from: Nominal Animal on July 08, 2023, 11:54:39 pmThat works. Thanks for posting it. I didn't know about wprintf until now.

author: Nominal Animal
author_rank: Super Contributor
timestamp: July 09, 2023, 06:15:16 pm
content: Not many do, even though the above has been correct standard code since C99.You can find more details in thewprintf(),setlocale(),fwide(),fgetws(), and wcs*() functions' (wcscmp(),wcslen(), etc.) man pages.While the wide character set is its own, it should support the first 64k of Unicode characters usingL"\uHHHH". Not all implementations correctly support the rest viaL"\UHHHHHHHH"(whereHdenotes a hexadecimal digit, with values from 00000000 to 0010FFFF possible in current Unicode versions), only some do. You can use thembstowcs()/mbsrtowcs()/wcstombs()/wcsrtombs()standard C functions to convert between wide character and multibyte strings, or better yet, the POSIXiconv_open()/iconv()/iconv_close()interface: it is built-in in all POSIXy systems including Linux, and available as alibiconvlibrary on all other OSes, and the closest thing to a standard interface for conversion between character sets (including wide character strings) we have.(I've always been a bit of a character set/non-latin alphabet support proponent, explaining why I know the above. Back before the turn of the century, when browsers hadn't yet grown support of theaccept-charsetparameter in web forms and I implemented web backends, I used hidden form fields with a set of prototype values like € and á to detect what character set the browser actually used to encode the text the user supplied, with automagic conversion support for anything between cp850 (Windows Western European) and macRoman and ISO 8859-nand UTF-8, using my own conversion code.)

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 09, 2023, 07:55:08 pm
content: But if you want to deal with UTF-8, you're out luck. Aren't you?

author: ejeffrey
author_rank: Super Contributor
timestamp: July 09, 2023, 09:25:52 pm
content: Quote from: SiliconWizard on July 09, 2023, 07:55:08 pmIf you want to deal with utf-8 in memory you are out of luck. But you can read and write utf-8 files and use wchar_t for the in memory representation. As long as you always use wide IO functions and never try to peek inside or interpret the data you shouldn't have to care about the representation which is unfortunate platform dependent.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 09, 2023, 09:55:36 pm
content: Quote from: ejeffrey on July 09, 2023, 09:25:52 pmYes, as I get it, you need to go via wide char and associated functions - at least for the kind of things the OP is trying to achieve.I personally do not use much of the standard library when dealing with UTF-8 myself, except of course the base functions that don't care about the character representation (strlen, strcopy, ...)

author: Nominal Animal
author_rank: Super Contributor
timestamp: July 09, 2023, 10:25:19 pm
content: I forgot to mention that even my example fails, if the wide string contains combining diacriticals, because the wprintf() interface countscharacters, notglyphs.For example, the stringL"åa"==L"\u0061\u030A\u0061"is counted as three characters, because it is: the middle one isU+030A COMBINING RING ABOVE.Quote from: SiliconWizard on July 09, 2023, 07:55:08 pmWhat do you mean? (I suspect I agree, in that C standard or POSIX haven't kept up, but I'm not sure it is what you refer to.)In POSIXy systems, after thesetlocale(LC_ALL,"")(orsetlocale(LC_CTYPE,"")call if you are only interested in strings and character sets and not the other locale features like date and time formatting), you can usenl_langinfo(CODESET)to obtain an immutable string to the current character set. If it matches"UTF-8", then your multibyte strings (mbs*(), *mbs()) are in UTF-8. Otherwise, you can use iconv_open() with that string to convert to either wide characters or to UTF-8, or vice versa.If you want to know how manycharactersthere are in a multibyte string, just uselen= mbstowcs(NULL,string,0).Unfortunately, there is no multibyte equivalent ofprintf()/wprintf()(would bembprintf(), I guess), so there is no standard C stream interfaces that counted output or input in multibyte characters instead ofchars.Thus, the wide character output functions (wprintf(),fwprintf(),swprintf(),vswprintf()et al) are the only ones we can use in standard C (since C99) or POSIX C to format output correctly. (For terminal applications,ncurseswlibrary also works.)Now, I am also a proponent ofUTF-8 Everywhere, simply because it is the one Unicode encoding that works everywhere without new/additional requirements, and is downwards ASCII-compatible. I personally believe that when using e.g. the GNU standard C library, thesource character setshould always be UTF-8. (IIRC, GCC et al. already useiconvfacilities to translate from the current locale to UTF-8.)I also think it is perfectly reasonable to say that a particular program or utility only works in locales using the UTF-8 character set.In Linux and MacOS X,sizeof (wchar_t) == 4, and ifnl_langinfo(CODESET) == "UTF-8", thenwchar_tis UTF-32 (which is UCS-4 compatible). So, in Linux and MacOS X, it is reasonable to require an UTF-8 -based locale, so that wide characters are UTF-32/UCS-4/Unicode.Which leads to interfaces I implement for myself.Conversion from UTF-8 to any other character set is mostly very straightforward. Each unicode character is 1 to 4 bytes long, and the sets of combining characters –– that occupy the same position as the previous character(s), i.e. do not change the "cursor" position –– is known.Converting combining character sequences to non-Unicode character sets is annoying, because there are so darn many (and the conversion tables require quite a lot of memory, relatively speaking, although they can be and are memory-mapped on architectures with virtual memory), and I'm still struggling whether one needs to do that. I do, however, believe that any printable character plus any following combining characters, should be calculated as a single glyph; and that double-wide characters should be calculated as two glyphs.This lets one do exactly what SiliconWizard implied (I think), and just treat everything as UTF-8, even with auto-conversion to/from whatever character set the user uses. Formatting patterns like "this string using at most 8 cursor positions" would then be correctly counted, even for input combining say Japanese and Finnish non-ASCII characters.However, GUI toolkits.GTK GStrings (gchar *) are usually UTF-8, so that's okay.Qt, however, uses UTF-16 internally.In Linux, paths are nul-terminated opaque byte sequences, where 0 terminates the path, and 47 (/) is path component separator (no file or directory name may contain 47), and may use whatever character set the user wants. Environment variables are opaque byte sequences terminated by 0, with the first 61 (=) being the separator between the name and the value part.This starts reminding a lot of the Python separation betweenstrandbytes, doesn't it? However, because not all inputs are valid UTF-8, I personally like to use illegal Unicode code points U+1FFF00 to U+1FFFFF (which encode to 4-byte sequences; only code points U+000000 to U+10FFFF are actually valid, although there are some holes in there too) to represent raw bytes that do not form a valid UTF-8 code sequence. This ensures I can support all possible input byte sequences, and output them as is, without erroring out or mangling invalid sequences.With UTF-8 locales, that approach has very little to no overhead; with other locales, there is additional overhead on input and output when consuming or producing text, but not when consuming or producing byte sequences. Also, there is no problem in mixing text and byte sequences; no "mode" is needed at all.Yet, this is a new, non-standard interface. With standard C and POSIX C, we have to get by with wide characters and wide strings.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 09, 2023, 10:29:21 pm
content: Yes, you pretty much detailed what I meant.And I must modulate what I quickly said above - about the base C string functions when dealing with UTF-8. I listed strlen() has an example, but obviously it only counts the number of char's, not the number of UTF-8 characters. So even for string length (in terms of characters), I use my own function.

author: Nominal Animal
author_rank: Super Contributor
timestamp: July 09, 2023, 10:46:06 pm
content: The question is, whether one should also use a function that returns the number of cursor positions the UTF-8 string occupies, or not.For example, "x᷒" is two characters but one cursor position wide, but "＠" is just one character, U+FF20, that takes two cursor positions, a so-called full-width form.With such a function, one can align strings properly even in terminals, but it does increase the maintenance burdena lot.It is not difficult to extract the set of combining characters (that take zero additional cursor positions) and the set of full-width forms from the Unicode tables and compress it to a fast and small hash table, but the damned Unicode tables are still evolving, which means it really should be a dynamically linked library that one can just regenerate when needed. But then, to keep the speed (because dynamic library call per code point is slow), all the useful related code should be in that library.And then we have one more additional library to maintain.On the other hand, if we just assume one code point = one cursor position, we can properly align and tabulatemosttext, it just .. fails for combining characters and full-width forms –– but no maintenance needed.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 09, 2023, 10:50:57 pm
content: It sure can be a rabbit hole, depending on what your goal is.Another kind of rabbit hole is capitalizing UTF-8 text.

author: HwAoRrDk
author_rank: Super Contributor
timestamp: July 10, 2023, 10:28:20 am
content: Quote from: SiliconWizard on July 09, 2023, 10:50:57 pmWell, at least the Germans have been trying to help that situation by finally implementing capital Eszett. Unlike those dastardly Turks.