Repository: benhoyt/goawk
Pull request #236 (2024-08-08 01:27:41)

Make printf %c work on Unicode codepoints
I noticed goawk couldn't handle UTF-8 strings. While there already seems to be ample discussion about this, one of the UTF-8 features that doesn't seem to incur issues about O(n) speed is `printf("%c",[CODEPOINT])` which on the one true awk, Ray Gardner's awk, and gawk, will work with unicode codepoints and utf8 strings:

```
$ for awk in nawk gawk goawk; do $awk 'BEGIN {printf("%c", "ß")}' | xxd; done
00000000: c39f                                     ..
00000000: c39f                                     ..
00000000: c3                                       .
$ for awk in nawk gawk goawk; do $awk 'BEGIN {printf("%c", 400)}' | xxd; done
00000000: c690                                     ..
00000000: c690                                     ..
00000000: 90                                       .
```

This is standard behavior in both the one true awk (since they added unicode support, this feature is mentioned in the second edition awk book) and gawk. Similar to "\u[CODEPOINT]" 


benhoyt: Hi @oliverkwebb -- thanks for the contribution. I didn't want to do this earlier as the GoAWK functions mostly deal in bytes, but I don't think it hurts to pull in the Unicode behaviour for printf's %c. So I think I'll include this. However, I've put up an alternative at https://github.com/benhoyt/goawk/pull/237 that keeps using the `[]byte` and doesn't require decoding the entire string (if it's longer than 1 rune) to `[]rune` first.

I've also fixed/updated the tests, which were breaking, and added a couple more. If you're okay with #237, let's close this one and merge that. Alternatively you could update this branch with those changes if you'd like your name on it.
oliverkwebb: Commit [6b2baea](https://github.com/benhoyt/goawk/pull/236/commits/6b2baea26a3fbe0c0772ea5c09009c739372d887)
oliverkwebb: (P.S. I'm not sure why you don't have write access to my PR branch, I have "allow edits by maintainers" checked and if there's another setting for it, I can't find it)

Patch:
diff --git a/interp/functions.go b/interp/functions.go
index a10f6af..ffe91f6 100644
--- a/interp/functions.go
+++ b/interp/functions.go
@@ -415,15 +415,16 @@ func (p *interp) sprintf(format string, args []value) (string, error) {
 			n, isStr := a.isTrueStr()
 			if isStr {
 				s := p.toString(a)
-				if len(s) > 0 {
-					c = []byte{s[0]}
+				_, size := utf8.DecodeRuneInString(s)
+				if size > 0 {
+					c = []byte(s[:size])
 				} else {
 					c = []byte{0}
 				}
 			} else {
-				// Follow the behaviour of awk and mawk, where %c
-				// operates on bytes (0-255), not Unicode codepoints
-				c = []byte{byte(n)}
+				c = make([]byte, utf8.UTFMax)
+				size := utf8.EncodeRune(c, rune(n))
+				c = c[:size]
 			}
 			v = c
 		}


Test patch:
diff --git a/interp/interp_test.go b/interp/interp_test.go
index 124ca30..2083b90 100644
--- a/interp/interp_test.go
+++ b/interp/interp_test.go
@@ -85,14 +85,14 @@ NR==3, NR==5 { print NR }
 	{`BEGIN { printf "%.1g", 42 }  # !windows-gawk`, "", "4e+01", "", ""}, // for some reason gawk gives "4e+001" on Windows
 	{`BEGIN { printf "%d", 12, 34 }`, "", "12", "", ""},
 	{`BEGIN { printf "%d" }`, "", "", "format error: got 0 args, expected 1", "not enough arg"},
-	// Our %c handling is mostly like awk's, except for multiples
-	// 256, where awk is weird, and we're like mawk
 	{`BEGIN { printf "%c", 0 }`, "", "\x00", "", ""},
 	{`BEGIN { printf "%c", 127 }`, "", "\x7f", "", ""},
-	{`BEGIN { printf "%c", 128 }  # !gawk`, "", "\x80", "", ""},
-	{`BEGIN { printf "%c", 255 }  # !gawk`, "", "\xff", "", ""},
-	{`BEGIN { printf "%c", 256 }  # !awk !gawk`, "", "\x00", "", ""},
+	{`BEGIN { printf "%c", 128 }  # !windows-gawk`, "", "\u0080", "", ""},
+	{`BEGIN { printf "%c", 255 }  # !windows-gawk`, "", "ÿ", "", ""},
+	{`BEGIN { printf "%c", 256 }  # !windows-gawk`, "", "Ā", "", ""},
+	{`BEGIN { printf "%c", 4660 }  # !windows-gawk`, "", "\u1234", "", ""},
 	{`BEGIN { printf "%c", "xyz" }`, "", "x", "", ""},
+	{`BEGIN { printf "%c %c %c", "Ā", "ĀĀĀ", "Āx" }  # !windows-gawk`, "", "Ā Ā Ā", "", ""},
 	{`BEGIN { printf "%c", "" }  # !awk`, "", "\x00", "", ""},
 	{`BEGIN { printf }  # !awk !posix - doesn't error on this`, "", "", "parse error at 1:16: expected printf args, got none", "printf: no arguments"},
 	{`BEGIN { printf("%%%dd", 4) }`, "", "%4d", "", ""},