Repository: benhoyt/goawk (go)

Issue:
Reconsider handling of bytes vs characters in length() and similar functions
Per @arnoldrobbins comment https://github.com/rmyorston/busybox-w32/issues/174#issuecomment-599194834, the POSIX spec says that `length` and similar functions should work with characters, not bytes. The [POSIX spec](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html) says:

> The index, length, match, and substr functions should not be confused with similar functions in the ISO C standard; the awk versions deal with characters, while the ISO C standard deals with bytes.

Given that, I wonder why (onetrueawk) `awk`'s `length()` function counts bytes instead of characters, at least by default. Here's output on my macOS Catalina machine:

```
$ echo 絵 | nawk '{print length}'    # onetrueawk/awk 20200228
3
$ echo 絵 | gawk '{print length}'    # gawk 5.0.1
1
$ echo 絵 | mawk '{print length}'    # mawk 1.3.4 2020012
3
$ echo 絵 | goawk '{print length}'    # goawk 1.6.0
3

# for reference:
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
```

So I feel good that GoAWK has the majority on its side :-), but as you point out, the `nawk/mawk/goawk/ behavior does seem wrong according to POSIX. What do you recommend, @arnoldrobbins?

Pull request:
Use char instead of byte indexes for length, substr, index, and match
I've done this as a runtime config setting, Config.Bytes, and the
default is character mode. So this is a backwards-incompatible change,
but I think it's 1) warranted given GoAWK tries to conform to POSIX,
and 2) won't break most scripts, even ones that use non-ASCII, unless
they use constant indexes for substr() on non-ASCII strings.

This definitely affects performance, as some operations that were O(1)
are now O(N) in the length of the string. Here are the results for the
relevant functions (though obviously it'll depend on the string len):

```
name             old time/op  new time/op  delta
BuiltinLength-8   615ns ± 3%  4179ns ± 3%  +579.68%  (p=0.008 n=5+5)
BuiltinSubstr-8   985ns ± 0%  3874ns ± 0%  +293.12%  (p=0.008 n=5+5)
BuiltinIndex-8   1.24µs ± 0%  4.70µs ± 6%  +278.85%  (p=0.008 n=5+5)
BuiltinMatch-8   2.90µs ± 0%  3.30µs ± 3%   +13.70%  (p=0.008 n=5+5)
[Geo mean]       2.35µs       3.98µs        +69.30%
```

Fixes #35

Patch:
diff --git a/benchmarks.txt b/benchmarks.txt
index c9e82dc..d91fe6b 100644
--- a/benchmarks.txt
+++ b/benchmarks.txt
@@ -2,170 +2,170 @@ goos: linux
 goarch: amd64
 pkg: github.com/benhoyt/goawk/interp
 cpu: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
-BenchmarkGlobalVars-8             	 2503768	       470.3 ns/op
-BenchmarkGlobalVars-8             	 2531346	       470.9 ns/op
-BenchmarkGlobalVars-8             	 2538981	       471.6 ns/op
-BenchmarkGlobalVars-8             	 2533771	       470.6 ns/op
-BenchmarkGlobalVars-8             	 2534832	       470.6 ns/op
-BenchmarkLocalVars-8              	 2402841	       498.4 ns/op
-BenchmarkLocalVars-8              	 2394294	       498.2 ns/op
-BenchmarkLocalVars-8              	 2393726	       498.4 ns/op
-BenchmarkLocalVars-8              	 2399044	       498.5 ns/op
-BenchmarkLocalVars-8              	 2397291	       497.7 ns/op
-BenchmarkIncrDecr-8               	 2657637	       447.9 ns/op
-BenchmarkIncrDecr-8               	 2660187	       450.6 ns/op
-BenchmarkIncrDecr-8               	 2659303	       450.2 ns/op
-BenchmarkIncrDecr-8               	 2663043	       449.4 ns/op
-BenchmarkIncrDecr-8               	 2662501	       448.3 ns/op
-BenchmarkSimpleBuiltins-8         	 1000000	      1027 ns/op
+BenchmarkGlobalVars-8             	 2406184	       487.1 ns/op
+BenchmarkGlobalVars-8             	 2460460	       486.5 ns/op
+BenchmarkGlobalVars-8             	 2463079	       482.4 ns/op
+BenchmarkGlobalVars-8             	 2477720	       481.3 ns/op
+BenchmarkGlobalVars-8             	 2493673	       480.3 ns/op
+BenchmarkLocalVars-8              	 2316904	       513.6 ns/op
+BenchmarkLocalVars-8              	 2329515	       513.7 ns/op
+BenchmarkLocalVars-8              	 2328044	       514.6 ns/op
+BenchmarkLocalVars-8              	 2299606	       514.2 ns/op
+BenchmarkLocalVars-8              	 2322249	       515.5 ns/op
+BenchmarkIncrDecr-8               	 2627095	       457.7 ns/op
+BenchmarkIncrDecr-8               	 2636932	       452.1 ns/op
+BenchmarkIncrDecr-8               	 2607880	       457.3 ns/op
+BenchmarkIncrDecr-8               	 2623930	       450.3 ns/op
+BenchmarkIncrDecr-8               	 2647590	       459.3 ns/op
 BenchmarkSimpleBuiltins-8         	 1000000	      1014 ns/op
+BenchmarkSimpleBuiltins-8         	 1000000	      1019 ns/op
+BenchmarkSimpleBuiltins-8         	 1000000	      1019 ns/op
 BenchmarkSimpleBuiltins-8         	 1000000	      1014 ns/op
-BenchmarkSimpleBuiltins-8         	 1000000	      1026 ns/op
-BenchmarkSimpleBuiltins-8         	 1000000	      1016 ns/op
-BenchmarkBuiltinMatch-8           	  384088	      2914 ns/op
-BenchmarkBuiltinMatch-8           	  406046	      2897 ns/op
-BenchmarkBuiltinMatch-8           	  366280	      2897 ns/op
-BenchmarkBuiltinMatch-8           	  346806	      2894 ns/op
-BenchmarkBuiltinMatch-8           	  390363	      2903 ns/op
-BenchmarkBuiltinLength-8          	 1878363	       613.5 ns/op
-BenchmarkBuiltinLength-8          	 1958827	       633.9 ns/op
-BenchmarkBuiltinLength-8          	 1948826	       609.0 ns/op
-BenchmarkBuiltinLength-8          	 1955524	       608.6 ns/op
-BenchmarkBuiltinLength-8          	 1968524	       609.4 ns/op
-BenchmarkBuiltinIndex-8           	  921309	      1238 ns/op
-BenchmarkBuiltinIndex-8           	  900532	      1244 ns/op
-BenchmarkBuiltinIndex-8           	  938265	      1246 ns/op
-BenchmarkBuiltinIndex-8           	  929520	      1238 ns/op
-BenchmarkBuiltinIndex-8           	  897415	      1238 ns/op
-BenchmarkBuiltinSubstr-8          	 1211053	       985.0 ns/op
-BenchmarkBuiltinSubstr-8          	 1212769	       986.9 ns/op
-BenchmarkBuiltinSubstr-8          	 1214893	       987.0 ns/op
-BenchmarkBuiltinSubstr-8          	 1216012	       984.3 ns/op
-BenchmarkBuiltinSubstr-8          	 1216729	       984.0 ns/op
-BenchmarkBuiltinSplitSpace-8      	   34539	     34041 ns/op
-BenchmarkBuiltinSplitSpace-8      	   34416	     34048 ns/op
-BenchmarkBuiltinSplitSpace-8      	   34674	     34039 ns/op
-BenchmarkBuiltinSplitSpace-8      	   34377	     34170 ns/op
-BenchmarkBuiltinSplitSpace-8      	   34402	     33953 ns/op
-BenchmarkBuiltinSplitRegex-8      	   13962	     85301 ns/op
-BenchmarkBuiltinSplitRegex-8      	   13940	     85778 ns/op
-BenchmarkBuiltinSplitRegex-8      	   13897	     85422 ns/op
-BenchmarkBuiltinSplitRegex-8      	   13945	     85582 ns/op
-BenchmarkBuiltinSplitRegex-8      	   13924	     85514 ns/op
-BenchmarkBuiltinSub-8             	   86475	     13370 ns/op
-BenchmarkBuiltinSub-8             	   86738	     13448 ns/op
-BenchmarkBuiltinSub-8             	   86506	     13399 ns/op
-BenchmarkBuiltinSub-8             	   86326	     13435 ns/op
-BenchmarkBuiltinSub-8             	   86161	     13403 ns/op
-BenchmarkBuiltinSubAmpersand-8    	   86769	     13398 ns/op
-BenchmarkBuiltinSubAmpersand-8    	   85935	     13371 ns/op
-BenchmarkBuiltinSubAmpersand-8    	   86556	     13407 ns/op
-BenchmarkBuiltinSubAmpersand-8    	   86776	     13401 ns/op
-BenchmarkBuiltinSubAmpersand-8    	   85720	     13381 ns/op
-BenchmarkBuiltinGsub-8            	   73568	     15900 ns/op
-BenchmarkBuiltinGsub-8            	   72930	     15892 ns/op
-BenchmarkBuiltinGsub-8            	   73154	     15863 ns/op
-BenchmarkBuiltinGsub-8            	   72122	     15871 ns/op
-BenchmarkBuiltinGsub-8            	   72841	     15847 ns/op
-BenchmarkBuiltinGsubAmpersand-8   	   72757	     15918 ns/op
-BenchmarkBuiltinGsubAmpersand-8   	   72279	     15893 ns/op
-BenchmarkBuiltinGsubAmpersand-8   	   72517	     15896 ns/op
-BenchmarkBuiltinGsubAmpersand-8   	   73027	     15870 ns/op
-BenchmarkBuiltinGsubAmpersand-8   	   73020	     15863 ns/op
-BenchmarkBuiltinSprintf-8         	  119961	      9553 ns/op
-BenchmarkBuiltinSprintf-8         	  120421	      9559 ns/op
-BenchmarkBuiltinSprintf-8         	  119349	      9542 ns/op
-BenchmarkBuiltinSprintf-8         	  120516	      9551 ns/op
-BenchmarkBuiltinSprintf-8         	  120670	      9578 ns/op
-BenchmarkRecursiveFunc-8          	   76262	     15425 ns/op
-BenchmarkRecursiveFunc-8          	   75009	     15407 ns/op
-BenchmarkRecursiveFunc-8          	   75832	     15397 ns/op
-BenchmarkRecursiveFunc-8          	   75458	     15461 ns/op
-BenchmarkRecursiveFunc-8          	   75937	     15404 ns/op
-BenchmarkFuncCall-8               	  411193	      2754 ns/op
-BenchmarkFuncCall-8               	  402937	      2757 ns/op
-BenchmarkFuncCall-8               	  410762	      2758 ns/op
-BenchmarkFuncCall-8               	  392224	      2762 ns/op
-BenchmarkFuncCall-8               	  400747	      2752 ns/op
-BenchmarkNativeFunc-8             	  108250	     10621 ns/op
-BenchmarkNativeFunc-8             	  108920	     10660 ns/op
-BenchmarkNativeFunc-8             	  109059	     10651 ns/op
-BenchmarkNativeFunc-8             	  108302	     10624 ns/op
-BenchmarkNativeFunc-8             	  107480	     10641 ns/op
-BenchmarkForLoop-8                	  237738	      4703 ns/op
-BenchmarkForLoop-8                	  249146	      4675 ns/op
-BenchmarkForLoop-8                	  245126	      4676 ns/op
-BenchmarkForLoop-8                	  249796	      4685 ns/op
-BenchmarkForLoop-8                	  248128	      4684 ns/op
-BenchmarkForInLoop-8              	  574825	      1949 ns/op
-BenchmarkForInLoop-8              	  603043	      1918 ns/op
-BenchmarkForInLoop-8              	  598104	      1893 ns/op
-BenchmarkForInLoop-8              	  591661	      1894 ns/op
-BenchmarkForInLoop-8              	  599649	      1872 ns/op
-BenchmarkIfStatement-8            	 5286699	       223.7 ns/op
-BenchmarkIfStatement-8            	 5337613	       223.9 ns/op
-BenchmarkIfStatement-8            	 5335720	       223.9 ns/op
-BenchmarkIfStatement-8            	 5296618	       223.9 ns/op
-BenchmarkIfStatement-8            	 5248736	       223.7 ns/op
-BenchmarkCondExpr-8               	 5775140	       206.2 ns/op
-BenchmarkCondExpr-8               	 5790969	       206.4 ns/op
-BenchmarkCondExpr-8               	 5687274	       206.2 ns/op
-BenchmarkCondExpr-8               	 5799328	       206.5 ns/op
-BenchmarkCondExpr-8               	 5749263	       206.4 ns/op
-BenchmarkSimplePattern-8          	 9373663	       128.2 ns/op
-BenchmarkSimplePattern-8          	 9576802	       128.2 ns/op
-BenchmarkSimplePattern-8          	 9661042	       126.6 ns/op
-BenchmarkSimplePattern-8          	 9694028	       127.1 ns/op
-BenchmarkSimplePattern-8          	 9667344	       128.2 ns/op
-BenchmarkGetField-8               	 3030187	       410.5 ns/op
-BenchmarkGetField-8               	 3016743	       415.2 ns/op
-BenchmarkGetField-8               	 3023193	       407.7 ns/op
-BenchmarkGetField-8               	 3026401	       411.4 ns/op
-BenchmarkGetField-8               	 3023929	       414.0 ns/op
-BenchmarkSetField-8               	 1306578	       917.6 ns/op
-BenchmarkSetField-8               	 1322950	       906.2 ns/op
-BenchmarkSetField-8               	 1301011	       906.2 ns/op
-BenchmarkSetField-8               	 1315780	       904.3 ns/op
-BenchmarkSetField-8               	 1317502	       908.7 ns/op
-BenchmarkRegexMatch-8             	  943762	      1216 ns/op
-BenchmarkRegexMatch-8             	  971356	      1211 ns/op
-BenchmarkRegexMatch-8             	  970927	      1210 ns/op
-BenchmarkRegexMatch-8             	  975307	      1222 ns/op
-BenchmarkRegexMatch-8             	  934994	      1211 ns/op
-BenchmarkBinaryOperators-8        	 2391631	       498.6 ns/op
-BenchmarkBinaryOperators-8        	 2369451	       498.8 ns/op
-BenchmarkBinaryOperators-8        	 2390767	       498.3 ns/op
-BenchmarkBinaryOperators-8        	 2389590	       498.8 ns/op
-BenchmarkBinaryOperators-8        	 2395941	       499.5 ns/op
-BenchmarkConcatSmall-8            	 1653039	       720.6 ns/op
-BenchmarkConcatSmall-8            	 1646798	       719.2 ns/op
-BenchmarkConcatSmall-8            	 1654406	       720.7 ns/op
-BenchmarkConcatSmall-8            	 1656332	       720.6 ns/op
-BenchmarkConcatSmall-8            	 1647835	       718.8 ns/op
-BenchmarkConcatLarge-8            	     151	   7893331 ns/op
-BenchmarkConcatLarge-8            	     150	   7793927 ns/op
-BenchmarkConcatLarge-8            	     151	   7793702 ns/op
-BenchmarkConcatLarge-8            	     148	   7835946 ns/op
-BenchmarkConcatLarge-8            	     152	   8011698 ns/op
-BenchmarkComparisons-8            	 2806938	       423.8 ns/op
-BenchmarkComparisons-8            	 2829532	       420.5 ns/op
-BenchmarkComparisons-8            	 2841798	       420.3 ns/op
-BenchmarkComparisons-8            	 2837600	       421.0 ns/op
-BenchmarkComparisons-8            	 2821762	       420.9 ns/op
-BenchmarkArrayOperations-8        	 1755943	       682.1 ns/op
-BenchmarkArrayOperations-8        	 1750881	       680.4 ns/op
-BenchmarkArrayOperations-8        	 1753864	       681.2 ns/op
-BenchmarkArrayOperations-8        	 1744816	       680.9 ns/op
-BenchmarkArrayOperations-8        	 1754348	       680.4 ns/op
-BenchmarkAssign-8                 	 2548276	       469.3 ns/op
-BenchmarkAssign-8                 	 2542956	       467.3 ns/op
-BenchmarkAssign-8                 	 2553214	       468.0 ns/op
-BenchmarkAssign-8                 	 2548398	       468.4 ns/op
-BenchmarkAssign-8                 	 2540133	       469.4 ns/op
-BenchmarkAugAssign-8              	  747728	      1473 ns/op
-BenchmarkAugAssign-8              	  794076	      1473 ns/op
-BenchmarkAugAssign-8              	  791324	      1472 ns/op
-BenchmarkAugAssign-8              	  786385	      1473 ns/op
-BenchmarkAugAssign-8              	  787680	      1475 ns/op
+BenchmarkSimpleBuiltins-8         	 1000000	      1019 ns/op
+BenchmarkBuiltinMatch-8           	  356436	      3257 ns/op
+BenchmarkBuiltinMatch-8           	  365192	      3260 ns/op
+BenchmarkBuiltinMatch-8           	  361885	      3250 ns/op
+BenchmarkBuiltinMatch-8           	  359179	      3271 ns/op
+BenchmarkBuiltinMatch-8           	  356587	      3310 ns/op
+BenchmarkBuiltinLength-8          	  287806	      4061 ns/op
+BenchmarkBuiltinLength-8          	  290668	      4036 ns/op
+BenchmarkBuiltinLength-8          	  288462	      4023 ns/op
+BenchmarkBuiltinLength-8          	  290515	      4032 ns/op
+BenchmarkBuiltinLength-8          	  288842	      4039 ns/op
+BenchmarkBuiltinIndex-8           	  251661	      4614 ns/op
+BenchmarkBuiltinIndex-8           	  251402	      4610 ns/op
+BenchmarkBuiltinIndex-8           	  255114	      4651 ns/op
+BenchmarkBuiltinIndex-8           	  253759	      4611 ns/op
+BenchmarkBuiltinIndex-8           	  253326	      4603 ns/op
+BenchmarkBuiltinSubstr-8          	  304550	      3866 ns/op
+BenchmarkBuiltinSubstr-8          	  288042	      3878 ns/op
+BenchmarkBuiltinSubstr-8          	  303380	      3871 ns/op
+BenchmarkBuiltinSubstr-8          	  303454	      3938 ns/op
+BenchmarkBuiltinSubstr-8          	  301251	      3879 ns/op
+BenchmarkBuiltinSplitSpace-8      	   33603	     34934 ns/op
+BenchmarkBuiltinSplitSpace-8      	   33228	     35060 ns/op
+BenchmarkBuiltinSplitSpace-8      	   33790	     35013 ns/op
+BenchmarkBuiltinSplitSpace-8      	   33452	     34977 ns/op
+BenchmarkBuiltinSplitSpace-8      	   33823	     35144 ns/op
+BenchmarkBuiltinSplitRegex-8      	   13699	     87919 ns/op
+BenchmarkBuiltinSplitRegex-8      	   13699	     86570 ns/op
+BenchmarkBuiltinSplitRegex-8      	   13525	     86566 ns/op
+BenchmarkBuiltinSplitRegex-8      	   13669	     86370 ns/op
+BenchmarkBuiltinSplitRegex-8      	   13778	     86210 ns/op
+BenchmarkBuiltinSub-8             	   83263	     13506 ns/op
+BenchmarkBuiltinSub-8             	   85422	     13515 ns/op
+BenchmarkBuiltinSub-8             	   86358	     13498 ns/op
+BenchmarkBuiltinSub-8             	   85944	     13552 ns/op
+BenchmarkBuiltinSub-8             	   86341	     13513 ns/op
+BenchmarkBuiltinSubAmpersand-8    	   84591	     13625 ns/op
+BenchmarkBuiltinSubAmpersand-8    	   79032	     13510 ns/op
+BenchmarkBuiltinSubAmpersand-8    	   85719	     13597 ns/op
+BenchmarkBuiltinSubAmpersand-8    	   85870	     13517 ns/op
+BenchmarkBuiltinSubAmpersand-8    	   84552	     13492 ns/op
+BenchmarkBuiltinGsub-8            	   71922	     16081 ns/op
+BenchmarkBuiltinGsub-8            	   71772	     16060 ns/op
+BenchmarkBuiltinGsub-8            	   71998	     16039 ns/op
+BenchmarkBuiltinGsub-8            	   70380	     16030 ns/op
+BenchmarkBuiltinGsub-8            	   69432	     16267 ns/op
+BenchmarkBuiltinGsubAmpersand-8   	   74451	     16153 ns/op
+BenchmarkBuiltinGsubAmpersand-8   	   68089	     16088 ns/op
+BenchmarkBuiltinGsubAmpersand-8   	   71169	     16054 ns/op
+BenchmarkBuiltinGsubAmpersand-8   	   71498	     16182 ns/op
+BenchmarkBuiltinGsubAmpersand-8   	   71031	     16010 ns/op
+BenchmarkBuiltinSprintf-8         	  118070	      9734 ns/op
+BenchmarkBuiltinSprintf-8         	  118974	      9713 ns/op
+BenchmarkBuiltinSprintf-8         	  115725	      9712 ns/op
+BenchmarkBuiltinSprintf-8         	  118255	      9679 ns/op
+BenchmarkBuiltinSprintf-8         	  117768	      9747 ns/op
+BenchmarkRecursiveFunc-8          	   73342	     15756 ns/op
+BenchmarkRecursiveFunc-8          	   72694	     15779 ns/op
+BenchmarkRecursiveFunc-8          	   71516	     15790 ns/op
+BenchmarkRecursiveFunc-8          	   70902	     15777 ns/op
+BenchmarkRecursiveFunc-8          	   73366	     15775 ns/op
+BenchmarkFuncCall-8               	  403035	      2834 ns/op
+BenchmarkFuncCall-8               	  390921	      2845 ns/op
+BenchmarkFuncCall-8               	  402879	      2858 ns/op
+BenchmarkFuncCall-8               	  383294	      2832 ns/op
+BenchmarkFuncCall-8               	  389851	      2832 ns/op
+BenchmarkNativeFunc-8             	  106911	     10776 ns/op
+BenchmarkNativeFunc-8             	  106364	     10775 ns/op
+BenchmarkNativeFunc-8             	  103417	     10792 ns/op
+BenchmarkNativeFunc-8             	  104362	     10765 ns/op
+BenchmarkNativeFunc-8             	  104515	     10848 ns/op
+BenchmarkForLoop-8                	  258400	      4482 ns/op
+BenchmarkForLoop-8                	  247102	      4506 ns/op
+BenchmarkForLoop-8                	  259666	      4495 ns/op
+BenchmarkForLoop-8                	  234979	      4622 ns/op
+BenchmarkForLoop-8                	  250666	      4838 ns/op
+BenchmarkForInLoop-8              	  545274	      1974 ns/op
+BenchmarkForInLoop-8              	  598783	      2097 ns/op
+BenchmarkForInLoop-8              	  526208	      2068 ns/op
+BenchmarkForInLoop-8              	  536114	      1960 ns/op
+BenchmarkForInLoop-8              	  579930	      1977 ns/op
+BenchmarkIfStatement-8            	 4960626	       235.8 ns/op
+BenchmarkIfStatement-8            	 4640866	       245.8 ns/op
+BenchmarkIfStatement-8            	 5029837	       246.0 ns/op
+BenchmarkIfStatement-8            	 5010278	       238.3 ns/op
+BenchmarkIfStatement-8            	 5106625	       235.7 ns/op
+BenchmarkCondExpr-8               	 5647002	       203.9 ns/op
+BenchmarkCondExpr-8               	 5414143	       207.9 ns/op
+BenchmarkCondExpr-8               	 4712512	       236.7 ns/op
+BenchmarkCondExpr-8               	 5533946	       206.8 ns/op
+BenchmarkCondExpr-8               	 5771953	       212.6 ns/op
+BenchmarkSimplePattern-8          	 9278107	       131.2 ns/op
+BenchmarkSimplePattern-8          	 9403346	       132.6 ns/op
+BenchmarkSimplePattern-8          	 8935683	       130.7 ns/op
+BenchmarkSimplePattern-8          	 9428971	       139.7 ns/op
+BenchmarkSimplePattern-8          	 9409366	       132.4 ns/op
+BenchmarkGetField-8               	 2687168	       454.7 ns/op
+BenchmarkGetField-8               	 2647993	       462.0 ns/op
+BenchmarkGetField-8               	 2577046	       469.9 ns/op
+BenchmarkGetField-8               	 2720770	       448.4 ns/op
+BenchmarkGetField-8               	 2695162	       467.4 ns/op
+BenchmarkSetField-8               	 1257975	       945.5 ns/op
+BenchmarkSetField-8               	 1281472	       943.7 ns/op
+BenchmarkSetField-8               	 1283065	       926.0 ns/op
+BenchmarkSetField-8               	 1302412	       950.2 ns/op
+BenchmarkSetField-8               	 1270246	       950.7 ns/op
+BenchmarkRegexMatch-8             	  872226	      1290 ns/op
+BenchmarkRegexMatch-8             	  941103	      1266 ns/op
+BenchmarkRegexMatch-8             	  962432	      1224 ns/op
+BenchmarkRegexMatch-8             	  946812	      1230 ns/op
+BenchmarkRegexMatch-8             	  953548	      1233 ns/op
+BenchmarkBinaryOperators-8        	 2346142	       512.5 ns/op
+BenchmarkBinaryOperators-8        	 2341258	       514.3 ns/op
+BenchmarkBinaryOperators-8        	 2339524	       515.2 ns/op
+BenchmarkBinaryOperators-8        	 2369112	       522.5 ns/op
+BenchmarkBinaryOperators-8        	 2354088	       518.3 ns/op
+BenchmarkConcatSmall-8            	 1603718	       760.5 ns/op
+BenchmarkConcatSmall-8            	 1587866	       728.9 ns/op
+BenchmarkConcatSmall-8            	 1634402	       741.8 ns/op
+BenchmarkConcatSmall-8            	 1560367	       750.4 ns/op
+BenchmarkConcatSmall-8            	 1578094	       756.3 ns/op
+BenchmarkConcatLarge-8            	     145	   8533993 ns/op
+BenchmarkConcatLarge-8            	     152	   7731278 ns/op
+BenchmarkConcatLarge-8            	     129	   8575358 ns/op
+BenchmarkConcatLarge-8            	     146	   8482356 ns/op
+BenchmarkConcatLarge-8            	     140	   8094782 ns/op
+BenchmarkComparisons-8            	 2561576	       461.0 ns/op
+BenchmarkComparisons-8            	 2602875	       442.5 ns/op
+BenchmarkComparisons-8            	 2746752	       437.5 ns/op
+BenchmarkComparisons-8            	 2673856	       438.6 ns/op
+BenchmarkComparisons-8            	 2695856	       436.6 ns/op
+BenchmarkArrayOperations-8        	 1762815	       694.3 ns/op
+BenchmarkArrayOperations-8        	 1727587	       677.9 ns/op
+BenchmarkArrayOperations-8        	 1756686	       671.6 ns/op
+BenchmarkArrayOperations-8        	 1753754	       686.9 ns/op
+BenchmarkArrayOperations-8        	 1715215	       686.9 ns/op
+BenchmarkAssign-8                 	 2577278	       461.6 ns/op
+BenchmarkAssign-8                 	 2509764	       462.6 ns/op
+BenchmarkAssign-8                 	 2598711	       460.6 ns/op
+BenchmarkAssign-8                 	 2579676	       462.0 ns/op
+BenchmarkAssign-8                 	 2565871	       461.0 ns/op
+BenchmarkAugAssign-8              	  791190	      1493 ns/op
+BenchmarkAugAssign-8              	  795606	      1483 ns/op
+BenchmarkAugAssign-8              	  792934	      1473 ns/op
+BenchmarkAugAssign-8              	  797379	      1478 ns/op
+BenchmarkAugAssign-8              	  810571	      1473 ns/op
 PASS
-ok  	github.com/benhoyt/goawk/interp	271.744s
+ok  	github.com/benhoyt/goawk/interp	268.231s
diff --git a/goawk.go b/goawk.go
index 1602f12..10a5b34 100644
--- a/goawk.go
+++ b/goawk.go
@@ -44,7 +44,7 @@ import (
 )
 
 const (
-	version    = "v1.10.0"
+	version    = "v1.11.0"
 	copyright  = "GoAWK " + version + " - Copyright (c) 2021 Ben Hoyt"
 	shortUsage = "usage: goawk [-F fs] [-v var=value] [-f progfile | 'prog'] [file ...]"
 	longUsage  = `Standard AWK arguments:
@@ -56,6 +56,7 @@ const (
         load AWK source from progfile (multiple allowed)
 
 Additional GoAWK arguments:
+  -b    use byte indexes for index(), length(), match(), and substr()
   -cpuprofile file
         write CPU profile to file
   -d    debug mode (print parsed AST to stderr)
@@ -77,6 +78,7 @@ func main() {
 	debug := false
 	debugTypes := false
 	memprofile := ""
+	useBytes := false
 
 	var i int
 	for i = 1; i < len(os.Args); i++ {
@@ -109,6 +111,8 @@ func main() {
 			}
 			i++
 			vars = append(vars, os.Args[i])
+		case "-b":
+			useBytes = true
 		case "-cpuprofile":
 			if i+1 >= len(os.Args) {
 				errorExitf("flag needs an argument: -cpuprofile")
@@ -212,6 +216,7 @@ func main() {
 	config := &interp.Config{
 		Argv0: filepath.Base(os.Args[0]),
 		Args:  expandWildcardsOnWindows(args),
+		Bytes: useBytes,
 		Vars:  []string{"FS", fieldSep},
 	}
 	for _, v := range vars {
diff --git a/interp/functions.go b/interp/functions.go
index 469188d..26b446c 100644
--- a/interp/functions.go
+++ b/interp/functions.go
@@ -98,49 +98,98 @@ func (p *interp) callBuiltin(op Token, argExprs []Expr) (value, error) {
 	// Then switch on the function for the ordinary functions
 	switch op {
 	case F_LENGTH:
-		switch len(args) {
-		case 0:
-			return num(float64(len(p.line))), nil
-		default:
-			return num(float64(len(p.toString(args[0])))), nil
+		var s string
+		if len(args) > 0 {
+			s = p.toString(args[0])
+		} else {
+			s = p.line
 		}
+		var n int
+		if p.bytes {
+			n = len(s)
+		} else {
+			n = utf8.RuneCountInString(s)
+		}
+		return num(float64(n)), nil
 
 	case F_MATCH:
 		re, err := p.compileRegex(p.toString(args[1]))
 		if err != nil {
 			return null(), err
 		}
-		loc := re.FindStringIndex(p.toString(args[0]))
+		s := p.toString(args[0])
+		loc := re.FindStringIndex(s)
 		if loc == nil {
 			p.matchStart = 0
 			p.matchLength = -1
 			return num(0), nil
 		}
-		p.matchStart = loc[0] + 1
-		p.matchLength = loc[1] - loc[0]
+		if p.bytes {
+			p.matchStart = loc[0] + 1
+			p.matchLength = loc[1] - loc[0]
+		} else {
+			p.matchStart = utf8.RuneCountInString(s[:loc[0]]) + 1
+			p.matchLength = utf8.RuneCountInString(s[loc[0]:loc[1]])
+		}
 		return num(float64(p.matchStart)), nil
 
 	case F_SUBSTR:
 		s := p.toString(args[0])
 		pos := int(args[1].num())
-		if pos > len(s) {
-			pos = len(s) + 1
-		}
-		if pos < 1 {
-			pos = 1
-		}
-		maxLength := len(s) - pos + 1
-		length := maxLength
-		if len(args) == 3 {
-			length = int(args[2].num())
-			if length < 0 {
-				length = 0
+		if p.bytes {
+			if pos > len(s) {
+				pos = len(s) + 1
 			}
-			if length > maxLength {
-				length = maxLength
+			if pos < 1 {
+				pos = 1
 			}
+			maxLength := len(s) - pos + 1
+			length := maxLength
+			if len(args) == 3 {
+				length = int(args[2].num())
+				if length < 0 {
+					length = 0
+				}
+				if length > maxLength {
+					length = maxLength
+				}
+			}
+			return str(s[pos-1 : pos-1+length]), nil
+		} else {
+			// Count characters till we get to pos.
+			chars := 1
+			start := 0
+			for start = range s {
+				chars++
+				if chars > pos {
+					break
+				}
+			}
+			if pos >= chars {
+				start = len(s)
+			}
+
+			// Count characters from start till we reach length.
+			var end int
+			if len(args) == 3 {
+				length := int(args[2].num())
+				chars = 0
+				for end = range s[start:] {
+					chars++
+					if chars > length {
+						break
+					}
+				}
+				if length >= chars {
+					end = len(s)
+				} else {
+					end += start
+				}
+			} else {
+				end = len(s)
+			}
+			return str(s[start:end]), nil
 		}
-		return str(s[pos-1 : pos-1+length]), nil
 
 	case F_SPRINTF:
 		s, err := p.sprintf(p.toString(args[0]), args[1:])
@@ -152,7 +201,16 @@ func (p *interp) callBuiltin(op Token, argExprs []Expr) (value, error) {
 	case F_INDEX:
 		s := p.toString(args[0])
 		substr := p.toString(args[1])
-		return num(float64(strings.Index(s, substr) + 1)), nil
+		index := strings.Index(s, substr)
+		if p.bytes {
+			return num(float64(index + 1)), nil
+		} else {
+			if index < 0 {
+				return num(float64(0)), nil
+			}
+			index = utf8.RuneCountInString(s[:index])
+			return num(float64(index + 1)), nil
+		}
 
 	case F_TOLOWER:
 		return str(strings.ToLower(p.toString(args[0]))), nil
@@ -495,7 +553,7 @@ func (p *interp) initNativeFuncs(funcs map[string]interface{}) error {
 var errorType = reflect.TypeOf((*error)(nil)).Elem()
 
 // Check that native function with given name is okay to call from
-// AWK, return a *interp.Error if not. This checks that f is actually
+// AWK, return an *interp.Error if not. This checks that f is actually
 // a function, and that its parameter and return types are good.
 func checkNativeFunc(name string, f interface{}) error {
 	if KeywordToken(name) != ILLEGAL {
diff --git a/interp/interp.go b/interp/interp.go
index 0c466f2..82ca600 100644
--- a/interp/interp.go
+++ b/interp/interp.go
@@ -118,6 +118,7 @@ type interp struct {
 	exitStatus  int
 	regexCache  map[string]*regexp.Regexp
 	formatCache map[string]cachedFormat
+	bytes       bool
 }
 
 // Various const configuration. Could make these part of Config if
@@ -201,6 +202,11 @@ type Config struct {
 	// array, for example []string{"USER", "bob", "HOME", "/home/bob"}.
 	// If nil (the default), values from os.Environ() are used.
 	Environ []string
+
+	// Set to true to use byte indexes instead of character indexes for
+	// the index, length, match, and substr functions. Note: the default
+	// was changed from bytes to characters in GoAWK version 1.11.
+	Bytes bool
 }
 
 // ExecProgram executes the parsed program using the given interpreter
@@ -241,6 +247,7 @@ func ExecProgram(program *Program, config *Config) (int, error) {
 	p.noExec = config.NoExec
 	p.noFileWrites = config.NoFileWrites
 	p.noFileReads = config.NoFileReads
+	p.bytes = config.Bytes
 	err := p.initNativeFuncs(config.Funcs)
 	if err != nil {
 		return 0, err


Test patch:
diff --git a/goawk_test.go b/goawk_test.go
index 4d109af..9352f90 100644
--- a/goawk_test.go
+++ b/goawk_test.go
@@ -351,6 +351,10 @@ func TestCommandLine(t *testing.T) {
 		{[]string{"-v", "RS=;", `$0`}, "a b;c\nd;e", "a b\nc\nd\ne\n", ""},
 		{[]string{"-vRS=;", `$0`}, "a b;c\nd;e", "a b\nc\nd\ne\n", ""},
 
+		// Byte index vs character index mode
+		{[]string{`{ print length }  # !windows-gawk`}, "絵\n", "1\n", ""},
+		{[]string{"-b", `{ print length }`}, "絵\n", "3\n", ""},
+
 		// ARGV/ARGC handling
 		{[]string{`
 			BEGIN {
@@ -440,6 +444,11 @@ func runGoAWK(args []string, stdin string) (stdout, stderr string, err error) {
 }
 
 func runAWKs(t *testing.T, testArgs []string, testStdin, testOutput, testError string) {
+	for _, arg := range testArgs {
+		if strings.Contains(arg, "!"+runtime.GOOS+"-"+awkExe) {
+			t.Skipf("skipping on %s under %s", runtime.GOOS, awkExe)
+		}
+	}
 	cmd := exec.Command(awkExe, testArgs...)
 	if testStdin != "" {
 		cmd.Stdin = bytes.NewReader([]byte(testStdin))
diff --git a/interp/interp_test.go b/interp/interp_test.go
index 72417f9..8d8f19e 100644
--- a/interp/interp_test.go
+++ b/interp/interp_test.go
@@ -406,25 +406,29 @@ BEGIN {
 	{`BEGIN { print match("x food y", "fo"), RSTART, RLENGTH }`, "", "3 3 2\n", "", ""},
 	{`BEGIN { print match("x food y", "fox"), RSTART, RLENGTH }`, "", "0 0 -1\n", "", ""},
 	{`BEGIN { print match("x food y", /[fod]+/), RSTART, RLENGTH }`, "", "3 3 4\n", "", ""},
+	{`BEGIN { print match("絵 fööd y", /[föd]+/), RSTART, RLENGTH }`, "", "3 3 4\n", "", ""},
 	{`{ print length, length(), length("buzz"), length("") }`, "foo bar", "7 7 4 0\n", "", ""},
+	{`BEGIN { print length("a"), length("絵") }  # !awk`, "", "1 1\n", "", ""},
 	{`BEGIN { print index("foo", "f"), index("foo0", 0), index("foo", "o"), index("foo", "x") }`, "", "1 4 2 0\n", "", ""},
+	{`BEGIN { print index("föö", "f"), index("föö0", 0), index("föö", "ö"), index("föö", "x") }`, "", "1 4 2 0\n", "", ""},
 	{`BEGIN { print atan2(1, 0.5), atan2(-1, 0) }`, "", "1.10715 -1.5708\n", "", ""},
 	{`BEGIN { print sprintf("%3d", 42) }`, "", " 42\n", "", ""},
 	{`BEGIN { print sprintf("%d", 12, 34) }`, "", "12\n", "", ""},
 	{`BEGIN { print sprintf("%d") }`, "", "", "format error: got 0 args, expected 1", "not enough arg"},
 	{`BEGIN { print sprintf("%d", 12, 34) }`, "", "12\n", "", ""},
 	{`BEGIN { print sprintf("% 5d", 42) }`, "", "   42\n", "", ""},
-	{`BEGIN { print substr("food", 1) }`, "", "food\n", "", ""},
-	{`BEGIN { print substr("food", 1, 2) }`, "", "fo\n", "", ""},
-	{`BEGIN { print substr("food", 1, 4) }`, "", "food\n", "", ""},
-	{`BEGIN { print substr("food", 1, 8) }`, "", "food\n", "", ""},
-	{`BEGIN { print substr("food", 2) }`, "", "ood\n", "", ""},
-	{`BEGIN { print substr("food", 2, 2) }`, "", "oo\n", "", ""},
-	{`BEGIN { print substr("food", 2, 3) }`, "", "ood\n", "", ""},
-	{`BEGIN { print substr("food", 2, 8) }`, "", "ood\n", "", ""},
-	{`BEGIN { print substr("food", 0, 8) }`, "", "food\n", "", ""},
-	{`BEGIN { print substr("food", -1, 8) }`, "", "food\n", "", ""},
-	{`BEGIN { print substr("food", 5, 8) }`, "", "\n", "", ""},
+	{`BEGIN { print substr("food", 1), substr("fööd", 1) }  # !windows-gawk`, "", "food fööd\n", "", ""},
+	{`BEGIN { print substr("food", 1, 2), substr("fööd", 1, 2) }  # !windows-gawk`, "", "fo fö\n", "", ""},
+	{`BEGIN { print substr("food", 1, 4), substr("fööd", 1, 4) }  # !windows-gawk`, "", "food fööd\n", "", ""},
+	{`BEGIN { print substr("food", 1, 8), substr("fööd", 1, 8) }  # !windows-gawk`, "", "food fööd\n", "", ""},
+	{`BEGIN { print substr("food", 2), substr("fööd", 2) }  # !windows-gawk`, "", "ood ööd\n", "", ""},
+	{`BEGIN { print substr("food", 2, 2), substr("fööd", 2, 2) }  # !windows-gawk`, "", "oo öö\n", "", ""},
+	{`BEGIN { print substr("food", 2, 3), substr("fööd", 2, 3) }  # !windows-gawk`, "", "ood ööd\n", "", ""},
+	{`BEGIN { print substr("food", 2, 8), substr("fööd", 2, 8) }  # !windows-gawk`, "", "ood ööd\n", "", ""},
+	{`BEGIN { print substr("food", 0, 8), substr("fööd", 0, 8) }  # !windows-gawk`, "", "food fööd\n", "", ""},
+	{`BEGIN { print substr("food", -1, 8), substr("fööd", -1, 8) }  # !windows-gawk`, "", "food fööd\n", "", ""},
+	{`BEGIN { print substr("food", 5, 8), substr("fööd", 5, 8) }`, "", " \n", "", ""},
+	{`BEGIN { print substr("food", 2, -3), substr("fööd", 2, -3) }`, "", " \n", "", ""},
 	{`BEGIN { n = split("ab c d ", a); for (i=1; i<=n; i++) print a[i] }`, "", "ab\nc\nd\n", "", ""},
 	{`BEGIN { n = split("ab,c,d,", a, ","); for (i=1; i<=n; i++) print a[i] }`, "", "ab\nc\nd\n\n", "", ""},
 	{`BEGIN { n = split("ab,c.d,", a, /[,.]/); for (i=1; i<=n; i++) print a[i] }`, "", "ab\nc\nd\n\n", "", ""},
@@ -653,33 +657,37 @@ func TestInterp(t *testing.T) {
 			testName = testName[:70]
 		}
 
-		if awkExe != "" && !strings.Contains(test.src, "!"+awkExe) {
-			// Run it through external awk program first
-			t.Run("awk_"+testName, func(t *testing.T) {
-				cmd := exec.Command(awkExe, test.src, "-")
-				if test.in != "" {
-					cmd.Stdin = strings.NewReader(test.in)
-				}
-				out, err := cmd.CombinedOutput()
-				if err != nil {
-					if test.awkErr != "" {
-						if strings.Contains(string(out), test.awkErr) {
-							return
-						}
-						t.Fatalf("expected error %q, got:\n%s", test.awkErr, out)
-					} else {
-						t.Fatalf("error running %s: %v:\n%s", awkExe, err, out)
-					}
-				}
+		// Run it through external awk program first
+		t.Run("awk_"+testName, func(t *testing.T) {
+			if awkExe != "" && strings.Contains(test.src, "!"+awkExe) {
+				t.Skipf("skipping under %s", awkExe)
+			}
+			if strings.Contains(test.src, "!"+runtime.GOOS+"-"+awkExe) {
+				t.Skipf("skipping on %s under %s", runtime.GOOS, awkExe)
+			}
+			cmd := exec.Command(awkExe, test.src, "-")
+			if test.in != "" {
+				cmd.Stdin = strings.NewReader(test.in)
+			}
+			out, err := cmd.CombinedOutput()
+			if err != nil {
 				if test.awkErr != "" {
-					t.Fatalf(`expected error %q, got ""`, test.awkErr)
-				}
-				normalized := normalizeNewlines(string(out))
-				if normalized != test.out {
-					t.Fatalf("expected %q, got %q", test.out, normalized)
+					if strings.Contains(string(out), test.awkErr) {
+						return
+					}
+					t.Fatalf("expected error %q, got:\n%s", test.awkErr, out)
+				} else {
+					t.Fatalf("error running %s: %v:\n%s", awkExe, err, out)
 				}
-			})
-		}
+			}
+			if test.awkErr != "" {
+				t.Fatalf(`expected error %q, got ""`, test.awkErr)
+			}
+			normalized := normalizeNewlines(string(out))
+			if normalized != test.out {
+				t.Fatalf("expected %q, got %q", test.out, normalized)
+			}
+		})
 
 		// Then test it in GoAWK
 		t.Run(testName, func(t *testing.T) {
@@ -1028,6 +1036,47 @@ func TestSafeMode(t *testing.T) {
 	}
 }
 
+func TestBytesMode(t *testing.T) {
+	tests := []struct {
+		src string
+		in  string
+		out string
+	}{
+		{`BEGIN { print match("food", "foo"), RSTART, RLENGTH }`, "", "1 1 3\n"},
+		{`BEGIN { print match("x food y", "fo"), RSTART, RLENGTH }`, "", "3 3 2\n"},
+		{`BEGIN { print match("x food y", "fox"), RSTART, RLENGTH }`, "", "0 0 -1\n"},
+		{`BEGIN { print match("x food y", /[fod]+/), RSTART, RLENGTH }`, "", "3 3 4\n"},
+		{`BEGIN { print match("絵 fööd y", /[föd]+/), RSTART, RLENGTH }`, "", "5 5 6\n"},
+		{`{ print length, length(), length("buzz"), length("") }`, "foo bar", "7 7 4 0\n"},
+		{`BEGIN { print length("a"), length("絵") }  # !awk`, "", "1 3\n"},
+		{`BEGIN { print index("foo", "f"), index("foo0", 0), index("foo", "o"), index("foo", "x") }`, "", "1 4 2 0\n"},
+		{`BEGIN { print index("föö", "f"), index("föö0", 0), index("föö", "ö"), index("föö", "x") }`, "", "1 6 2 0\n"},
+		{`BEGIN { print substr("food", 1), substr("fööd", 1) }`, "", "food fööd\n"},
+		{`BEGIN { print substr("food", 1, 2), substr("fööd", 1, 2) }`, "", "fo f\xc3\n"},
+		{`BEGIN { print substr("food", 1, 4), substr("fööd", 1, 4) }`, "", "food fö\xc3\n"},
+		{`BEGIN { print substr("food", 1, 8), substr("fööd", 1, 8) }`, "", "food fööd\n"},
+		{`BEGIN { print substr("food", 2), substr("fööd", 2) }`, "", "ood ööd\n"},
+		{`BEGIN { print substr("food", 2, 2), substr("fööd", 2, 2) }`, "", "oo ö\n"},
+		{`BEGIN { print substr("food", 2, 3), substr("fööd", 2, 3) }`, "", "ood ö\xc3\n"},
+		{`BEGIN { print substr("food", 2, 8), substr("fööd", 2, 8) }`, "", "ood ööd\n"},
+		{`BEGIN { print substr("food", 0, 8), substr("fööd", 0, 8) }`, "", "food fööd\n"},
+		{`BEGIN { print substr("food", -1, 8), substr("fööd", -1, 8) }`, "", "food fööd\n"},
+		{`BEGIN { print substr("food", 5, 8), substr("fööd", 5, 8) }`, "", " \xb6d\n"},
+		{`BEGIN { print substr("food", 2, -3), substr("fööd", 2, -3) }`, "", " \n"},
+	}
+	for _, test := range tests {
+		testName := test.src
+		if len(testName) > 70 {
+			testName = testName[:70]
+		}
+		t.Run(testName, func(t *testing.T) {
+			testGoAWK(t, test.src, test.in, test.out, "", nil, func(config *interp.Config) {
+				config.Bytes = true
+			})
+		})
+	}
+}
+
 func TestConfigVarsCorrect(t *testing.T) {
 	prog, err := parser.ParseProgram([]byte(`BEGIN { print x }`), nil)
 	if err != nil {