Repository: tetratelabs/wazero
Pull request #63 (2021-12-02 09:22:10)

Adds skeleton lexer and benchmarks for text format
This adds an incomplete lexer for the Web Assembly text format. This is
left incomplete for the purpose of a smaller review. The code may change
as we add a parser and complete lexing floating point, but probably not
too much.

This also adds benchmarks as it is very easy to hurt performance unless
you compare before and after every change.

One design note is this uses a single loop instead of an iterator api more
common in lexers (ex call .nextToken etc). This is to save allocations and
the need to define yet another intermediate type.

For example, lexing the fibonnaci example is allocation free (at the
moment anyway!). It is important to note that the comparison against
Wat2Wasm isn't fair because they don't expose the lexing component.
Later, when we parse it will become more fair. It would also be ideal to
compare against wasm-tools somehow, but I've not found a C bridge to
that (rust project).

```
/private/var/folders/vd/1cf8zdb1721f4z5rjggy8bp40000gn/T/GoLand/___BenchmarkLex_in_github_com_tetratelabs_wazero_wasm_wat.test -test.v -test.paniconexit0 -test.bench ^\QBenchmarkLex\E$ -test.run ^$
goos: darwin
goarch: amd64
pkg: github.com/tetratelabs/wazero/wasm/wat
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkLex
BenchmarkLex/example_vs_utf8.ValidString
BenchmarkLex/example_vs_utf8.ValidString-16         	 2007007	       588.0 ns/op	     704 B/op	       1 allocs/op
BenchmarkLex/example_vs_wasmtime.Wat2Wasm
BenchmarkLex/example_vs_wasmtime.Wat2Wasm-16        	   45301	     26035 ns/op	     848 B/op	       3 allocs/op
BenchmarkLex/example
BenchmarkLex/example-16                             	 1000000	      1025 ns/op	       0 B/op	       0 allocs/op
```

codefromthecrypt: Switched lexer to a incremental parse loop. This will allow us to more cheaply handle postion, column, etc and also avoid allocations when it comes to integrating with a real parser.
codefromthecrypt: here are some early results of the general logic. There will be some optimizations left as there are some escapes noted in `go build -gcflags -m wasm/wat/*.go` which may not be great. Meanwhile, I'll move along functionality wise

```
BenchmarkLex
BenchmarkLex/base
BenchmarkLex/base-16         	67579126	        15.85 ns/op
BenchmarkLex/whitespace_chars
BenchmarkLex/whitespace_chars-16         	61862394	        19.48 ns/op
BenchmarkLex/unicode_line_comment
BenchmarkLex/unicode_line_comment-16     	26326196	        45.67 ns/op
BenchmarkLex/unicode_block_comment
BenchmarkLex/unicode_block_comment-16    	20848278	        52.55 ns/op
```
codefromthecrypt: changed the example to use fibonacci and rejigged things to ease test writing. there's now some visible tech debt on token types that have similar branches. I'll work on that probably a lot of it will get better with a bitflag table to identify the token type from the current first byte
codefromthecrypt: today I reworked logic to eliminate some (not all) tech debt and work on performance. I'm happy on performance for now as reached an emotional threshold of 1us to lex the example.
codefromthecrypt: I thought about @mathetake's comments last night and in the middle of it :D: he's right. error reporting should win vs responsibility separation. anything that nests is particularly tricky to understand. The part that confuses this lexing is that parens are used both in block comments and in S expressions. Right now, we are only counting nesting in block comments because we are required to for unicode skipping to work. The better solution I think vs comments is to enforce nesting in normal parens, also. This brings coherence to the two places we are nesting as they need similar error reporting.

Ex. now at the lexing layer, we allow ")(" which is valid tokenization, but invalid code. The error for ")(" will be very similar to the same broken block comment ";)(;". If we will move paren tokenization up into the same place that handles white space, we reduce questions about the overlap in characters and also can tighten reporting. Counting and enforcing parens costs a little code, but we'd have to do it at the parsing layer if not here. So, it ends up at worst a wash, performance wise.

codefromthecrypt: @mathetake thanks for the pointers, PTAL at the latest commit if you have a minute 🙇 

Patch:
diff --git a/go.mod b/go.mod
index 63f44bb1..d5cc77c3 100644
--- a/go.mod
+++ b/go.mod
@@ -4,6 +4,8 @@ module github.com/tetratelabs/wazero
 go 1.16
 
 require (
+	// only used in wat benchmarks
+	github.com/bytecodealliance/wasmtime-go v0.31.0
 	github.com/stretchr/testify v1.5.1
 	// Once we reach some maturity, remove this dep and implement our own assembler.
 	github.com/twitchyliquid64/golang-asm v0.15.1
diff --git a/go.sum b/go.sum
index 713c6111..4e2c3809 100644
--- a/go.sum
+++ b/go.sum
@@ -1,3 +1,5 @@
+github.com/bytecodealliance/wasmtime-go v0.31.0 h1:AbMdV1pwjw/0Ito5yARcGzY366cq5NIiDk5vpy1c2Lw=
+github.com/bytecodealliance/wasmtime-go v0.31.0/go.mod h1:q320gUxqyI8yB+ZqRuaJOEnGkAnHh6WtJjMaT2CW4wI=
 github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
diff --git a/wasm/wat/lexer.go b/wasm/wat/lexer.go
new file mode 100644
index 00000000..7e7c1c02
--- /dev/null
+++ b/wasm/wat/lexer.go
@@ -0,0 +1,256 @@
+package wat
+
+import (
+	"fmt"
+	"unicode/utf8"
+)
+
+// parseToken allows a parser to inspect a token without necessarily allocating strings
+// * source is the underlying byte stream: do not modify this
+// * tokenType is the token type
+// * beginPos is the byte position in the source where the token begins, inclusive
+// * endPos is the byte position in the source where the token ends, exclusive
+//
+// Returning an error will short-circuit any future invocations.
+type parseToken func(source []byte, tok tokenType, beginLine, beginCol, beginPos, endPos int) error
+
+// lex invokes the parser function for each token, the source is exhausted.
+//
+// Errors from the parser or during tokenization exit early, such as dangling block comments or unexpected characters.
+func lex(source []byte, parser parseToken) error {
+	length := len(source)
+	// p is the position in the source, and a parameter to the parser.
+	p := 0
+	// line is the line number in the source as determined by unescaped '\n' characters, and a parameter to the parser.
+	line := 1
+	// col is the UTF-8 aware column number, and a parameter to the parser
+	col := 0
+
+	// Web assembly expressions are grouped by parenthesis, even the minimal example "(module)". We track nesting level
+	// to help report problems instead of bubbling to the parser layer.
+	nestLevel := 0
+
+	// Block comments, ex. (; comment ;), can span multiple lines and also nest, ex. (; one (; two ;) ).
+	// There may be no block comments, but we declare the variable that tracks them here, as it is more efficient vs
+	// inline processing.
+	blockCommentLevel := 0
+
+	for ; p < length; p = p + 1 {
+		b1 := source[p]
+
+		// The spec does not consider newlines apart from '\n'. Notably, a bare '\r' is not a newline here.
+		// See https://www.w3.org/TR/wasm-core-1/#text-comment
+		if b1 == '\n' {
+			line = line + 1
+			col = 0
+			continue // next line
+		}
+
+		// Otherwise, advance the column regardless of whether we are whitespace or not
+		col = col + 1                              // the current character is at least one byte long
+		if b1 == ' ' || b1 == '\t' || b1 == '\r' { // fast path ASCII whitespace
+			continue // next whitespace
+		}
+
+		// Handle parens and comments, noting block comments, ex. "(; look! ;)", can be nested.
+		switch b1 {
+		case '(':
+			peekPos := p + 1
+			if peekPos == length { // invalid regardless of block comment or not. nothing opens at EOF!
+				return fmt.Errorf("%d:%d found '(' at end of input", line, col)
+			}
+			if source[peekPos] == ';' { // next block comment
+				p = peekPos // continue after "(;"
+				col = col + 1
+				blockCommentLevel = blockCommentLevel + 1
+				continue
+			} else if blockCommentLevel == 0 {
+				if e := parser(source, tokenLParen, line, col, p, p+1); e != nil {
+					return e
+				}
+				nestLevel = nestLevel + 1
+				continue
+			}
+		case ')':
+			if blockCommentLevel == 0 {
+				if nestLevel == 0 {
+					return fmt.Errorf("%d:%d found ')' before '('", line, col)
+				}
+				if e := parser(source, tokenRParen, line, col, p, p+1); e != nil {
+					return e
+				}
+				nestLevel = nestLevel - 1
+				continue
+			}
+		case ';': // possible line comment or block comment end
+			peekPos := p + 1
+			if peekPos < length {
+				b2 := source[peekPos]
+				if blockCommentLevel > 0 && b2 == ')' {
+					p = peekPos // continue after ";)"
+					col = col + 1
+					blockCommentLevel = blockCommentLevel - 1
+					continue
+				}
+
+				if b2 == ';' { // line comment
+					// Start after ";;" and run until the end. Note UTF-8 (multi-byte) characters are allowed.
+					peekPos = peekPos + 1
+					col = col + 1
+
+				LineComment:
+					for peekPos < length {
+						peeked := source[peekPos]
+						if peeked == '\n' {
+							break LineComment // EOL bookkeeping will proceed on the next iteration
+						}
+
+						col = col + 1
+						s := utf8Size[peeked] // While unlikely, it is possible the byte peeked is invalid unicode
+						if s == 0 {
+							return fmt.Errorf("%d:%d found an invalid byte in line comment: 0x%x", line, col, peeked)
+						}
+						peekPos = peekPos + s
+					}
+
+					// -1 because for loop will + 1: This optimizes speed of tokenization over line comments.
+					p = peekPos - 1 // at the '\n'
+					continue        // end of line comment
+				}
+			}
+		}
+
+		// non-ASCII is only supported in comments. Check UTF-8 size as we may need to set position > column!
+		if blockCommentLevel > 0 {
+			s := utf8Size[b1] // While unlikely, it is possible the current byte is invalid unicode
+			if s == 0 {
+				return fmt.Errorf("%d:%d found an invalid byte in block comment: 0x%x", line, col, b1)
+			}
+			p = p + s - 1 // -1 because for loop will + 1: This optimizes speed of tokenization over block comments.
+			continue
+		}
+
+		// One design-affecting constraint is that all tokens begin and end with a 7-bit ASCII character. This
+		// simplifies line and column counting and how to detect the end of a token.
+		//
+		// Note: Even though string allows unicode contents it is enclosed by ASCII double-quotes (").
+		//
+		// See https://www.w3.org/TR/wasm-core-1/#characters%E2%91%A0
+
+		// Track positions passed to the parser
+		// beginLine == line because no token is allowed to include an unescaped '\n'
+		beginCol := col  // the start column of the token (fixed)
+		beginPos := p    // the start position of the token (fixed)
+		peekPos := p + 1 // when finished scanning, this becomes endPos (the position after the token).
+
+		tok := firstTokenByte[b1]
+		switch tok {
+		case tokenSN: // min 2 bytes for sign and number; ambiguous: could be tokenFN
+			return fmt.Errorf("%d:%d TODO: signed", line, col)
+		case tokenUN: // min 1 byte; ambiguous when >=3 bytes as could be tokenFN
+			if peekPos < length {
+				peeked := source[peekPos]
+				if peeked == 'x' {
+					return fmt.Errorf("%d:%d TODO: hex", line, col)
+				}
+			Number:
+				// Start after the number and run until the end. Note all allowed characters are single byte.
+				for ; peekPos < length; peekPos = peekPos + 1 {
+					switch source[peekPos] {
+					case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '_':
+						p = peekPos
+						col = col + 1
+					default:
+						break Number // end of this token (or malformed, which the next loop will notice)
+					}
+				}
+			}
+		case tokenString: // min 2 bytes for empty string ("")
+			hitQuote := false
+			// Start at the second character and run until the end. Note UTF-8 (multi-byte) characters are allowed.
+		String:
+			for peekPos < length {
+				peeked := source[peekPos]
+				if peeked == '"' { // TODO: escaping and banning disallowed characters like newlines.
+					hitQuote = true
+					break String
+				}
+
+				col = col + 1
+				s := utf8Size[peeked] // While unlikely, it is possible the current byte is invalid unicode
+				if s == 0 {
+					return fmt.Errorf("%d:%d found an invalid byte in string token: 0x%x", line, col, peeked)
+				}
+				peekPos = peekPos + s
+			}
+
+			if !hitQuote {
+				return fmt.Errorf("%d:%d expected end quote", line, col)
+			}
+
+			p = peekPos
+			// set the position to after the quote
+			peekPos = peekPos + 1
+			col = col + 1
+		case tokenKeyword, tokenID, tokenReserved: // min 1 byte; end with zero or more idChar
+			// Start after the first character and run until the end. Note all allowed characters are single byte.
+		IdChars:
+			for ; peekPos < length; peekPos = peekPos + 1 {
+				if !idChar[source[peekPos]] {
+					break IdChars // end of this token (or malformed, which the next loop will notice)
+				}
+				col = col + 1
+			}
+			p = peekPos - 1
+		default:
+			if b1 > 0x7F { // non-ASCII
+				r, _ := utf8.DecodeRune(source[line:])
+				return fmt.Errorf("%d:%d expected an ASCII character, not %s", line, col, string(r))
+			}
+			return fmt.Errorf("%d:%d unexpected character %s", line, col, string(b1))
+		}
+
+		// Unsigned floating-point constants for infinity or canonical NaN (not a number) clash with keyword
+		// representation. For example, "nan" and "inf" are floating-point constants, while "nano" and "info" are
+		// possible keywords. See https://www.w3.org/TR/wasm-core-1/#floating-point%E2%91%A6
+		//
+		// TODO: Ex. inf nan nan:0xfffffffffffff or nan:0x400000
+
+		if e := parser(source, tok, line, beginCol, beginPos, peekPos); e != nil {
+			return e
+		}
+	}
+
+	if blockCommentLevel > 0 {
+		return fmt.Errorf("%d:%d expected block comment end ';)', but reached end of input", line, col+1)
+	}
+	if nestLevel > 0 {
+		return fmt.Errorf("%d:%d expected ')', but reached end of input", line, col+1)
+	}
+	return nil // EOF
+}
+
+// utf8Size returns the size of the UTF-8 rune based on its first byte, or zero.
+//
+// Note: We don't validate the subsequent bytes make a well-formed UTF-8 rune intentionally for performance and to keep
+// lexing allocation free. Meanwhile, the impact is that we might skip over malformed bytes.
+var utf8Size = [256]int{
+	// 1  2  3  4  5  6  7  8  9  A  B  C  D  E  F
+	0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x00-0x0F
+	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x10-0x1F
+	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x20-0x2F
+	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x30-0x3F
+	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x40-0x4F
+	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x50-0x5F
+	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x60-0x6F
+	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x70-0x7F
+	// 1  2  3  4  5  6  7  8  9  A  B  C  D  E  F
+	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x80-0x8F
+	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x90-0x9F
+	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xA0-0xAF
+	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xB0-0xBF
+	0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, // 0xC0-0xCF
+	2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, // 0xD0-0xDF
+	3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, // 0xE0-0xEF
+	4, 4, 4, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xF0-0xFF
+}
diff --git a/wasm/wat/token.go b/wasm/wat/token.go
new file mode 100644
index 00000000..ddfc73da
--- /dev/null
+++ b/wasm/wat/token.go
@@ -0,0 +1,173 @@
+package wat
+
+// token is the set of tokens defined by the WebAssembly Text Format 1.0
+// See https://www.w3.org/TR/wasm-core-1/#tokens%E2%91%A0
+type tokenType byte
+
+const (
+	tokenInvalid tokenType = iota
+	// tokenKeyword is a potentially empty sequence of idChar characters prefixed by a lowercase letter.
+	//
+	// For example, in the below, 'local.get' 'i32.const' and 'i32.lt_s' are keywords:
+	//		local.get $y
+	//		i32.const 6
+	//		i32.lt_s
+	//
+	// See https://www.w3.org/TR/wasm-core-1/#text-keyword
+	tokenKeyword
+
+	// tokenUN is an unsigned integer in decimal or hexadecimal notation, optionally separated by underscores.
+	//
+	// For example, the following tokens represent the same number: 10
+	//		(i32.const 10)
+	//		(i32.const 1_0)
+	//		(i32.const 0x0a)
+	//		(i32.const 0x0_A)
+	//
+	// See https://www.w3.org/TR/wasm-core-1/#text-int
+	tokenUN
+
+	// tokenSN is a signed integer in decimal or hexadecimal notation, optionally separated by underscores.
+	//
+	// For example, the following tokens represent the same number: 10
+	//		(i32.const +10)
+	//		(i32.const +1_0)
+	//		(i32.const +0x0a)
+	//		(i32.const +0x0_A)
+	//
+	// See https://www.w3.org/TR/wasm-core-1/#text-int
+	tokenSN
+
+	// tokenFN represents an IEEE-754 floating point number in decimal or hexadecimal notation, optionally separated by
+	// underscores. This also includes special constants for infinity ('inf') and NaN ('nan').
+	//
+	// For example, the right-hand side of the following S-expressions are all valid floating point tokens:
+	//		(f32.const +nan)
+	//		(f64.const -nan:0xfffffffffffff)
+	//		(f64.const -inf)
+	//      (f64.const +0x0.0p0)
+	//      (f32.const 0.0e0)
+	//		(i32.const +0x0_A)
+	//		(f32.const 1.e10)
+	//      (f64.const 0x1.fff_fff_fff_fff_fp+1_023)
+	//		(f64.const 1.7976931348623157e+308)
+	tokenFN
+
+	// tokenString is a UTF-8 sequence enclosed by quotation marks, representing an encoded byte string. A tokenString
+	// can contain any character except ASCII control characters, quotation marks ('"') and backslash ('\'): these must
+	// be escaped. tokenString characters correspond to UTF-8 encoding except the special case of '\hh', which allows
+	// raw bytes expressed as hexadecimal.
+	//
+	// For example, the following tokens represent the same raw bytes: 0xe298ba0a
+	//		(data (i32.const 0) "☺\n")
+	//		(data (i32.const 0) "\u{263a}\u{0a}")
+	//		(data (i32.const 0) "\e2\98\ba\0a")
+	//
+	// See https://www.w3.org/TR/wasm-core-1/#strings%E2%91%A0
+	tokenString
+
+	// tokenID is a sequence of idChar characters prefixed by a '$':
+	//
+	// For example, in the below, '$y' is an id:
+	//		local.get $y
+	//		i32.const 6
+	//		i32.lt_s
+	//
+	// See https://www.w3.org/TR/wasm-core-1/#text-id
+	tokenID
+
+	// tokenLParen is a left paren: '('
+	tokenLParen
+
+	// tokenRParen is a right paren: ')'
+	tokenRParen
+
+	// tokenReserved is a sequence of idChar characters which are neither a tokenID nor a tokenString.
+	//
+	// For example, '0$y' is a tokenReserved, because it doesn't start with a letter or '$'.
+	//
+	// See https://www.w3.org/TR/wasm-core-1/#text-reserved
+	tokenReserved
+)
+
+// tokenNames is index-coordinated with tokenType
+var tokenNames = [...]string{
+	"tokenInvalid",
+	"tokenKeyword",
+	"tokenUN",
+	"tokenSN",
+	"tokenFN",
+	"tokenString",
+	"tokenID",
+	"tokenLParen",
+	"tokenRParen",
+	"tokenReserved",
+}
+
+// String returns the string name of this token.
+func (t tokenType) String() string {
+	return tokenNames[t]
+}
+
+// constants below help format a somewhat readable lookup table that eases identification of tokens.
+const (
+	// xx is an invalid token start byte
+	xx = tokenInvalid
+	// xs is the start of tokenString ('"')
+	xs = tokenString
+	// xi is the start of tokenID ('$')
+	xi = tokenID
+	// lp is the start of tokenLParen ('(')
+	lp = tokenLParen
+	// rp is the start of tokenRParen (')')
+	rp = tokenRParen
+	// un is the start of a tokenUN (or tokenFN)
+	un = tokenUN
+	// sn is the start of a tokenSN (or tokenFN)
+	sn = tokenSN
+	// xk is the start of a tokenKeyword
+	xk = tokenKeyword
+	// xr is the start of tokenReserved (a valid, but not defined above).
+	xr = tokenReserved
+)
+
+// firstTokenByte is information about the firstTokenByte byte in a token. All expected token starts are ASCII, but we
+// switch to avoid a range check.
+var firstTokenByte = [256]tokenType{
+	//   1   2   3   4   5   6   7   8   9   A   B   C   D   E   F
+	xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0x00-0x0F
+	xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0x10-0x1F
+	xx, xr, xs, xr, xi, xr, xr, xr, lp, rp, xr, sn, xx, sn, xr, xr, // 0x20-0x2F
+	un, un, un, un, un, un, un, un, un, un, xr, xx, xr, xr, xr, xr, // 0x30-0x3F
+	xr, xr, xr, xr, xr, xr, xr, xr, xr, xr, xr, xr, xr, xr, xr, xr, // 0x40-0x4F
+	xr, xr, xr, xr, xr, xr, xr, xr, xr, xr, xr, xx, xr, xx, xr, xr, // 0x50-0x5F
+	xr, xk, xk, xk, xk, xk, xk, xk, xk, xk, xk, xk, xk, xk, xk, xk, // 0x60-0x6F
+	xk, xk, xk, xk, xk, xk, xk, xk, xk, xk, xk, xx, xr, xx, xr, xx, // 0x70-0x7F
+}
+
+// idChar is a printable ASCII character that does not contain a space, quotation mark, comma, semicolon, or bracket.
+// See https://www.w3.org/TR/wasm-core-1/#text-idchar
+var idChar = buildIdChars()
+
+func buildIdChars() (result [256]bool) {
+	for i := 0; i < 128; i++ {
+		result[i] = _idChar(byte(i))
+	}
+	return
+}
+
+func _idChar(ch byte) bool {
+	switch ch {
+	case '!', '#', '$', '%', '&', '\'', '*', '+', '-', '.', '/', ':', '<', '=', '>', '?', '@', '\\', '^', '_', '`', '|', '~':
+		return true
+	}
+	switch {
+	case ch >= '0' && ch <= '9':
+		fallthrough
+	case ch >= 'a' && ch <= 'z':
+		fallthrough
+	case ch >= 'A' && ch <= 'Z':
+		return true
+	}
+	return false
+}


Test patch:
diff --git a/wasm/wat/lexer_test.go b/wasm/wat/lexer_test.go
new file mode 100644
index 00000000..6b0e5a30
--- /dev/null
+++ b/wasm/wat/lexer_test.go
@@ -0,0 +1,571 @@
+package wat
+
+import (
+	"errors"
+	"fmt"
+	"testing"
+	"unicode/utf8"
+
+	"github.com/bytecodealliance/wasmtime-go"
+	"github.com/stretchr/testify/require"
+)
+
+// exampleWat was at one time in the wasmtime repo under cranelift. We added a unicode comment for fun!
+const exampleWat = `(module
+  ;; 私たちはフィボナッチ数列を使います。何故ならみんなやってるからです。
+  (memory 1)
+  (func $main (local i32 i32 i32 i32)
+    (set_local 0 (i32.const 0))
+    (set_local 1 (i32.const 1))
+    (set_local 2 (i32.const 1))
+    (set_local 3 (i32.const 0))
+    (block
+    (loop
+        (br_if 1 (i32.gt_s (get_local 0) (i32.const 5)))
+        (set_local 3 (get_local 2))
+        (set_local 2 (i32.add (get_local 2) (get_local 1)))
+        (set_local 1 (get_local 3))
+        (set_local 0 (i32.add (get_local 0) (i32.const 1)))
+        (br 0)
+    )
+    )
+    (i32.store (i32.const 0) (get_local 2))
+  )
+  (start $main)
+  (data (i32.const 0) "0000")
+)`
+
+// TestLex_Example is intentionally verbose to catch line/column/position bugs
+func TestLex_Example(t *testing.T) {
+	tokens, e := lexTokens(exampleWat)
+	require.NoError(t, e)
+	require.Equal(t, []*token{
+		{tokenLParen, 1, 1, 0, "("},
+		{tokenKeyword, 1, 2, 1, "module"},
+		{tokenLParen, 3, 3, 118, "("},
+		{tokenKeyword, 3, 4, 119, "memory"},
+		{tokenUN, 3, 11, 126, "1"},
+		{tokenRParen, 3, 12, 127, ")"},
+		{tokenLParen, 4, 3, 131, "("},
+		{tokenKeyword, 4, 4, 132, "func"},
+		{tokenID, 4, 9, 137, "$main"},
+		{tokenLParen, 4, 15, 143, "("},
+		{tokenKeyword, 4, 16, 144, "local"},
+		{tokenKeyword, 4, 22, 150, "i32"},
+		{tokenKeyword, 4, 26, 154, "i32"},
+		{tokenKeyword, 4, 30, 158, "i32"},
+		{tokenKeyword, 4, 34, 162, "i32"},
+		{tokenRParen, 4, 37, 165, ")"},
+		{tokenLParen, 5, 5, 171, "("},
+		{tokenKeyword, 5, 6, 172, "set_local"},
+		{tokenUN, 5, 16, 182, "0"},
+		{tokenLParen, 5, 18, 184, "("},
+		{tokenKeyword, 5, 19, 185, "i32.const"},
+		{tokenUN, 5, 29, 195, "0"},
+		{tokenRParen, 5, 30, 196, ")"},
+		{tokenRParen, 5, 31, 197, ")"},
+		{tokenLParen, 6, 5, 203, "("},
+		{tokenKeyword, 6, 6, 204, "set_local"},
+		{tokenUN, 6, 16, 214, "1"},
+		{tokenLParen, 6, 18, 216, "("},
+		{tokenKeyword, 6, 19, 217, "i32.const"},
+		{tokenUN, 6, 29, 227, "1"},
+		{tokenRParen, 6, 30, 228, ")"},
+		{tokenRParen, 6, 31, 229, ")"},
+		{tokenLParen, 7, 5, 235, "("},
+		{tokenKeyword, 7, 6, 236, "set_local"},
+		{tokenUN, 7, 16, 246, "2"},
+		{tokenLParen, 7, 18, 248, "("},
+		{tokenKeyword, 7, 19, 249, "i32.const"},
+		{tokenUN, 7, 29, 259, "1"},
+		{tokenRParen, 7, 30, 260, ")"},
+		{tokenRParen, 7, 31, 261, ")"},
+		{tokenLParen, 8, 5, 267, "("},
+		{tokenKeyword, 8, 6, 268, "set_local"},
+		{tokenUN, 8, 16, 278, "3"},
+		{tokenLParen, 8, 18, 280, "("},
+		{tokenKeyword, 8, 19, 281, "i32.const"},
+		{tokenUN, 8, 29, 291, "0"},
+		{tokenRParen, 8, 30, 292, ")"},
+		{tokenRParen, 8, 31, 293, ")"},
+		{tokenLParen, 9, 5, 299, "("},
+		{tokenKeyword, 9, 6, 300, "block"},
+		{tokenLParen, 10, 5, 310, "("},
+		{tokenKeyword, 10, 6, 311, "loop"},
+		{tokenLParen, 11, 9, 324, "("},
+		{tokenKeyword, 11, 10, 325, "br_if"},
+		{tokenUN, 11, 16, 331, "1"},
+		{tokenLParen, 11, 18, 333, "("},
+		{tokenKeyword, 11, 19, 334, "i32.gt_s"},
+		{tokenLParen, 11, 28, 343, "("},
+		{tokenKeyword, 11, 29, 344, "get_local"},
+		{tokenUN, 11, 39, 354, "0"},
+		{tokenRParen, 11, 40, 355, ")"},
+		{tokenLParen, 11, 42, 357, "("},
+		{tokenKeyword, 11, 43, 358, "i32.const"},
+		{tokenUN, 11, 53, 368, "5"},
+		{tokenRParen, 11, 54, 369, ")"},
+		{tokenRParen, 11, 55, 370, ")"},
+		{tokenRParen, 11, 56, 371, ")"},
+		{tokenLParen, 12, 9, 381, "("},
+		{tokenKeyword, 12, 10, 382, "set_local"},
+		{tokenUN, 12, 20, 392, "3"},
+		{tokenLParen, 12, 22, 394, "("},
+		{tokenKeyword, 12, 23, 395, "get_local"},
+		{tokenUN, 12, 33, 405, "2"},
+		{tokenRParen, 12, 34, 406, ")"},
+		{tokenRParen, 12, 35, 407, ")"},
+		{tokenLParen, 13, 9, 417, "("},
+		{tokenKeyword, 13, 10, 418, "set_local"},
+		{tokenUN, 13, 20, 428, "2"},
+		{tokenLParen, 13, 22, 430, "("},
+		{tokenKeyword, 13, 23, 431, "i32.add"},
+		{tokenLParen, 13, 31, 439, "("},
+		{tokenKeyword, 13, 32, 440, "get_local"},
+		{tokenUN, 13, 42, 450, "2"},
+		{tokenRParen, 13, 43, 451, ")"},
+		{tokenLParen, 13, 45, 453, "("},
+		{tokenKeyword, 13, 46, 454, "get_local"},
+		{tokenUN, 13, 56, 464, "1"},
+		{tokenRParen, 13, 57, 465, ")"},
+		{tokenRParen, 13, 58, 466, ")"},
+		{tokenRParen, 13, 59, 467, ")"},
+		{tokenLParen, 14, 9, 477, "("},
+		{tokenKeyword, 14, 10, 478, "set_local"},
+		{tokenUN, 14, 20, 488, "1"},
+		{tokenLParen, 14, 22, 490, "("},
+		{tokenKeyword, 14, 23, 491, "get_local"},
+		{tokenUN, 14, 33, 501, "3"},
+		{tokenRParen, 14, 34, 502, ")"},
+		{tokenRParen, 14, 35, 503, ")"},
+		{tokenLParen, 15, 9, 513, "("},
+		{tokenKeyword, 15, 10, 514, "set_local"},
+		{tokenUN, 15, 20, 524, "0"},
+		{tokenLParen, 15, 22, 526, "("},
+		{tokenKeyword, 15, 23, 527, "i32.add"},
+		{tokenLParen, 15, 31, 535, "("},
+		{tokenKeyword, 15, 32, 536, "get_local"},
+		{tokenUN, 15, 42, 546, "0"},
+		{tokenRParen, 15, 43, 547, ")"},
+		{tokenLParen, 15, 45, 549, "("},
+		{tokenKeyword, 15, 46, 550, "i32.const"},
+		{tokenUN, 15, 56, 560, "1"},
+		{tokenRParen, 15, 57, 561, ")"},
+		{tokenRParen, 15, 58, 562, ")"},
+		{tokenRParen, 15, 59, 563, ")"},
+		{tokenLParen, 16, 9, 573, "("},
+		{tokenKeyword, 16, 10, 574, "br"},
+		{tokenUN, 16, 13, 577, "0"},
+		{tokenRParen, 16, 14, 578, ")"},
+		{tokenRParen, 17, 5, 584, ")"},
+		{tokenRParen, 18, 5, 590, ")"},
+		{tokenLParen, 19, 5, 596, "("},
+		{tokenKeyword, 19, 6, 597, "i32.store"},
+		{tokenLParen, 19, 16, 607, "("},
+		{tokenKeyword, 19, 17, 608, "i32.const"},
+		{tokenUN, 19, 27, 618, "0"},
+		{tokenRParen, 19, 28, 619, ")"},
+		{tokenLParen, 19, 30, 621, "("},
+		{tokenKeyword, 19, 31, 622, "get_local"},
+		{tokenUN, 19, 41, 632, "2"},
+		{tokenRParen, 19, 42, 633, ")"},
+		{tokenRParen, 19, 43, 634, ")"},
+		{tokenRParen, 20, 3, 638, ")"},
+		{tokenLParen, 21, 3, 642, "("},
+		{tokenKeyword, 21, 4, 643, "start"},
+		{tokenID, 21, 10, 649, "$main"},
+		{tokenRParen, 21, 15, 654, ")"},
+		{tokenLParen, 22, 3, 658, "("},
+		{tokenKeyword, 22, 4, 659, "data"},
+		{tokenLParen, 22, 9, 664, "("},
+		{tokenKeyword, 22, 10, 665, "i32.const"},
+		{tokenUN, 22, 20, 675, "0"},
+		{tokenRParen, 22, 21, 676, ")"},
+		{tokenString, 22, 23, 678, "\"0000\""},
+		{tokenRParen, 22, 29, 684, ")"},
+		{tokenRParen, 23, 1, 686, ")"},
+	}, tokens)
+}
+
+func TestLex(t *testing.T) {
+	tests := []struct {
+		name     string
+		input    string
+		expected []*token
+	}{
+		{
+			name:  "empty",
+			input: "",
+		},
+		{
+			name:     "only parens",
+			input:    "()",
+			expected: []*token{{tokenLParen, 1, 1, 0, "("}, {tokenRParen, 1, 2, 1, ")"}},
+		},
+		{
+			name:     "shortest keywords",
+			input:    "a z",
+			expected: []*token{{tokenKeyword, 1, 1, 0, "a"}, {tokenKeyword, 1, 3, 2, "z"}},
+		},
+		{
+			name:     "shortest tokens - EOL",
+			input:    "(a)\n",
+			expected: []*token{{tokenLParen, 1, 1, 0, "("}, {tokenKeyword, 1, 2, 1, "a"}, {tokenRParen, 1, 3, 2, ")"}},
+		},
+		{
+			name:     "only tokens",
+			input:    "(module)",
+			expected: []*token{{tokenLParen, 1, 1, 0, "("}, {tokenKeyword, 1, 2, 1, "module"}, {tokenRParen, 1, 8, 7, ")"}},
+		},
+		{
+			name:  "only white space characters",
+			input: " \t\r\n",
+		},
+		{
+			name:     "after white space characters - EOL",
+			input:    " \t\na",
+			expected: []*token{{tokenKeyword, 2, 1, 3, "a"}},
+		},
+		{
+			name:     "after white space characters - Windows EOL",
+			input:    " \t\r\na",
+			expected: []*token{{tokenKeyword, 2, 1, 4, "a"}},
+		},
+		{
+			name:  "only line comment - EOL before EOF",
+			input: ";; TODO\n",
+		},
+		{
+			name:  "only line comment - EOF",
+			input: ";; TODO",
+		},
+		{
+			name:  "only unicode line comment - EOF",
+			input: ";; брэд-ЛГТМ",
+		},
+		{
+			name:     "after line comment",
+			input:    ";; TODO\na",
+			expected: []*token{{tokenKeyword, 2, 1, 8, "a"}},
+		},
+		{
+			name:     "double line comment",
+			input:    ";; TODO\n;; YOLO\na",
+			expected: []*token{{tokenKeyword, 3, 1, 16, "a"}},
+		},
+		{
+			name:     "after unicode line comment",
+			input:    ";; брэд-ЛГТМ\na",
+			expected: []*token{{tokenKeyword, 2, 1, 21, "a"}},
+		},
+		{
+			name:     "after line comment - Windows EOL",
+			input:    ";; TODO\r\na",
+			expected: []*token{{tokenKeyword, 2, 1, 9, "a"}},
+		},
+		{
+			name:     "after redundant line comment",
+			input:    ";;;; TODO\na",
+			expected: []*token{{tokenKeyword, 2, 1, 10, "a"}},
+		},
+		{
+			name:     "after line commenting out block comment",
+			input:    ";; TODO (; ;)\na",
+			expected: []*token{{tokenKeyword, 2, 1, 14, "a"}},
+		},
+		{
+			name:     "after line commenting out open block comment",
+			input:    ";; TODO (;\na",
+			expected: []*token{{tokenKeyword, 2, 1, 11, "a"}},
+		},
+		{
+			name:     "after line commenting out close block comment",
+			input:    ";; TODO ;)\na",
+			expected: []*token{{tokenKeyword, 2, 1, 11, "a"}},
+		},
+		{
+			name:  "only block comment - EOL before EOF",
+			input: "(; TODO ;)\n",
+		},
+		{
+			name:  "only block comment - Windows EOL before EOF",
+			input: "(; TODO ;)\r\n",
+		},
+		{
+			name:  "only block comment - EOF",
+			input: "(; TODO ;)",
+		},
+		{
+			name:     "double block comment",
+			input:    "(; TODO ;)(; YOLO ;)a",
+			expected: []*token{{tokenKeyword, 1, 21, 20, "a"}},
+		},
+		{
+			name:     "double block comment - EOL",
+			input:    "(; TODO ;)\n(; YOLO ;)\na",
+			expected: []*token{{tokenKeyword, 3, 1, 22, "a"}},
+		},
+		{
+			name:     "after block comment",
+			input:    "(; TODO ;)a",
+			expected: []*token{{tokenKeyword, 1, 11, 10, "a"}},
+		},
+		{
+			name:  "only nested block comment - EOL before EOF",
+			input: "(; TODO (; (YOLO) ;) ;)\n",
+		},
+		{
+			name:  "only nested block comment - EOF",
+			input: "(; TODO (; (YOLO) ;) ;)",
+		},
+		{
+			name:  "only unicode block comment - EOF",
+			input: "(; брэд-ЛГТМ ;)",
+		},
+		{
+			name:     "after nested block comment",
+			input:    "(; TODO (; (YOLO) ;) ;)a",
+			expected: []*token{{tokenKeyword, 1, 24, 23, "a"}},
+		},
+		{
+			name:     "after nested block comment - EOL",
+			input:    "(; TODO (; (YOLO) ;) ;)\n a",
+			expected: []*token{{tokenKeyword, 2, 2, 25, "a"}},
+		},
+		{
+			name:     "after nested block comment - Windows EOL",
+			input:    "(; TODO (; (YOLO) ;) ;)\r\n a",
+			expected: []*token{{tokenKeyword, 2, 2, 26, "a"}},
+		},
+		{
+			name:     "white space between parens",
+			input:    "( )",
+			expected: []*token{{tokenLParen, 1, 1, 0, "("}, {tokenRParen, 1, 3, 2, ")"}},
+		},
+		{
+			name:  "nested parens",
+			input: "(())",
+			expected: []*token{
+				{tokenLParen, 1, 1, 0, "("},
+				{tokenLParen, 1, 2, 1, "("},
+				{tokenRParen, 1, 3, 2, ")"},
+				{tokenRParen, 1, 4, 3, ")"},
+			},
+		},
+		{
+			name:     "empty string",
+			input:    `""`,
+			expected: []*token{{tokenString, 1, 1, 0, `""`}},
+		},
+		{
+			name:     "unicode string",
+			input:    "\"брэд-ЛГТМ\"",
+			expected: []*token{{tokenString, 1, 1, 0, "\"брэд-ЛГТМ\""}},
+		},
+		{
+			name:     "string inside tokens with newline",
+			input:    "(\"\n\")", // TODO newline char isn't actually allowed unless escaped!
+			expected: []*token{{tokenLParen, 1, 1, 0, "("}, {tokenString, 1, 2, 1, "\"\n\""}, {tokenRParen, 1, 5, 4, ")"}},
+		},
+		{
+			name:     "unsigned shortest - EOL",
+			input:    "1\n",
+			expected: []*token{{tokenUN, 1, 1, 0, "1"}},
+		},
+		{
+			name:     "unsigned shortest - EOF",
+			input:    "1",
+			expected: []*token{{tokenUN, 1, 1, 0, "1"}},
+		},
+		{
+			name:     "unsigned shortest inside tokens",
+			input:    "(1)",
+			expected: []*token{{tokenLParen, 1, 1, 0, "("}, {tokenUN, 1, 2, 1, "1"}, {tokenRParen, 1, 3, 2, ")"}},
+		},
+		{
+			name:     "unsigned shortest then string",
+			input:    `1"1"`,
+			expected: []*token{{tokenUN, 1, 1, 0, "1"}, {tokenString, 1, 2, 1, `"1"`}},
+		},
+		{
+			name:     "unsigned - EOL",
+			input:    "123\n",
+			expected: []*token{{tokenUN, 1, 1, 0, "123"}},
+		},
+		{
+			name:     "unsigned - EOF",
+			input:    "123",
+			expected: []*token{{tokenUN, 1, 1, 0, "123"}},
+		},
+		{
+			name:     "unsigned inside tokens",
+			input:    "(123)",
+			expected: []*token{{tokenLParen, 1, 1, 0, "("}, {tokenUN, 1, 2, 1, "123"}, {tokenRParen, 1, 5, 4, ")"}},
+		},
+		{
+			name:     "unsigned then string",
+			input:    `123"123"`,
+			expected: []*token{{tokenUN, 1, 1, 0, "123"}, {tokenString, 1, 4, 3, `"123"`}},
+		},
+		{
+			name:     "unsigned then keyword",
+			input:    "1a", // whitespace is optional between tokens, and a keyword can be single-character!
+			expected: []*token{{tokenUN, 1, 1, 0, "1"}, {tokenKeyword, 1, 2, 1, "a"}},
+		},
+	}
+
+	for _, tt := range tests {
+		tc := tt
+		t.Run(tc.name, func(t *testing.T) {
+			tokens, e := lexTokens(tc.input)
+			require.NoError(t, e)
+			require.Equal(t, tc.expected, tokens)
+		})
+	}
+}
+
+func TestLex_Errors(t *testing.T) {
+	tests := []struct {
+		name        string
+		input       []byte
+		expectedErr error
+	}{
+		{
+			name:        "close paren before open paren",
+			input:       []byte(")("),
+			expectedErr: errors.New("1:1 found ')' before '('"),
+		},
+		{
+			name:        "unbalanced nesting",
+			input:       []byte("(()"),
+			expectedErr: errors.New("1:4 expected ')', but reached end of input"),
+		},
+		{
+			name:        "open paren at end of input",
+			input:       []byte("("),
+			expectedErr: errors.New("1:1 found '(' at end of input"),
+		},
+		{
+			name:        "begin block comment at end of input",
+			input:       []byte("(;"),
+			expectedErr: errors.New("1:3 expected block comment end ';)', but reached end of input"),
+		},
+		{
+			name:        "half line comment",
+			input:       []byte("; TODO"),
+			expectedErr: errors.New("1:1 unexpected character ;"),
+		},
+		{
+			name:        "open block comment",
+			input:       []byte("(; TODO"),
+			expectedErr: errors.New("1:8 expected block comment end ';)', but reached end of input"),
+		},
+		{
+			name:        "close block comment",
+			input:       []byte(";) TODO"),
+			expectedErr: errors.New("1:1 unexpected character ;"),
+		},
+		{
+			name:        "unbalanced nested block comment",
+			input:       []byte("(; TODO (; (YOLO) ;)"),
+			expectedErr: errors.New("1:21 expected block comment end ';)', but reached end of input"),
+		},
+		{
+			name:        "dangling unicode",
+			input:       []byte(" 私"),
+			expectedErr: errors.New("1:2 expected an ASCII character, not 私"),
+		},
+		{
+			name:        "NULL in block comment",
+			input:       append([]byte("(; "), 0),
+			expectedErr: errors.New("1:4 found an invalid byte in block comment: 0x0"),
+		},
+		{
+			name:        "NULL in block comment unicode",
+			input:       append([]byte("(; "), 0),
+			expectedErr: errors.New("1:4 found an invalid byte in block comment: 0x0"),
+		},
+		{
+			name:        "NULL in line comment",
+			input:       append([]byte(";; "), 0),
+			expectedErr: errors.New("1:4 found an invalid byte in line comment: 0x0"),
+		},
+		{
+			name:        "NULL in string",
+			input:       append([]byte("\""), 0),
+			expectedErr: errors.New("1:2 found an invalid byte in string token: 0x0"),
+		},
+	}
+
+	for _, tt := range tests {
+		tc := tt
+		t.Run(tc.name, func(t *testing.T) {
+			require.Equal(t, lex(tc.input, noopParseToken), tc.expectedErr)
+		})
+	}
+}
+
+func lexTokens(input string) ([]*token, error) {
+	var tokens []*token
+	e := lex([]byte(input), func(source []byte, tok tokenType, line, col, beginPos, endPos int) (err error) {
+		tokens = append(tokens, &token{tok, line, col, beginPos, string(source[beginPos:endPos])})
+		return
+	})
+	return tokens, e
+}
+
+var noopParseToken parseToken = func(source []byte, tok tokenType, beginLine, beginCol, beginPos, endPos int) error {
+	return nil
+}
+
+func BenchmarkLex(b *testing.B) {
+	benchmarks := []struct {
+		name string
+		data []byte
+	}{
+		{"example", []byte(exampleWat)},
+		{"whitespace chars", []byte("(                        \nmodule)\n")}, // 34 bytes
+		{"unicode line comment", []byte("( ;; брэд-ЛГТМ   \nmodule)\n")},     // 28 bytes
+		{"unicode block comment", []byte("( (; брэд-ЛГТМ ;)\nmodule)\n")},    // 28 bytes
+	}
+
+	for _, bm := range benchmarks {
+		b.Run(bm.name+" vs utf8.ValidString", func(b *testing.B) {
+			b.ReportAllocs()
+			for i := 0; i < b.N; i++ {
+				utf8.ValidString(string(bm.data))
+			}
+		})
+		// Not a fair comparison as we are only lexing and not writing back %.wasm
+		// If possible, we should find a way to isolate only the lexing C functions.
+		b.Run(bm.name+" vs wasmtime.Wat2Wasm", func(b *testing.B) {
+			b.ReportAllocs()
+			for i := 0; i < b.N; i++ {
+				_, err := wasmtime.Wat2Wasm(string(bm.data))
+				if err != nil {
+					panic(err)
+				}
+			}
+		})
+		b.Run(bm.name, func(b *testing.B) {
+			b.ReportAllocs()
+			for i := 0; i < b.N; i++ {
+				err := lex(bm.data, noopParseToken)
+				if err != nil {
+					panic(err)
+				}
+			}
+		})
+	}
+}
+
+type token struct {
+	tokenType
+	line, col, pos int
+	value          string
+}
+
+// String helps format to allow copy/pasting of expected values
+func (t *token) String() string {
+	return fmt.Sprintf("{%s, %d, %d, %d, %q}", t.tokenType, t.line, t.col, t.pos, t.value)
+}
diff --git a/wasm/wat/token_test.go b/wasm/wat/token_test.go
new file mode 100644
index 00000000..a8aa111e
--- /dev/null
+++ b/wasm/wat/token_test.go
@@ -0,0 +1,31 @@
+package wat
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestTokenType_String(t *testing.T) {
+	tests := []struct {
+		input    tokenType
+		expected string
+	}{
+		{tokenKeyword, "tokenKeyword"},
+		{tokenUN, "tokenUN"},
+		{tokenSN, "tokenSN"},
+		{tokenFN, "tokenFN"},
+		{tokenString, "tokenString"},
+		{tokenID, "tokenID"},
+		{tokenLParen, "tokenLParen"},
+		{tokenRParen, "tokenRParen"},
+		{tokenReserved, "tokenReserved"},
+	}
+
+	for _, tt := range tests {
+		tc := tt
+		t.Run(tc.expected, func(t *testing.T) {
+			require.Equal(t, tc.expected, tc.input.String())
+		})
+	}
+}