Repository: benhoyt/goawk (go)

Issue:
Semicolon not allowed as terminator for normal pattern without action
Hi, I've studied the Posix awk spec (same as linked in the readme) because I noticed a parsing difference between awk and gawk. This lead me to test other implementations as well, and I noticed a bug in GoAWK's parsing.

GoAWK fails to accept semicolon as the terminator for a normal pattern without action, this is allowed by the Posix grammar and also accepted by other major awk implementations.

Example of a failing program:

```awk
normal_pattern ; item
```

The program should work the same way it does when the two items are on separate lines, but instead it gives the following parse error:

```
<cmdline>:1:16: expected , instead of ;
normal_pattern ; item
               ^
```

I've implemented a possible fix for this and could submit a PR if you wish.

<details>
<summary>The relevant parts of the Posix grammar</summary>

```
program          : item_list
                 | item_list item
                 ;

item_list        : /* empty */
                 | item_list item terminator
                 ;

item             : action
                 | pattern action
                 | normal_pattern
                 | ...
                 ;

terminator       : terminator NEWLINE
                 |            ';'
                 |            NEWLINE
                 ;
```
</details>

<details>
<summary>And the original difference that got me here with some pondering about the posix spec...</summary>

First I noticed that awk accepts the following code, but gawk (and mawk) produces a syntax error at `function`:

```awk
normal_pattern function name() {}
```

From this, I deduced that awk does not require the terminator between items at any point, but I wanted to see what would be correct according to Posix.

After reading the spec, however, it seems to me that a terminator is required between any two items.
But all major implementations seem to treat the terminator as optional when the previous item ends with a closing brace. In my opinion this is a bug in either my reading comprehension or in the Posix spec, and the status quo of the implementations is the way to go.

The only possibly applicable exception mentioned in the text (that I could find) was about function definitions, not actions in general:
> In the function definition, <newline> characters shall be optional before the opening brace and after the closing brace.
</details>

Pull request description:
Allow semicolon as terminator for an item
Fixes #145

Regarding the exception 2. this implementation relies on `p.stmtsBrace()` to consume the possible semicolon after the items ending in closing brace. But if that ever changes, the test cases should fail.

ps. sorry minor branch mistake, second try

Patch:
diff --git a/parser/parser.go b/parser/parser.go
index 491b346..b9fe681 100644
--- a/parser/parser.go
+++ b/parser/parser.go
@@ -150,9 +150,30 @@ type parser struct {
 // Parse an entire AWK program.
 func (p *parser) program() *Program {
 	prog := &Program{}
-	p.optionalNewlines()
+
+	// Terminator "(SEMICOLON|NEWLINE) NEWLINE*" is required after each item
+	// with two exceptions where it is optional:
+	//
+	// 1. after the last item, or
+	// 2. when the previous item ended with a closing brace.
+	//
+	// NOTE: The second exception does not seem to be correct according to
+	// the POSIX grammar definition, but it is the common behaviour for the
+	// major AWK implementations.
+	needsTerminator := false
+
 	for p.tok != EOF {
+		if needsTerminator {
+			if !p.matches(NEWLINE, SEMICOLON) {
+				panic(p.errorf("expected ; or newline between items"))
+			}
+			p.next()
+			needsTerminator = false
+		}
+		p.optionalNewlines()
 		switch p.tok {
+		case EOF:
+			break
 		case BEGIN:
 			p.next()
 			prog.Begin = append(prog.Begin, p.stmtsBrace())
@@ -170,7 +191,7 @@ func (p *parser) program() *Program {
 			if !p.matches(LBRACE, EOF) {
 				pattern = append(pattern, p.expr())
 			}
-			if !p.matches(LBRACE, EOF, NEWLINE) {
+			if !p.matches(LBRACE, EOF, NEWLINE, SEMICOLON) {
 				p.commaNewlines()
 				pattern = append(pattern, p.expr())
 			}
@@ -178,11 +199,12 @@ func (p *parser) program() *Program {
 			action := ast.Action{pattern, nil}
 			if p.tok == LBRACE {
 				action.Stmts = p.stmtsBrace()
+			} else {
+				needsTerminator = true
 			}
 			prog.Actions = append(prog.Actions, action)
 			p.inAction = false
 		}
-		p.optionalNewlines()
 	}
 
 	p.resolveUserCalls(prog)


Test patch:
diff --git a/interp/interp_test.go b/interp/interp_test.go
index f035de2..13de6ff 100644
--- a/interp/interp_test.go
+++ b/interp/interp_test.go
@@ -808,6 +808,15 @@ BEGIN { foo(5); bar(10) }
 	{`BEGIN { print("x" > "out") }`, "", "1\n", "", ""},
 	{`BEGIN { printf("x" > "out") }`, "", "1", "", ""},
 
+	// Semicolon is allowed as item terminator
+	{"function f(){} ; 0", "", "", "", ""}, // after function
+	{"{}             ; 0", "", "", "", ""}, // after action
+	{"1              ; 0", "", "", "", ""}, // after normal pattern without action
+	// also after the last item
+	{"function f(){} ;  ", "", "", "", ""},
+	{"{}             ;  ", "", "", "", ""},
+	{"1              ;  ", "", "", "", ""},
+
 	// Grammar should allow blocks wherever statements are allowed
 	{`BEGIN { if (1) printf "x"; else printf "y" }`, "", "x", "", ""},
 	{`BEGIN { printf "x"; { printf "y"; printf "z" } }`, "", "xyz", "", ""},