erlang/otp — pull request #431 by archimed-shaman (2014-07-15 09:10:24+00:00)
Title: Leex line wrap issue

Looks like there is a bug with line counting in parsetools. To reproduce it, create test.xrl:
```
Definitions.
Rules.
[a]+[\n]*= : {token, {first, TokenLine}}.
[a]+ : {token, {second, TokenLine}}.
[\s\r\n\t]+ : skip_token.
Erlang code.
```
and run:
```
[archimed@pc leex_test]$ erl
Erlang/OTP 17 [erts-6.1] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V6.1 (abort with ^G)
1> leex:file(test).
{ok,"./test.erl"}
2> c(test).
{ok,test}
3> test:string("aaaa\naa").
{ok,[{second,1},{second,3}],3}
```
Parser detects three lines instead of two in the input
```
aaaa
aa
```
The only cause, I found [here](http://erlang.org/doc/man/leex.html), is pushing back an newline:
`Note that pushing back a newline will mean the line numbering will no longer be correct.`
However, there is no any character back-off.
In my opinion, this issue appears, because the parsing FSM does not rollback the line counter after attempting the first rule "[a]+[\n]*=".
I'm not sure, the patch will not break something else :) But the existing unit tests are passed, so I hope, it would be useful.

Here is the dbg trace. You can notice the moment, when the line counter becomes invalid.
```
4> dbg:tracer().
{ok,<0.46.0>}
5> dbg:p(all, c).
{ok,[{matched,nonode@nohost,26}]}
6> dbg:tpl(test, dbg:fun2ms(fun(_) -> return_trace() end)).
{ok,[{matched,nonode@nohost,25},{saved,1}]}
7> test:string("aaaa\naa").
(<0.33.0>) call test:string("aaaa\naa")
(<0.33.0>) call test:string("aaaa\naa",1)
(<0.33.0>) call test:string("aaaa\naa",1,"aaaa\naa",[])
(<0.33.0>) call test:yystate()
(<0.33.0>) returned from test:yystate/0 -> 3
(<0.33.0>) call test:yystate(3,"aaaa\naa",1,0,reject,0)
(<0.33.0>) call test:yystate(1,"aaa\naa",1,1,reject,0)
(<0.33.0>) call test:yystate(1,"aa\naa",1,2,1,1)
(<0.33.0>) call test:yystate(1,"a\naa",1,3,1,2)
(<0.33.0>) call test:yystate(1,"\naa",1,4,1,3)
(<0.33.0>) call test:yystate(2,"aa",2,5,1,4)
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) call test:yysuf("aaaa\naa",4)
(<0.33.0>) returned from test:yysuf/2 -> "\naa"
(<0.33.0>) call test:yyaction(1,4,"aaaa\naa",1)
(<0.33.0>) returned from test:yyaction/4 -> {token,{second,1}}
(<0.33.0>) call test:string_cont("\naa",2,{token,{second,1}},[])
(<0.33.0>) call test:string("\naa",2,"\naa",[{second,1}])
(<0.33.0>) call test:yystate()
(<0.33.0>) returned from test:yystate/0 -> 3
(<0.33.0>) call test:yystate(3,"\naa",2,0,reject,0)
(<0.33.0>) call test:yystate(4,"aa",3,1,reject,0)
(<0.33.0>) returned from test:yystate/6 -> {2,1,"aa",3,4}
(<0.33.0>) returned from test:yystate/6 -> {2,1,"aa",3,4}
(<0.33.0>) call test:yyaction(2,1,"\naa",2)
(<0.33.0>) returned from test:yyaction/4 -> skip_token
(<0.33.0>) call test:string_cont("aa",3,skip_token,[{second,1}])
(<0.33.0>) call test:string("aa",3,"aa",[{second,1}])
(<0.33.0>) call test:yystate()
(<0.33.0>) returned from test:yystate/0 -> 3
(<0.33.0>) call test:yystate(3,"aa",3,0,reject,0)
(<0.33.0>) call test:yystate(1,"a",3,1,reject,0)
(<0.33.0>) call test:yystate(1,[],3,2,1,1)
(<0.33.0>) returned from test:yystate/6 -> {1,2,[],3,1}
(<0.33.0>) returned from test:yystate/6 -> {1,2,[],3,1}
(<0.33.0>) returned from test:yystate/6 -> {1,2,[],3,1}
(<0.33.0>) call test:yyaction(1,2,"aa",3)
(<0.33.0>) returned from test:yyaction/4 -> {token,{second,3}}
(<0.33.0>) call test:string_cont([],3,{token,{second,3}},[{second,1}])
(<0.33.0>) call test:string([],3,[],[{second,3},{second,1}])
(<0.33.0>) call test:yyrev([{second,3},{second,1}])
(<0.33.0>) returned from test:yyrev/1 -> [{second,1},{second,3}]
(<0.33.0>) returned from test:string/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string_cont/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string_cont/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string_cont/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/2 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/1 -> {ok,[{second,1},{second,3}],3}
{ok,[{second,1},{second,3}],3}
```