bentley/rgbds — pull request #34 by chastai (2014-10-10 14:59:36+00:00)
Title: fix a bug in the lexer involving double quote escaping and semicolons

The bug showed up when a semicolon was located anywhere after \".

These three test cases are syntaxically correct but didn't compile:

1)
SECTION "HOME", HOME
db "\";"

2)
SECTION "HOME", HOME
db "\""
nop
;

3)
SECTION "HOME", HOME
db "\"" ;

The problem was located in yy_create_buffer(). Basicaly, this function loads an
entire source file, uniformizes EOL terminators and filters out comments without
touching literal strings.

However, bounds of literal strings were wrongly guessed because \" was
interpreted as two characters (and so the double quote was not escaped).

In test 1, the string terminates early and so ;" is filtered out as it was a
comment and so the assembler complains of an unterminated string.
In test 2 and 3, the string is in fact interpreted as two strings, the second
one terminates at EOF in these cases and so comments are not filtered out and
that makes the assembler complains.

A special case must be taken into account:

4)
SECTION "HOME", HOME
db "\\\\" ;

So we need to ignore \\\\ as well.

Note that there is still a problem left: in yy_create_buffer() a string may
span multiple lines but not in the lexer. However in this case I think the lexer
would quit at the first newline so there should be nothing to worry about.