17: allow leading newlines and remove MAIN state
17: leave recognition of double-quotes to parser
17: use a single regexp to recognize WORD tokens
17: factor out tokenizing
Merge pull request #33 from euromake/bug17

Tokenize only non-recursive constructs