First cut on formatting compiler directives

There is a weird discrepancy between executing in fsi and in unit tests
Use pre- and post- conditions to attach directives
Add a simple repro of the lexer bug
Tokenize function was fixed

It turns out EOL token causes the issue
Remove unused FSharpLexerTests project