rodjek/puppet-lint — pull request #317 by rodjek (2014-09-18 22:17:29+00:00)
Title: Cache parsing state in Lexer rather than recalculating

Reported in #315, puppet-lint is painfully slow when processing large files. I
reproduced this by creating a very large manifest (basically the same block of
code concatenated over and over).

```
$ ls -la ~/test.pp
-rw-r--r-- 1 tsharpe staff 165928 18 Sep 14:23 /Users/tsharpe/test.pp
$ wc -l ~/test.pp
6296 /Users/tsharpe/test.pp
```

On 1.0.1, running the full suite of checks:
```
$ time bin/puppet-lint ~/test.pp
bin/puppet-lint ~/test.pp > 1.0.1.output 169.96s user 0.26s system 99% cpu 2:50.26 total
```

Basically, the problem was that every time the lexer was creating a new token
it would search through every byte of the file tokenised so far to count how
many newlines had been found in order to add the line number to the Tokens. By
changing the lexer so that it now keeps a counter which it increments everytime
a :NEWLINE Token is created, we can avoid all this stupidity (keeping track of
the state in regards to column number also provided a small win, but not as
large as line number).

On this branch, running the full suite of checks:
```
$ time bin/puppet-lint ~/test.pp
bin/puppet-lint ~/test.pp > 315.output 9.43s user 0.08s system 99% cpu 9.513 total
```

Closes #315