temoto/robotstxt.go — pull request #8 by mynameisfiber (2012-10-02 16:02:55+00:00)
Title: Tiny optimization in scanner.go

The tokenizer was calling `isSpace` quite often which was using `strings.Index` to see if the current `rune` was a whitespace character or not. This involved a type cast and many string comparisons... the attached code simply keeps things in rune space (thus avoiding a memory allocation at each comparison) and is testing comparisons based on frequency of use (so the natural space is the first comparison, then the horizontal tab, then vertical).

Anyways, this small changed sped up the `(*ByteScanner).Scan` operation by about 30%!