c compiler makes a difference!

tl/dr: icc,gcc > clang

Benchmarks on a fresh iMac

    Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz

with different compilers:

    Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
    gcc version 4.9.1 (Homebrew gcc 4.9.1)
    icl version 15.0.0 (clang version 3.4.0 compatibility)

One thing that wasn't clear previously is why the generated c code was
much slower (30%) than the hand written version. Curiously, this was only
the case on laptops, and not on lampsrv10. The bottom line is that gcc and
icc do a better job on the code we generate than clang, which is the default
on Mac OS.

            Generated  Hand-Written
    clang     2.279       1.690
    gcc       1.759       1.676
    icc       1.797       1.801
    icc       1.775       1.762   -fno-alias -xHost

Looking at the generated code, one obvious difference is using `memcmp`
(hand written) versus while loops (generated). This turned out to be
insignificant, as most of the time is spent tokenizing. The difference
there is between array indexing and pointer increments.

Generated:

    int32_t x9 = x5;
    for (;;) {
      int32_t x10 = x5;
      char x11 = x4[x10];
      bool x12 = x11 != '\t';
      if (!x12) break;
      x5 += 1;
    }
    int32_t x17 = x5;
    x5 += 1;

Hand written:

    char* phrase_start = cur_line;
    char* phrase_end = phrase_start;
    while (*phrase_end++ != '\t');

While one would hope that compilers do enough strength reduction to produce
essentially equivalent assembly for the two versions, this is not always
the case: clang indeed emits an indirect comparison.

From generated:

    LBB4_3:                                 ##   Parent Loop BB4_2 Depth=1
                                            ## =>  This Inner Loop Header: Depth=2
      movq  %rdx, %r8
      movl  %ebx, %ecx
      movl  %esi, %eax
      leaq  1(%r8), %rdx
      leal  1(%rcx), %ebx
      leal  1(%rax), %esi
      cmpb  $9, (%rdi,%r14)
      leaq  1(%r14), %r14
      jne LBB4_3
    ## BB#4:                                ## %.loopexit
                                            ##   in Loop: Header=BB4_2 Depth=1
      movslq  %r12d, %rbx
      addq  %r14, %rbx

From hand written:

    LBB0_13:                                ##   Parent Loop BB0_10 Depth=1
                                            ## =>  This Inner Loop Header: Depth=2
      cmpb  $9, (%rbx)
      leaq  1(%rbx), %rbx
      jne LBB0_13
      .align  4, 0x90

GCC 4.9 does a better job (from generated):

    L37:
      addq  $1, %rax
      addl  $1, %ecx
      cmpb  $9, (%rax)
      jne L37

And so does ICC (from generated):

    L_B1.12:                        # Preds L_B1.10 L_B1.12         # Infreq
            incl      %r14d                                         #38.0
            movslq    %r14d, %rax                                   #38.0
            cmpb      $9, (%rax,%r12)                               #38.0
            jne       L_B1.12       # Prob 82%                      #38.0