Refactor TF IDF to use WeightedDocumentTokens

Inspired by the "BagOfTokens" class in secondstring. It's still
just as slow, however.

7487s -> 9854s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 9854.3s

Testfiles: 23
Don't re-tokenize/rescore existing documents each time

9854s -> 4312s

       Min: 37.878788 (bible_books_asv_abbreviations.csv)
      Mean: 85.796106
    Median: 95.857988
       Max: 100.000000

     SCORE: 73.177627

      Time: 4312.1s

 Testfiles: 23
Don't re-tokenize query for every comparison.

Considerably faster. 4312s -> 929s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 929.4s

Testfiles: 23
@tokens -> @tf_idf_tokens in tf_idf_index.rb

Less ambiguous because there's also the WeightedDocumentTokens class.
Opps -- commit the RakeFile
Create `rake profile` task to help performance tuning
Only need the set of tokens for WeightedDocumentTokens

929s -> 808s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 808.3s

Testfiles: 23
Don't make multiple copies of the same string during indexing

Reduces GC load. 808s -> 773s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 773.4s

Testfiles: 23
Use Set#merge to build candidates list

773s -> 523s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 522.8s

Testfiles: 23
Minor optimizations

- Don't need to keep term counts around in WeightedDocumentToken's
- Use .max instead of .sort.last to select winner

523s -> 469s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 469.3s

Testfiles: 23
Save a method call in WeightedDocumentToken#cosine_similarity

469s -> 430s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 429.7s

Testfiles: 23
Write cosine_similarity in C w/RubyInline

430s -> 237s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 236.6s

Testfiles: 23
More optimizations to cosine_similarity_fast

237s -> 149s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 149.3s

Testfiles: 23
Just search all docs if the candidate selection gathers too many

149s -> 126s

      Min: 37.878788 (bible_books_asv_abbreviations.csv)
     Mean: 85.796106
   Median: 95.857988
      Max: 100.000000

    SCORE: 73.177627

     Time: 126.1s

Testfiles: 23
Use downcasing tokenizer. Better results.

SCORE: 73.2 -> 76.9 (Mean: 85.8% -> 90.8%)
Time: 126s -> 134s

       Min: 44.067797 (second_string_va_univ.csv)
      Mean: 90.778755
    Median: 95.857988
       Max: 100.000000

     SCORE: 76.901513

      Time: 134.3s

 Testfiles: 23
Don't add documents to candidates via common tokens

Cutoff set conservative enough to not affect results.

134s -> 82s

       Min: 44.067797 (second_string_va_univ.csv)
      Mean: 90.778755
    Median: 95.857988
       Max: 100.000000

     SCORE: 76.901513

      Time: 82.0s

 Testfiles: 23