bmuller/ankusa — pull request #6 by rurounijones (2014-05-30 10:12:18+00:00)
Title: Improve performance of Stopword lookup

Ankusa::STOPWORDS is created once and then searched for every single
word that we are classifying.

Change it from an Array with O(n) average time complexity to a Set
(Hash-Table) with O(1) average time complexity.

In tests on 1000 emails (Single threaded) the run-time was reduced
from 6.4 seconds to 2.4 seconds.

(Output below created using stackprof and Benchmark)

Before:

```
==================================
Mode: cpu(1000)
Samples: 1522 (2.75% miss rate)
GC: 107 (7.03%)
==================================
TOTAL (pct) SAMPLES (pct) FRAME
1158 (76.1%) 1002 (65.8%) Ankusa::TextHash.valid_word?
156 (10.2%) 99 (6.5%) String#numeric?
88 (5.8%) 88 (5.8%) block (2 levels) in Ankusa::MemoryStorage#get_vocabulary_sizes
57 (3.7%) 57 (3.7%) rescue in String#numeric?
69 (4.5%) 40 (2.6%) Ankusa::TextHash#add_word
42 (2.8%) 40 (2.6%) Ankusa::TextHash.atomize
29 (1.9%) 29 (1.9%) String#stem
21 (1.4%) 21 (1.4%) Ankusa::MemoryStorage#get_word_counts
9 (0.6%) 9 (0.6%) block (2 levels) in Ankusa::NaiveBayesClassifier#log_likelihoods
7 (0.5%) 7 (0.5%) block in Ankusa::Classifier#get_word_probs
9 (0.6%) 5 (0.3%) block in Ankusa::Classifier#get_word_probs
134 (8.8%) 5 (0.3%) Ankusa::Classifier#get_word_probs
92 (6.0%) 4 (0.3%) Ankusa::Classifier#vocab_sizes
4 (0.3%) 4 (0.3%) Ankusa::MemoryStorage#get_total_word_count
2 (0.1%) 2 (0.1%) String#to_ascii
1415 (93.0%) 1 (0.1%) Ankusa::NaiveBayesClassifier#classify
1 (0.1%) 1 (0.1%) Ankusa::MemoryStorage#get_doc_count
1228 (80.7%) 1 (0.1%) block in Ankusa::TextHash#add_text

user system total real
6.330000 0.010000 6.340000 ( 6.344521)
```

After

```
==================================
Mode: cpu(1000)
Samples: 582 (6.73% miss rate)
GC: 137 (23.54%)
==================================
TOTAL (pct) SAMPLES (pct) FRAME
158 (27.1%) 109 (18.7%) String#numeric?
86 (14.8%) 86 (14.8%) block (2 levels) in Ankusa::MemoryStorage#get_vocabulary_sizes
49 (8.4%) 49 (8.4%) rescue in String#numeric?
39 (6.7%) 34 (5.8%) Ankusa::TextHash.atomize
62 (10.7%) 33 (5.7%) Ankusa::TextHash#add_word
29 (5.0%) 29 (5.0%) String#stem
24 (4.1%) 24 (4.1%) Set#include?
21 (3.6%) 21 (3.6%) Ankusa::MemoryStorage#get_word_counts
147 (25.3%) 12 (2.1%) Ankusa::Classifier#get_word_probs
11 (1.9%) 11 (1.9%) block (2 levels) in Ankusa::NaiveBayesClassifier#log_likelihoods
10 (1.7%) 10 (1.7%) Ankusa::MemoryStorage#get_total_word_count
7 (1.2%) 7 (1.2%) block in Ankusa::Classifier#get_word_probs
16 (2.7%) 6 (1.0%) block in Ankusa::Classifier#get_word_probs
5 (0.9%) 5 (0.9%) String#to_ascii
91 (15.6%) 4 (0.7%) Ankusa::Classifier#vocab_sizes
184 (31.6%) 2 (0.3%) Ankusa::TextHash.valid_word?
2 (0.3%) 2 (0.3%) block in Ankusa::NaiveBayesClassifier#classify
87 (14.9%) 1 (0.2%) block in Ankusa::MemoryStorage#get_vocabulary_sizes

user system total real
2.290000 0.010000 2.300000 ( 2.306815)

```