Repository: piskvorky/gensim

Issue:
Phraser requires unnecessary memory
Currently, `Phraser` objects (= the trimmed-down version of the full bigram finder `Phrases`) contains the actual bigrams in an internal attribute called [phrasegrams](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/phrases.py#L775). This is the biggest and most memory-intense part of a `Phraser` object.

`phrasegrams` is a dict of `{tuple of strings => (frequency [int], score [float])}`. But the `int` (the frequency count of that particular bigram) is unused. This means we're constructing that int, plus the wrapping tuple, for no good reason, inflating the necessary RAM. See also [mailing list discussion](https://groups.google.com/forum/#!topic/gensim/vsWXxUzuK6s).

Task:
* Drop the `int` from Phraser values, leaving only the `float`.
* And while at it, rename (deprecate) the `.vocab` attribute of Phrases to something more appropriate, for example `bigram_counts`.




Discussion:
Note the `vocab` dict of a `Phrases` also contains unigram counts. So if renamed, a name like `frequencies` might make sense. 

But since renaming could break user code (or necessitate more compatibility-maintaining cruft), I wouldn't rename it unless part of a major refactoring/cleanup. The name `vocab` isn't that bad in its context-of-initial-assignement (where it has a [helpful comment](url)), and is roughly in line with the naming of similar frequency-dicts elsewhere in gensim (such as the word2vec-and-related spaghetti-mountains). And there's a bunch of other variables/methods with `_vocab` in them in `phrases.py`, all somewhat related to the `vocab` dict's role, which may equally need better-names  – but those better-names aren't necessarily just search-and-replacing `vocab` with whatever new name is chosen for the dict-property. 
I would like to take this up. Should I rename the .vocab attribute ?
I agree with @gojomo: `.vocab` are really ok. If you only want to rename it - just add an getter/setter with new name: this don't break anything and allows to use both (backward compatible). I'm also +1 for variant if we don't rename / add getters to don't increase "compatibility-maintaining cruft". 




Resolving patch:
diff --git a/gensim/models/phrases.py b/gensim/models/phrases.py
--- a/gensim/models/phrases.py
+++ b/gensim/models/phrases.py
@@ -208,6 +208,13 @@ def load(cls, *args, **kwargs):
         """
         model = super(PhrasesTransformation, cls).load(*args, **kwargs)
         # update older models
+        # if value in phrasegrams dict is a tuple, load only the scores.
+
+        for component, score in getattr(model, "phrasegrams", {}).items():
+            if isinstance(score, tuple):
+                frequency, score_val = score
+                model.phrasegrams[component] = score_val
+
         # if no scoring parameter, use default scoring
         if not hasattr(model, 'scoring'):
             logger.info('older version of %s loaded without scoring function', cls.__name__)
@@ -815,7 +822,7 @@ def __init__(self, phrases_model):
         for bigram, score in phrases_model.export_phrases(corpus, self.delimiter, as_tuples=True):
             if bigram in self.phrasegrams:
                 logger.info('Phraser repeat %s', bigram)
-            self.phrasegrams[bigram] = (phrases_model.vocab[self.delimiter.join(bigram)], score)
+            self.phrasegrams[bigram] = score
             count += 1
             if not count % 50000:
                 logger.info('Phraser added %i phrasegrams', count)
@@ -858,7 +865,7 @@ def score_item(self, worda, wordb, components, scorer):
 
         """
         try:
-            return self.phrasegrams[tuple(components)][1]
+            return self.phrasegrams[tuple(components)]
         except KeyError:
             return -1
 


Test patch:
diff --git a/gensim/test/test_data/phraser-3.6.0.model b/gensim/test/test_data/phraser-3.6.0.model
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/phraser-3.6.0.model differ
diff --git a/gensim/test/test_data/phrases-3.6.0.model b/gensim/test/test_data/phrases-3.6.0.model
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/phrases-3.6.0.model differ
diff --git a/gensim/test/test_phrases.py b/gensim/test/test_phrases.py
--- a/gensim/test/test_phrases.py
+++ b/gensim/test/test_phrases.py
@@ -12,7 +12,6 @@
 import unittest
 
 import six
-
 import numpy as np
 
 from gensim.utils import to_unicode
@@ -646,6 +645,22 @@ def testEncoding(self):
         self.assertTrue(isinstance(transformed, six.text_type))
 
 
+class TestPhraserModelCompatibilty(unittest.TestCase):
+
+    def testCompatibilty(self):
+        phr = Phraser.load(datapath("phraser-3.6.0.model"))
+        model = Phrases.load(datapath("phrases-3.6.0.model"))
+
+        test_sentences = ['trees', 'graph', 'minors']
+        expected_res = ['trees', 'graph_minors']
+
+        phr_out = phr[test_sentences]
+        model_out = model[test_sentences]
+
+        self.assertEqual(phr_out, expected_res)
+        self.assertEqual(model_out, expected_res)
+
+
 if __name__ == '__main__':
     logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)
     unittest.main()