The Ngram Generation works with only two columns and as such is not a full set of ngrams (a previous incarnation ran to a little over 8gb which seemed somewhat wrong