Modified the ngram generation logic so that only vocabulary indices are communicated into the training, i.e. more work is pushed into the
(heavily multithreaded) ngram generator.