refactored code to generate ngrams only for units that dont have generated ngrams already