tokenizer for one doc is added as a function in master class