tokenize, stem, generate documents and implement a simple search interface