moved tokenizer and preprocessor into seperate files