Added tokenizer and shutdown hook