added tokenize_line method for code reuse