new input format: tokenized_lines (one sentence per line, tokens separated by whitespace)