tokenize now includes offset, line number, and column number.