Added an option to concatenate tokens to make the final output easier. Also, the output is now an AutomatonResult.
fix all the things