Tokenizer complete!

Call the get_tokens function to get a dynamically allocated array of
tokens. Additional notes:
    1. Newlines are all enveloped into a single token
    2. OR_STATE covers both | and ||