Interface Update

Major changes to interface (see token and token_stream structs). Not a
working build.
Better Tokenizer

Mostly done, except simple word processing. Needs major testing (did not
do incremental testing).
Better Tokenizer Complete

Better Tokenizer completed. Requires testing.
Better Tokenizer Fix 01

Some fixes for what should have been tested incrementally.
Better Tokenizer Fix 02

More fixes.
Better Tokenizer Fix 04

Forgive all these commits. I'd rather not edit through terminals.
Better Tokenizer Fix 04 (ACTUAL)

The real Fix 04. With some more changes.
Better Tokenizer Diagnostics

Added lines for diagnostic outputs.
Better Tokenizer Fix 05

Hopefully fixed the endless loop problem.
Better Tokenizer Fix 06

Fixed wrong buffer incrementation in tokenizer.
Better Tokenizer Fix 07

Fixed character not resetting on incrementation.
Better Tokenizer Final

Tokenizer now working perfectly. Added a free_token() function to
deallocate token memory.
Better Tokenizer Final 01

Added missing fixes and removed a diagnostic line.
Better Tokenizer Final 02

More forgotten things added (and removed).
Better Tokenizer Final 03

Fixed subshell close parentheses not being consumed and added into
subshell strings. Should be final fix.
Better Tokenizer Final 04

Really should work this time.