Improved EOF handling, but it's not perfect yet. Separated tokenizing functions into separate file for clarity