Slight changes to tokenizer

made available more options for function, must test out mroe abbrev and
such