changed algorithm of tokenizer and options parsing