tokenizer: fixed bug in reader and adapted test for tokenizer