fix a bug where tokens were not always being found (identified with the new test cases)