Fix an off-by-one error in test failure reporting.
Avoid str::slice()

It fails if not slicing at character boundaries.
Simplify test_tokenizer()
Remove comment tokens, they are always ignored.
The spec was updated: whitespace and comments are allowed here.
Just drop invalid declarations.

The tokenizer does not preserve the content of eg. bad-string,
so a minifier is already not possible anyway.
More preserved tokens. consume_primitive() can not fail.
Rename to Parethesis, SquareBracket and CurlyBracket.