dump_token()

+ Add a dedicated function for dumping tokens. And use it.

Signed-off-by: Max Reitz <xanclic@niwohlos.org>
Remove comments in tokenizer

+ The tokenizer now removes both single and multi-line comments.

Signed-off-by: Max Reitz <xanclic@niwohlos.org>
Use throw for emitting errors

* In tokenize(), it is much nicer to use throw with the error message to
  emit errors instead of always having the same three lines to handle
  the error everywhere.

Signed-off-by: Max Reitz <xanclic@niwohlos.org>
Link against libm

+ We'll need it.

Signed-off-by: Max Reitz <xanclic@niwohlos.org>
Support for floating point tokens

+ Now, floating point literals can be parsed; both in decimal and
  hexadecimal (binary) form (part of C11 only).

Signed-off-by: Max Reitz <xanclic@niwohlos.org>