Make tokenizer accept (nearly) all identifiers

Java accepts many Unicode characters as part of identifiers,
the rules for these are pretty complex, but can be replicated by
using Unicode character categories.

The tokenizer is modified in a way that it recognises all valid
identifiers by using Unicode categories:

According to the Java Language Specification v7, a character may
start a Java identifier if and only if one of the following is true:

* The character is uppercase, lowercase, titlecase, modifier or other
  letter
  (Unicode categories: Lu, Ll, Lt, Lm, Lo)

* The character type is LETTER_NUMBER
  (Unicode category: Nl)

* The character is a currency symbol
  (Unicode category: Sc)

* The character is a connecting punctuation character
  (Unicode category: Pc)

For the rest of the identifier, the following additional cases are
allowed:

* The character is a digit
  (Unicode category: Nd)

* It is a combining mark
  (Unicode category: Mc)

* It is a non-spacing mark
  (Unicode category: Mn)

An additional property is that it is an ignorable character. This
particular condition is ignored, because it would make the tokenizer
more complex, as it would have to remove those ignorable characters, and
since they are probably not used in the real world, it's not worth the
hassle.
Merge pull request #4 from brakhane/master

Make tokenizer accept (nearly) all identifiers