Add a __version__ attribute to javalang and
dynamically work out the version number in setup.py.

Signed-off-by: David Black <dblack@atlassian.com>
Make tokenizer accept (nearly) all identifiers

Java accepts many Unicode characters as part of identifiers,
the rules for these are pretty complex, but can be replicated by
using Unicode character categories.

The tokenizer is modified in a way that it recognises all valid
identifiers by using Unicode categories:

According to the Java Language Specification v7, a character may
start a Java identifier if and only if one of the following is true:

* The character is uppercase, lowercase, titlecase, modifier or other
  letter
  (Unicode categories: Lu, Ll, Lt, Lm, Lo)

* The character type is LETTER_NUMBER
  (Unicode category: Nl)

* The character is a currency symbol
  (Unicode category: Sc)

* The character is a connecting punctuation character
  (Unicode category: Pc)

For the rest of the identifier, the following additional cases are
allowed:

* The character is a digit
  (Unicode category: Nd)

* It is a combining mark
  (Unicode category: Mc)

* It is a non-spacing mark
  (Unicode category: Mn)

An additional property is that it is an ignorable character. This
particular condition is ignored, because it would make the tokenizer
more complex, as it would have to remove those ignorable characters, and
since they are probably not used in the real world, it's not worth the
hassle.
Merge pull request #4 from brakhane/master

Make tokenizer accept (nearly) all identifiers
Expand README to include examples and basic documentation.
Merge pull request #3 from dbaxa/expose_version_information

Add a __version__ attribute to javalang and
Version 0.9.5
Issue 6: Fix bug in unicode escape handling
Issue 7: Only attempt Javadoc detection for block comments
Merge pull request #8 from c2nes/bug/issue-7

Issue 7: Only attempt Javadoc detection for block comments