adding support for Token-based (using jflex).