Added the ability to annotate regex so token attributes can be inferred.