Generated tokens from a source string using a eBNF description of a context-free grammar.