Instances of tokens are called lexemes . For example, both 'do' and 'if' might be keywords, but of course it is important to know which strings was read. Likewise, 'foo' and 'bar' both are lexemes for identifiers, but of course we still need to know the particular string. A string is a finite sequence of symbols. The symbols are taken from a finite alphabet.
Regular expressions provide a notation to define sets of strings.
Suppose M and N are regular expressions. Both NFAs and DFAs are represented by transition graphs. Converting a regular expression to an NFA Read sect. An NFA accepts an input string if there is some path in the transition graph from the start to a final state the edge-labels of which form the input string. A DFA accepts an input string if the well-defined path in the transition graph that goes along the edges labeled with input string reaches a final state.