General purpose lexical scanner