Added do_lexical_analysis function.