second commit

added regular expression patterns for matching tokens
defined out intermediate parts some
added more comments