comments now treated as tokens; string preprocessing removed; tokens now acquiring body via indexing the main string