separated out tagged_sentence from sentence_state
refactored get_token_type