pivot fix for tokens ending in punctuation