Before revision: Pre-trained language models have achieved state-of-the-art results in various natural language processing tasks. Most of them are based on the Transformer architecture , which distinguishes tokens with the token position index of the input sequence.
After revision: Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained language models are based on the Transformer architecture , which distinguishes tokens with the token position index of the input sequence.