Transformer-based models such as BERT or Roberta have shown SOTA performance in various NLP tasks over the last few years.  Pre-trained models are trained on a huge corpus of text from the web. These models contain more accurate representations of words and sentences.