BERT was one of the first models in NLP that was trained in a two-step way:
1. BERT was trained on massive amounts of unlabeled data (no human annotation) in an unsupervised fashion.
2. BERT was then trained on small amounts of human-annotated data starting from the previous pre-trained model resulting in state-of-the-art performance.