# From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
## 3 SOTA Pre-Trained Language Models (PLMs)
### Summary and Comparison
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Year** & **Resource (PLMs)** & **Team** & **Architecture** & **Parameters** \\ \hline Mar-2018 & ELMO & AI2 & Bi-directional LM & 94M \\ Jun-2018 & GPT & OpenAI & Transformer Dec. & 117M \\ Oct-2018 & BERT-base & Google & Transformer Enc. & 110M \\ Oct-2018 & BERT-large & Google & Transformer Enc. & 340M \\ Jan-2019 & Transformer-XL & Google AI & Transformer & 257M \\ Jan-2019 & XLM & Facebook AI & Transformer & 570M \\ Feb-2019 & GPT-2-large & OpenAI & Transformer Dec. & 774M \\ Feb-2019 & GPT-2-medium & OpenAI & Transformer Dec. & 345M \\ Feb-2019 & GPT-2-small & OpenAI & Transformer Dec. & 124M \\ Feb-2019 & GPT-2-XL & OpenAI & Transformer Dec. & 1.5B \\ May-2019 & UNILM & Microsoft & Transformer Enc. & 340M \\ May-2019 & MASS & Microsoft Research Asia & Transformer & 120M \\ Jun-2019 & XLNET & Google Brain and CMU & Transformer Enc. & 340M \\ Jul-2019 & ERNIE 2.0 & Baidu & Transformer Enc. & 114M \\ Jul-2019 & RoBERTa (base) & Facebook AI & Transformer Enc. & 109M \\ Jul-2019 & RoBERTa (large) & Facebook AI & Transformer Enc. & 355M \\ Sep-2019 & ALBERT-B & Google and Toyota & Transformer Enc. & 12M \\ Sep-2019 & ALBERT-L & Google and Toyota & Transformer Enc. & 18M \\ Sep-2019 & CTRL & Salesforce & Transformer & 1.63B \\ Sep-2019 & Megatron-LM & Nvidia & Seq2 Seq & 8.3B \\ Sep-2019 & TinyBERT & Huawei & Transformer Enc. & 14.5M \\ Oct-2019 & BART & Facebook & Transformer & 460M \\ Oct-2019 & DistilBERT & HuggingFace & Transformer Enc. & 66M \\ Oct-2019 & DistilGPT2 & HuggingFace & Transformer Dec. & 82M \\ Oct-2019 & T5 & Google & Transformer & 11B \\ Nov-2019 & XLM-R (base) & Facebook AI & Transformer & 270M \\ Nov-2019 & XLM-R (large) & Facebook AI & Transformer & 550M \\ Jan-2020 & Reformer & Google research & Transformer & 149M \\ Fev-2020 & MT-DNN & Microsoft & Transformer & 330M \\ Feb-2020 & T-NLG & Microsoft & Transformer & 17.2B \\ Mar-2020 & ELECTRA & Google Brain & Transformer Enc. & 335M \\ Apr-2020 & Longformer (base) & AllenAI & Transformer Enc. & 149M \\ Apr-2020 & Longformer (large) & AllenAI & Transformer Enc. & 435M \\ May-2020 & GPT-3 & OpenAI & Transformer Dec. & 175B \\ Jun-2020 & DeBERTa (base) & Microsoft & Transformer Dec. & 140M \\ Jun-2020 & DeBERTa (large) & Microsoft & Transformer Dec. & 400M \\ Jun-2020 & DeBERTa (xlarge-v2) & Microsoft & Transformer Dec. & 750M \\ Jun-2020 & DeBERTa (xlarge-v2) & Microsoft & Transformer Dec. & 900M \\ Jun-2020 & DeBERTa (xxlarge-v2) & Microsoft & Transformer Dec. & 105B \\ Jun-2020 & MARGE & Facebook AI & Seq2 Seq & 960M \\ Jun-2020 & GShared & Google & Transformer & 600B \\ Mar-2021 & GPT-NEO & EleutherAI & Transformer Dec. & 2.7B \\ Jun-2021 & XLM-E & Microsoft & Transformer & 279M \\ Oct-2021 & MT-NLG & Nvidia and Microsoft & Transformer & 530B \\ Oct-2021 & T0 & Researchers & Transformer & 11B \\ Dec-2021 & GLaM & Google AI & Transformer & 1.2 Trillion \\ Dec-2021 & Gopher & Google AI & Transformer & 280B \\ Dec-2021 & XGLM & Meta AI & Transformer & 7.5B \\ Jan-2022 & LaMDA & Google & Transformer Dec. & 137B \\ Feb-2022 & GPT-NeoX-20B & EleutherAI and CoreWeave & Transformer Dec. & 20B \\ Mar-2022 & Chinchilla & DeepMind & Transformer & 70B \\ Apr-2022 & PaLM & Google AI & Transformer & 540B \\ May-2022 & OPT & Meta AI & Transformer Dec & 175B \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Summary of recent PLMs.