Before revision:  Here, we trained two auto-regressive language models (Transformer-XL, XLNet) and two auto-encoder models ( Bert, Albert ) on data from UniRef and BFD containing up to 393 billion amino acids (words) from 2.1 billion protein sequences (22- and 112-times the entire English Wikipedia) .
After revision:  Here, we trained two auto-regressive language models (Transformer-XL, XLNet) and two auto-encoder models ( BERT, Albert, Electra, T5 ) on data from UniRef and BFD containing up to 393 billion amino acids (words) from 2.1 billion protein sequences (22- and 112-times the entire English Wikipedia) .