Before revision:  Methodology: Here, we trained two auto-regressive language models (Transformer-XL and XLNet) and two auto-encoder models ( BERT and Albert) using 80 billion amino acids from 200 million protein sequences (UniRef100) and 393 billion amino acids  from 2.1 billion protein sequences ( BFD ).
After revision:  Methodology: Here, we trained two auto-regressive language models (Transformer-XL , XLNet) and two auto-encoder models ( BERT and Albert) using 80 billion amino acids from 200 million protein sequences (UniRef100) and 393 billion amino acids from 2.1 billion protein sequences ( BFD ).