Before revision:  Methodology: Here, we trained two auto-regressive language models (Transformer-XL and XLNet) and two auto-encoder models ( BERT and Albert) using 80 billion amino acids from 200 million protein sequences (UniRef100) and 393 billion amino acids  from 2.1 billion protein sequences ( BFD ).
After revision:  Methodology: Here, we trained two auto-regressive language models (Transformer-XL and XLNet) and two auto-encoder models ( Bert, Albert) on data from UniRef and BFD containing up to 393 billion amino acids from 2.1 billion protein sequences ( BFD ).