Before revision:  We validated the advantage of up-scaling LMs to larger models supported by bigger data by predicting secondary structure ( 3-states: Q3= 76-84, 8-states: Q8=65-73), sub-cellular localization for 10 cellular compartments ( Q10= 74) and whether a protein is membrane-bound or water-soluble (  Q2= 89). Dimensionality reduction revealed that the LM-embeddings from unlabeled data (only protein sequences) captured important biophysical properties governing protein shape. This implied learning some of the grammar of the language of life realized in protein sequences.
After revision:  We validated the advantage of up-scaling LMs to larger models supported by bigger data by predicting secondary structure ( 3-states: Q3= 76-84, 8-states: Q8=65-73), sub-cellular localization for 10 cellular compartments ( Q10= 74) and whether a protein is membrane-bound or water-soluble ( Q2= 91\%). For the per-residue predictions the transfer of the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without using evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that protein LMs learned some of the grammar of the language of life realized in protein sequences.