Before revision:  The LMs were trained on the Summit supercomputer , using 5616 GPUs  and one TPU Pod , using V3-512 cores. Results: The results of training these LMs on proteins was assessed by predicting secondary structure in three- and eight-states ( Q3= 75-83, Q8= 63-72), localization for 10 cellular compartments (Q10=74) and whether a protein is membrane-bound or water-soluble (Q2=89).
After revision:  The LMs were trained on the Summit supercomputer , using 5616 GPUs and one TPU Pod , using V3-512 or V3-1024). We validated the advantage of up-scaling LMs to larger models supported by bigger data by predicting secondary structure in three- and eight-states ( Q3= 75-83, Q8= 63-72), localization for 10 cellular compartments (Q10=74) and whether a protein is membrane-bound or water-soluble (Q2=89).