SlowGuess/NeurIPS_2023_OCR huggingface dataset:
High confidence (pLDDT>90) AlphaFold2 [35] predicted structures of generated sequences aligned to the experimentally solved structure of the target chorismate mutase (PDB: 1ECM) show that structure is preserved despite significant divergences in sequence identity; two examples are shown in Figure 16 and the full distributions of pLDDTs and TM-scores are shown in Figure 5A.
(a) Distribution of max seq id between generated and natural sequences
Figure 15: Statistics of natural and generated sequences.
(b) Per position entropy of natural and generated sequences
L Sampling methods for sequence generation experiments
When sampling sequences from PoET, we used nucleus sampling [50] with  $p = 0.90$ . When sampling sequences from ProGen, we tried nucleus sampling with  $p \in \{0.25, 0.50, 0.75\}$ , which are the three sampling strategies used by the authors of ProGen for lysozyme sequence generation. We found that when  $p < 0.75$ , the resulting libraries had low diversity ( $> 50\%$  of generated sequences were duplicates). Therefore, we only present results with  $p = 0.75$ ; under this setting, almost all generated sequences were unique.
M Limitations
Orphan Proteins While ProteinGym includes DMS data on some proteins with low MSA depth, it does not include any orphan proteins that have no homologs. Simulating orphan proteins by e.g. removing all homologs from the training set is computationally intractable as it would require retraining large models and may be sensitive to homology search parameters. As a result, our experiments are inconclusive with respect to orphan proteins.
Exhaustive scoring of all high order mutants Some methods such as GEMME can achieve similar albiet lower performance than PoET with substantially lower compute. However, no method is efficient enough to exhaustively score all variants with more than a couple of mutations. For
(a) Structure alignment with the highest scoring (according to PoET) generated sequence with maximum sequence identity of 0.8 to a natural sequence.
(b) Structure alignment with the highest scoring (according to PoET) generated sequence with maximum sequence identity of 0.5 to a natural sequence.
Figure 16: AlphaFold2 predicted structures of generated sequences aligned to the target chorismate mutase (PDB: 1ECM). Both alignments have RMSD  $\leq 1.02$  and TM Score  $\geq 0.9$ .
exploring the space of higher order mutants, direct generation is far more effective, and is enabled by PoET via sampling.
Limited compute for ablation studies Our ability to conduct ablations is ultimately constrained by compute resources. Ablation models were trained until validation set performance plateaued, but further training may have changed the results.
Figure 17: UMAP of natural chorismate mutases (blue), chorismate mutases functional in E. coli (orange), and sequences sampled from PoET when conditioned on the latter sequences (green). Sequences are compared using the Hamming distance metric.
N Supplementary Tables
Table S1: Average Spearman correlation between model scores and experimental measurements on ProteinGym by MSA depth. This is an extended version of the main result table (Table 1) that includes performance on the validation benchmark.
Model type	Model name	Val	Substitutions by MSA Depth	Indels
Low	Medium	High	All
Alignment-based model	Site independent	0.386	0.417	0.404	0.411	0.408	N/A
GEMME	0.418	0.445	0.449	0.522	0.463	N/A
EVE (ens.)	0.435	0.414	0.441	0.498	0.448	N/A
Unconditional PLM	ESM-1v (ens.)	0.355	0.356	0.372	0.510	0.398	N/A
ProGen2 (ens.)	0.397	0.357	0.416	0.448	0.411	0.407
Trancection L (no retrieval)	0.396	0.377	0.399	0.429	0.401	0.430
Conditional Hybrid	MSA Transformer (ens.)	0.440	0.372	0.421	0.477	0.423	N/A
PoET	0.481	0.476	0.466	0.542	0.484	0.510
Trancection L	0.447	0.441	0.437	0.472	0.445	0.464
TranceptEVE M	-	-	-	-	-	0.516
TranceptEVE L	0.470	0.454	0.463	0.508	0.471	0.466
PoET+TrancptEVE L	0.494	0.479	0.480	0.537	0.492	0.521
Table S2: Average Spearman correlation between model scores and experimental measurements on ProteinGym by mutation depth. Mutation depth is the number of substitutions a variant sequence has compared to the target sequence. Bolded values indicate the highest value in each column, with values above and below the double-line considered separately.
Model type	Model name	Val	Substitutions by Mutation Depth
1	2	3	4	5+	All
Alignment-based model	Site independent	0.386	0.408	0.325	0.306	0.316	0.409	0.408
GEMME	0.418	0.459	0.412	0.394	0.353	0.440	0.463
EVE (ens.)	0.435	0.449	0.409	0.405	0.351	0.429	0.448
Unconditional PLM	ESM-1v (ens.)	0.355	0.396	0.309	0.203	0.165	0.253	0.398
ProGen2 (ens.)	0.397	0.409	0.312	0.233	0.228	0.282	0.411
Trancection L (no retrieval)	0.396	0.392	0.398	0.418	0.334	0.422	0.401
Conditional Hybrid	MSA Transformer (ens.)	0.440	0.426	0.381	0.444	0.374	0.431	0.423
PoET	0.481	0.483	0.456	0.473	0.404	0.449	0.484
Trancection L	0.447	0.441	0.427	0.440	0.370	0.461	0.445
TranceptEVE M	-	-	-	-	-	-	-
TranceptEVE L	0.470	0.471	0.445	0.456	0.389	0.464	0.471
PoET + TranceptEVE L	0.494	0.492	0.465	0.478	0.411	0.471	0.492
Table S3: Average Spearman correlation between model scores and experimental measurements on ProteinGym by taxon.
Model type	Model name	Val	Substitutions by Taxon
Human	Other Eukary.	Prokary.	Virus	All
Alignment-based model	Site independent	0.386	0.383	0.471	0.402	0.414	0.408
GEMME	0.418	0.434	0.514	0.507	0.438	0.463
EVE (ens.)	0.435	0.403	0.499	0.500	0.435	0.448
Unconditional PLM	ESM-1v (ens.)	0.355	0.417	0.441	0.502	0.256	0.398
ProGen2 (ens.)	0.397	0.394	0.459	0.462	0.364	0.411
Trancection L (no retrieval)	0.396	0.359	0.436	0.450	0.395	0.401
Conditional Hybrid	MSA Transformer (ens.)	0.440	0.352	0.455	0.494	0.441	0.423
PoET	0.481	0.445	0.524	0.526	0.479	0.484
Trancection L	0.447	0.413	0.503	0.478	0.429	0.445
TranceptEVE M	-	-	-	-	-	-
TranceptEVE L	0.470	0.436	0.518	0.514	0.454	0.471
PoET + TranceptEVE L	0.494	0.450	0.538	0.536	0.486	0.492
O Supplementary Figures
Figure S1: Model performance for each dataset in the ProteinGym substitutions benchmark.
Figure S2: Model performance for each dataset in the ProteinGym indels benchmark.