Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Authors: Talal Widatalla
Richard W. Shuai
Brian Hie
Possu Huang
Venue: ICML 2025 poster (2025)
Keywords: Protein
Molecule
Diffusion
Graph-Neural-Network
Multi-task
MLM
Chemistry
Biology
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: We introduce a method for full-atom protein sequence design by simultaneously predicting discrete sequence and continuous 3D side-chain coordinates.

Abstract: Leading deep learning-based methods for fixed-backbone protein sequence design do not model protein sidechain conformation during sequence generation despite the large role the three-dimensional arrangement of sidechain atoms play in protein conformation, stability, and overall protein function. Instead, these models implicitly reason about crucial sidechain interactions based solely on backbone geometry and amino-acid sequence. To address this, we present FAMPNN (Full-Atom MPNN), a sequence design method that explicitly models both sequence identity and sidechain conformation for each residue, where the per-token distribution of a residue’s discrete amino acid identity and its continuous sidechain conformation are learned with a combined categorical cross-entropy and diffusion loss objective. We demonstrate learning these distributions jointly is a highly synergistic task that both improves sequence recovery while achieving state-of-the-art sidechain packing. Furthermore, benefits from explicit full-atom modeling generalize from sequence recovery to practical protein design applications, such as zero-shot prediction of experimental binding and stability measurements.

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Talal Widatalla 12
Richard W. Shuai
Brian L. Hie 234 Po-Ssu Huang 15
Abstract
task backbone-condittoned sequence generation task. These models typically graph neural network (GNN) to encode the backbone structure, then generate the sequence either autoregressively or through a masked language mod- eling objective (Dauparas et al., 2022; Hsu et al., 2022; Ingraham et al., 2019; Ruffolo et al., 2024; Zheng et al., 2023; Gao al., 2022a; Akpinaroglu al.). Because these methods do not model sidechains during sequence design, they must implicitly reason about sidechain interactions through the limited perspective of backbone structure and sequence identity alone.
Leading deep learning- based methods for fixed- backbone protein sequence design do not model protein sidechain conformation during sequence generation despite the large role the three- dimensional arrangement of sidechain atoms play in protein conformation, stability, and overall protein function. Instead, these models implic- itly reason about crucial sidechain interactions based on backbone geometry and known amino acid sequence labels. To address this, we present FAMPNN (Full-Atom MPNN), a sequence design method that explicitly models both sequence iden- tity and sidechain conformation for each residue, where the per-token distribution of a residue's discrete amino acid identity and its continuous sidechain conformation are learned with com- bined categorical cross-entropy and diffusion loss objective. We demonstrate that learning these dis- tributions jointly a highly synergistic task that both improves sequence recovery while achieving state-of-the-an sidechain packing. Furthermore, benefits from full-atom modeling generalize from sequence recovery to practical protein design ap- plications, such as zero-shot prediction of experi- mental binding and stability measurements.
We summarize our key contributions as follows:
We introduce a method which models the per-token distribution of residue's discrete sequence identity and continuous sidechain structure with combined cross-entropy and diffusion loss objective.
We implement lightweight, iterative sampling method for generating samples from this joint distribution.
We demonstrate that modeling this joint distribution allows for improved sequence design and experimental protein fitness prediction.
2. Related work
2.1. Fixed-backhone sequence design
The task of fixed-backbone sequence design is to design a protein sequence that will fold into a structure that matches the given protein backbone (Dauparas al., 2022). Tra- ditional physics-based approaches energy functions to combinatorially optimize sequence identity and sidechain conformation to find low-energy configurations (Holm & Sander, 1992: Dahiyat & Mayo, 1997).
1. Introduction
Most existing methods for designing protein sequences for given structure do not explicitly model or encode sidechains. Instead, they formulate the sequence design
"Equal contribution. Either may be listed first a CV. "Co-corresponding authors. 'Department of Biophysics, Stan- ford University, Stanford, CA Insitute, Palo Alto, CA "Department of Chemical Engineering, Stanford University. Stan- ford, CA "Stanford Data Science, Stanford University, Stanford, CA Department of Bioengineering, Stanford University. Stanford, CA. Correspondence to: Po-Ssu Huang <possu stanford.edu>> Brian L. Hie briianhie stanford.edu..
Recently, deep learning-based approaches have proven highly successful this task (Jing al., 2020; Norn et al., 2021; Hsu et al., 2022; Gao al., 2022b;a; Zheng al., 2023; Anand al., 2022; Akpinaroglu al.; Ruffolo et al.,
2024; Ingraham et al., 2023). In addition to being able to recover the sequence of natural proteins given their back- bone, these models are surprisingly capable of also guiding improvements in binding affinity (Shanker et al., 2024) and improving protein expression, solubility, and stability (Sum-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
a
AOXVXNVCXS....
Full-ator encoder
ARTVNNVCDS...
ARTVNNWCDS..
Node embeddings
Per-residue sidechain dilfusion
Confidence
module
Input backbone
All-atom structure
Partial sequence Partial sidechain
Diffusion
b
Full-atom orientations
200000000X
Predict a adechains
Rackbone distances
MLP
MLP
Sequence logits
soquence
MPNN
Full-atom MPNN
Full-atom GVP
Fullatton MPNN
MLP
MLP
Sidechain diffusion
Hepear
Unmaak residues aktnchain
Full-atom distances
Figure 1. (a) Full-atom sequence design overview, consisting of dual masked language modeling scheme on sequence and sidechain structure to co-generate sequence and sidechain atoms. (b) Given backbone with partially masked sequence and sidechains, the full-atom encoder uses modified MPNN and GVP layers to produce node embeddings, which are used for sequenee prediction and sidechain generation. (c) Sampling begins with fully-masked backbone and is iteratively unmasked generate sequence and sidechains.
ida al., 2024). Among these, ProteinMPNN currently mains the most widely used model for sequence design due to its proven experimental performance for various practical protein design applications (Dauparas et al., 2022; Bennett et al., 2023; Dauparas et al., 2023).
2.3. Sequence-sidechain co-generation methods
Closest work models that perform both sequence design and sidechain packing, AttnPacker is trained with an auxiliary task of simultaneously predicting amino acid sequence identity by randomly dropping out residue iden- titics in the input sequence. However, their work primarily focuses on use cases for sidechain packing rather than se- quence design. The 3DCNN model from Anand et al. can sample sequence and sidechains conditioned on neighboring sidechains (Anand et al., 2022), but it requires a computa- tionally expensive Markov Chain Monte Carlo (MCMC) procedure to iteratively redesign residues. The high demand for computational resources reduces its broad application. LigandMPNN extends ProteinMPNN to design sequences conditioned on ligand context, which can optionally include sidechain atoms ligand context, but it remains unable to co- generate sidechains and sequence (Dauparas al., 2023). Instead, LigandMPNN packs sidechains by relying on a separately trained model after the full sequence has been designed, meaning that the sequence designed com- pletely independently of predicted neighboring sidechain conformations.
2.2. Sidechain packing
The task of sidechain packing is, given a protein backbone and its amino acid sequence, to predict the sidechain coor- dinates of the ground truth structure. Traditional physics- based approaches for sidechain packing minimize energy by sampling rotamers from rotamer library (Krivov al., 2009; Xu & Berger, 2006; Miao et al., 2011). Following the groundbreaking success of deep learning approaches in protein structure prediction (Abramson et al., 2024; Jumper et al., 2021), recent specialized sidechain packing meth- ods have achieved strong performance over physics-based methods. DLPacker formulates the problem as an image- to-image task and trains 3D U-Net predict sidechain coordinates (Misiura et al., 2022). AttnPacker uses in- variant graph transformer with triangle updates followed by an equivariant Tensor Field Network Transformer to predict sidechain coordinates (McPartlon & Xu, 2023). PIPPack introduces invariant point message passing layers to predict binned X-angle distributions (Randolph & Kuhlman, 2023). DiffPack uses torsional diffusion autoregressively gener- ate the four sidechain torsional angles X1 to Xa (Zhang et al., 2024). FlowPacker uses torsional flow matching with an cquivariant graph neural network to simultaneously denoise all four sidechain torsional angles (Lee& & Kim, 2024).
2.4. Next-token modeling for continuous data
Autoregressive models for generating continuous data often rely on discretizing the data before tokenization (Ramesh et al., 2021: Esser et al., 2021: Chang et al., 2022). To eliminate the need for discrete tokenizers, recent approaches have used diffusion loss objective with autoregressive
2

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
modeling to model per-token distributions for continuous- valued data (Li et al., 2024; Fan al., 2024; Sun et al., 2024). Our work similarly uses diffusion loss to extend masked generative models to generate tokens with both discrete and continuous components.
full-atom sequence design procedure as a generative masked language modeling objective. Similar to MaskGIT, (Chang et al., 2022) we begin sampling with both sequence and sidechains fully masked and follow an iterative procedure: at each step, we predict all tokens in parallel, then unmask some subset of the sequence and sidechain tokens. We repeat this process until all tokens are unmasked, yielding a fully designed sequence and full-atom structure (Figure 1c). Sampling with many steps residues reduces to random- order autoregression.
3. Motivation
The structure and function of proteins are largely determined by the physicochemical interactions among its atoms and its system. Current fiee-backbone sequence design methods implicitly capture these interactions via backbone geome- try and amino acid identity labels alone, and the resulting sequences often achieve remarkable solubility and stabil- ity. However, these methods fall short on tasks involving nuanced sidechain arrangements. There are several inter- molecular interactions which are functions of the precise distances (e.g. Van der Waals force and Coulombic attrac- tion) and angles (e.g. hydrogen bonding and pi-stacking), amongst sidechain and backbone atoms. Thus, we hypoth- csize that by explicitly modeling full-atom structure in an efficient manner during sequence design, we can better cap- ture complex, yet crucial, sidechain interactions unavailable to backbone methods, improving the quality of designed sequences.
To additionally support sidechain packing, a crucial task in protein design, we introduce a separate masking scheme exclusively on sidechains. During training, among po- sitions with unmasked sequence identity, we randomly mask each residue's sidechain coordinates with probabil- ity p Uniform(0, 1). This enables the model to pre- dict sidechain conformations given either partial or com- plete sequence information, with varying amounts of known sidechain information. This also allows for the model to predict sequence identity given a mixture of sequence-only and spuuenc--nndssdechaae context.
4.2. Full-atom sequence design
Fulfilling the sequence design process while explicitly mod- eling sidechain structure requires addressing two primary considerations. (1) Determining protein sequence given its full-atom structure is trivial, as the composition of sidechain atoms completely identifies the underlying amino acids, and (2), sidechains for different amino acids have a varying num- ber of atoms, so we require a unifynng representation for all sidechains
To carry out this hypothesis, our design process aims to achieve two goals:
1. At each step in the sequence design process, the model should co-generate sidechain atoms along with its pre- diction of the sequence.
2. To predict sequence and sidechains, the model should encode the currently known sidechain atoms in addi- tion to the backbone and sequence.
4. 1. SIDECHAIN COORDINATE REPRESENTATION
To address these considerations, we represent residues in atom37 format, in which each residue is represented as fixed size matrix of size 37 x 3. Each row corresponds to the 3D coordinates of all 37 possible atom types. 4 of these rows correspond to the backbone atoms N, Ca, C, and O, while the remaining 33 rows correspond to the residue's 's sidechain atoms. For sidechains where particular atom type not present, the row for that atom type a "ghost atom" and is set the residue's Ca position, accommodat- ing consideration (2). Utilization of ghost atoms also allows us to address consideration (1) as we can prevent sidechain structure from revealing the amino-acid identity of masked sequence position by setting all sidechain atoms as ghost atoms, essentially "masking" the sidechain structure as well.
4. Methods
4.1. Generative masked language modeling scheme
We train FAMPNN with masked language modeling on sequence identity. Simultaneously, we provide sidechain coordinates of unmasked sequence context, and mask the sidechain coordinates of masked sequence positions. As seen in Figure 1a, this allows for prediction of masked se- quence and sidechain conformations given partial sequence and sidechain coordinates. To predict both sequence and sidechains, during training, we combine categorical cross- entropy loss for sequence prediction and per-token diffu- sion loss for sidechain conformation prediction, such that FAMPNN is a single model trained end-to-end on both ob- jectives (Appendix C).
4.2.2. FULL- ATOM ENCODER
We represent full-atom protein structure graph, en- coded with a graph neural network (GNN) using hybrid MPNN-GVP architecture (Dauparas et al., 2022; Jing et al.,
Because our method generates both sequence and sidechain coordinates in a per-token manner, we can formulate the
3

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
2020). This architecture consists of three primary compo- nents, which are an invariant backbone encoder, an invariant full-atom encoder, and an equivariant full-atom encoder, with respect to any composition of global rotations and reflections in protein coordinates.
Euclidean denoising of sidechain atoms.
4.3. Sidechain coordinate generation
We formulate sidechain coordinate generation as per-token Euclidean diffusion with diffusion loss (Li et al., 2024) on coordinates predicted from the node embeddings. During sidechain coordinate generation, we use sidechain repre- sentation that is invariant to global rotations and translations of the input backbone, which we have empirically seen im- proves training efficiency. For each residue, we transform all sidechain coordinates (including ghost atoms) into the local frame defined by its backbonande atoms (Appendix D.3.1). At inference, after a full denoising trajectory has been run, sidechain coordinates are transformed back into the global frame by inverting this transformation.
The first two components build off the architecture of Pro- teinMPNN (Dauparas et al., 2022). ProteinMPNN featurizes backbone structure as a k-NN graph, with a set of nodes V that represent each protein residue, and edges & = {ei-j }. featurized with backbone inter-atomic distances, defined for the 48 nearest neighbors of each residue. ProteinMPNN includes an initial structure encoder consisting of three pre- viously described (Ingraham et al., 2019) invariant MPNN layers, which update both edge representations Eaj and node representations Uy: MPNNEncoder((u, eij) = (U/ Egy). This output is passed to sequence decoder comprised of three additional invariant MPNN layers that combines causally- masked one-hot encodings of sequence S, updating only node representations: MPNNDecoder((', lely, s:D) (v("). This output is then passed into a final output head for se- quence prediction.
4.3. 1. PER-TOKEN EUCLIDEAN DIFFUSION
We use diffusion to model the distribution of a residuc's sidechain coordinates given its amino acid identity, the pro- tein backbone, and the currently unmasked sequence and sidechains. Similar to Chu et al., our diffusion scheme fol- lows the variance-exploding EDM scheme from Karras et al. (Karras et al., 2022; Chu et al., 2024), where we train a conse-conditioned denoiser Dp to minimize the L2 error from a Gaussian-noised version of the sidechain coordi- nates. We parametrize Ds using a lightweight multilayer perceptron (MLP). We use AdaLN (Peebles & Xie, 2023) to condition this diffusion MLP on the current noise level and one-hot encoding of the residue's 's amino acid identity sp To encode information about the backbone and unmasked sequence and sidechains, we also use AdaLN to pass in the node embedding v; from our full-atom encoder.
Our initial component, the invariant backbone encoder, is identical to MPNNEncoder, encoding the backbone struc- ture only. However, for our second component, we replace MPNNDecoder with a full-atom encoder, which is identical to the backbone encoder, MPNNEncoder, but with expanded fcaturization to all atoms. Similar to MPNNDecoder, we concatenate one-hot encodings of sequence identity, but re- move the causal mask, as FAMPNN is trained with an MLM objective, and sequence positions are randomly masked with corresponding token.
Lastly, to allow the model to reason over vector-valued in- teratomic orientations in addition to previously encoded scalar-valued interatomic distances, we require equivariant layers, since atomic orientations are equivariant to global ro- tations and reflections. Thus, the third and final component of the model are modified Geometric Vector Perceptrons (GVP) from Jing et al. (Jing et al., 2020). GVP layers con- sist of an equivariant track for learning vector features and an invariant track for learning scalar features. In FAMPNN, in addition to the backbone-level features used in Jing et al. (Jing et al., 2020), we use the equivariant track to encode unit vectors from Ca, all other atoms in residue i. For equivariant edge features, we include unit vectors from Ca, to all atoms in residue j. To the invariant track, we addition- ally incorporate distances from Ca,, to all atoms in residue j. as done in the invariant full-atom encoder.
At train time, when denoising the sidechain coordinates of residue i, we teacher-force the sequence s, rather than pre- dicting s, Similar to previous work (Abramson et al., 2024; Li et al., 2024), because the diffusion MLP is lightweight relative to the rest of the network, we train the MIP with larger batch size for better training efficiency. In prac- tice, for each training example, we clone the conditioning inputs to the MIP 8 times and sample 8 different noise levels for training the denoiser. At inference time, to gen- erate sidechain atoms for residue i, we first run the MPNN module generate a prediction of the sequence identity & then pass in both 8, and v; into the sidechain diffusion MI.P We then initialize all sidechain atoms from a Gaussian dis- tribution and run a diffusion trajectory to realize sidechain coordinates (Figure la).
To tie the three modules together, we initialize the invariant track of GVP with the final node and edge representations from the invariant full-atom encoder. This representation is then passed both to a final output head for sequence pre- diction and to the sidechain diffusion module for invariant
4.3.2. PREDICTED SIDECHAIN ERROR
Predicted error in protein structure prediction models is valuable for interpretability, design filtering, and as an op- timization objective for de novo design (Abramson et al.,
4

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Self-consistency TM-score (1)
Self-consistency RMSD (1)
Table I. Median sequence accuracy on CATH 4.2 set, re- ported by Gao et. al (Gao et al., 2022a). Bold values indicate the best results. 'Evaluated on CATH 4.3 test set (Hsu et al., 2022). *Reproduced by Gao et. al with CATH 4.2 (Gao et al., 2022a). Dashes (+) indicate that perplexity was not reported by authors.
2.0
Medteen McTMM6O
23
Medien WCTM
C.D
13
6.4-
100
200
300
130
200
302 Length
602
302
Model
Recovery % t
Perplexity
b
Length
GraphTrans
35.82
6.63
Modan 2CTM
0.5
Med CORMOD
StructGNN
35,91
6.40
0.0
GCA
37.64
6.05
0.4
ESM-IFI
38.30
6.44
0.2
GVP-large+
39.20
6.17
0.0
strpp
strpa
10s stepo
Extop
Fatloy
130 shope
Lanpa
GVP
39.47
5.36
AlphaDesign
41.31
6.30
ProteinMPNN
45.96
4.61
Figure 2. (a) Median self confisonnys TM-scores (left, higher is better) and RMSDs (right, lower better) for sequence design methods across 100 de novo backhones at each length in (100, 200, 300, 400, 500) generated by RFdiffusion. (b) Median self- consisteney TM-score (left) and RMSDs (right) across all 500 de novo backbones vs. number FAMPNN iterative unmasking steps. L steps denotes unmasking residue per step, which is equivalent to random order autoregression. Error bars represent 95% hootstrap confidence intervals around the median.
ESM-IFI (AF2DB)+
51.60
4.01
PiFold
51.66
4.55
Frame2Seq
46.53
Frame2Seq (Ensemble)
49.11
FAMPNN
49.66
4.99
which utilize pre-trained protein language models such as ESM2 (Lin al., 2023), as the pre-trained modules do explicitly holdout the CATH 4.2 test set.
2024; Frank et al., 2024; Bryant et al., 2022). Similar to other sidechain packing methods, we train a confidence module to predict packing error (Zhang al., 2024; Lee & Kim, 2024; McPartlon & Xu, 2023). Our confidence mod- ule estimates predicted sidechain error (pSCE) per-atom error in Angstroms (Figure 1). Similar to AlphaFold3, dur- ing confidence module training, we use diffusion rollout to obtain sidechain coordinates for computing error against the ground truth (Abramson et al., 2024). The confidence module encodes node embeddings from the full-atom en- coder with MPNNDecoder layers, then uses the predicted sidechain coordinates and amino acid type predict the per-atom error in evenly spaced bins from OA to 4A. At inference time, we compute PSCE by taking the expectation across these binned probabilities (Appendix D.4).
Because high sequence recovery does not always translate to high protein design success rates, another critical metric for evaluating protein design methods is self-consistency. measure of how well the designed sequence is predicted to recapitulate the input backbone. Specifically, pute the TM-score (structural similarity) and RMSD of the input structure and the AlphaFold2 structure prediction of the designed sequence, reported as scTM and scRMSD respectively (Zhang & Skolnick, 2005). Similar to the ProteinBench inverse folding benchmark, we evaluate on RFddiffusionggeeerate de novo backbones from lengths 100 to 500 (Ye et al., 2024a). For each length in {100, 200, 300, 400, 500), evaluate various sequence design methods 100 de novo backbones. Shown in Figure 2a, FAMPNN (0.3A) is competitive with commonly used ProteinMPNN in scTM and scRMSD. For a comparison of all benchmarked methods, see Appendix Table 6. In Figure 2b, we show the effect the number of iterative FAMPNN sampling steps on self-consistency. Performing only step of sequence design is insufficient to capture higher-order sequence inter- actions, resulting in worse relf-consistencyy but find that 10 steps is sufficient to achieve high self-consistency. Using smaller number of iterative steps significantly reduces in- ference time compared to fully autoregressive approach of L steps (c.g. ProteinMPNN and ESM-IF1), which requires one step per residue.
com-
5. Results
5.1. Full-atom sequence design
5.1.1. SEQUENCE RECOVERY AND SELF-CONSISTENCY
To benchmark FAMPNN's sequence recovery compared to other methods, report median sequence recovery the CATH 4.2 test set in Table 1. FAMPNN (0.0A) achieves a competitive 49.66% single-step sequence recovery, sur- passing the two modules, ProteinMPNN (45.96%) and GVP (39.47%), which make up the hybrid MPNN-GVP archi- tecture. We omit benchmarking of protein design models
5

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Table Sidechain packing performance comparison across CASP datasets. Bold values indicate the best results. Underlined values indicate second best. Methods marked with an asterisk (*) were trained on dataset that does not hold out CASP proteins.
a
Per-atom confidence
Per-residue confidence
Predicted sidechain (A)
Spearman p: 0..44
Spcarman 0.780
3.5
3.0
3.0
2.6
2.5-
Dataset
Method
Atom RMSD (A) &
2.0
All / Core/ / Surface
CASPI5
3DCNN*
0.897 / 0.489/ 1.003
ChromaDesign*
0.810 / 0.434 / 0.917
65 10 ba
15
20
25 20
as 4. 45
0.0 0.5
1.0
1.
85 20
25 K0 45
0.788 / 0..43/ / 0.879
True sidechein (A)
True sidachain error (A)
b
C
FlowPacker
0.765 / 0..52/ 0.859
FAMPNN (0.3A)
0.785 / 0.417 0.888
FAMPNN (0.0A)
0.690 / 0.350 0.789
5.2. Sidechain packing
Following FlowPacker (Lee & Kim, 2024), we compare our sidechain packing performance to other methods on CASP13, 14, and 15 targets (Table 2). Following Lee and Kim, we used MMseqs2 easy-search to remove all CASP13-15 homologues from our training and valida- tion dataset with a cutoff of 40% similarity. We measure sidechain packing performance by the average RMSD be- tween the predicted and ground-truth sidechains. Unlike most methods that predict sidechain coordinates via torsion angles (Anand et al., 2022; Zhang et al., 2024; Dauparas et al., 2023; Lee & Kim, 2024), our method uses an MSE- based objective in Cartesian space. This avoids the lever arm effect, where small angular errors cause large Cartesian deviations, and achieves superior RMSDs while remaining competitive on chi-angle accuracy (Appendix Table 8).
Figure 3. (a) Predicted sidechain error correlates with true sidechain packing error both per-atom (left) and per-residue by averaging over atoms (right). (b) Visualization of repacked tar- get from CASP15 (ID: T1145-DI). Sidechain color represents confidence, with blue representing low predicted error and red representing high predicted error. (c) Specific I'AMPNN-packe sidechains with per-atom confidence (colored). along with the ground truth conformation (gray).
Predicted sidechain error (A)
Spear man ø: 0.785
0.8
ScRMSD
0.4
0.-
We also demonstrate that PSCE serves as an effective confi- dence metric for sidechain packing. We evaluated pSCE on crystal structures from the CASP15 test set and found strong correlations in both per-atom errors (Spearman p = 0.843) and per-residue errors (Spearman o 0.780), as shown in Figure 3a. Figure 3b displays a visualization of con- be
0.0
0.2
0.4
0.6
0.8
1.0
1/2
Aligned sidechain RMSD (A)
Figure 4. Left, among designed sequences achieving scRMSD S 5A, our predicted sidechain error correlates well with the ac- tual RMSD between FAMPNN packed sidechains and those pre- dicted by AlphaFold2. when each sidechain is aligned in back- bone frame. Right, AlphaFold2 preedictiednss (blue) of designed sequences can closely match target backbone (gray), with pre- dicted sidechains closely matching FAMPNN-designed sidechains.
fidence on packed structure, where confidence meaningfully interpreted on a per-atom level. For example, we observe that sometimes, while the hydroxyl tip of a tyro- sine sidechain is predicted with high confidence, the model assigns lower confidence to the ring's rotational orientation (Figure 3c, top left).
1lphaFold22-reeditiionss
5.3. Full-atom conditioned protein fitness evaluation
Additionally, pSCE remains interpretable for designed se- quences predicted to fold close to the target backbone (scRMSD < 5Á). We find that pSCE correlates well with the RMSD between FaMeeN-ppccedd sidechains and Al- phaFold2 predictions, when sidechains are aligned to the backbone reference frame, suggesting its utility as a confi- dence metric for sequence design (Figure 4).
Current inverse folding, or structure-conditioneed sequence design models, have surprising capacity score mutant protein fitness zero-shot, without observing any fitness prop- erties during pretraining. Markedly, protein stability and protein- binding affinity are two properties where inverse folding models significantly outperform protein lan- guage models (Shanker et al., 2024; Notin et al., 2023;
6

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
a
Anlibody-Antigen binding affinity prediction
five antibody-antigen binding affinity datasets, and two ver- sions of a general protein-proted binding affinity dataset, against range of state-of-the-art unsupervised and super- vised methods for binding and stability prediction. We score the fitness of a mutation by masking the mutated position and sidechain, and taking the conditional likelihood of the mutant residue normalized by the conditional likelihood of the wild-type residue the mutant position. Here, the advantage of our method is the ability to score mutation conditioned not only on neighboring backbone and sequence context, but the full-atom structure of surrounding residues.
0.52
3.45
0:43
D:46
Spearman
0:22
6.54
G6
CR6261
CR9114
SVEMPIv2 binding affinity prediction
Duporstee Pezalee
To evaluate zero-shot performance for prediction of protein- protein binding affinity, we evaluate on SKEMPIv2, a widely used curation of experimentally measured binding affinities for thousands of sequence variants across hun- dreds of protein-proteii interactions totaling ~7000 data points. Shown in Figure 5a, FAMPNN significantly outper- forms unsupervised models on the entire SKEMPIv2 dataset and, notably, even supervised models on the test split from Bushuiev et al. (Bushuiev et al., 2024). To evaluate zero-shot prediction of protein stability, we evaluate on S669, Megas- cale, and FireProLDB, which are datasets of experimentally measured changes in protein stability (AAG) to diverse native proteins and are widely-used benchmarks for stability predictors. Shown in Figure 5b, FAMPNN modestly sur- passes both ProteinMPNN and ESM-IF, consistently across all three stability datasets. Once again, shown in Figure 5c, FAMPNN consistently outperforms state-of-the-arl un- supervised methods ProteinMPNN and ESM-IF1 on the therapeutically relevant task of antibody-antigen binding affinity prediction. These results demonstrate the utility of FAMPNN for the stabilization of proteins and enhancement of protein-protcin interaction.
Spearman
3.78
9.40
Premaarpee
Tremsio
SKEMPIv2
SKEMPIv2-TESS
Prote in stability prediction
FAMPANN
0.52
8.55
0.54
Spearman 0
0:52
3,47
0:45
0:45
0:32
5.21
6:22
S669
FiraProt
Megascale
Figure 5. Zero-shot performance of FAMPNN against state-of-the- art protein design methods in prediction of experimentally mea- sured protein fitness. (a) Prediction of A AG of binding of protein variants in the SKEMPIv2 dataset, reported with Spearman cor- relation. We additionally evaluate on a separate test subset of SKEMPIv2 recently curated to address significant data leakage issues in supervised models trained on previously proposed data splits (Bushuiev al., 2024). (b) Prediction of antibody-antigen binding affinity evaluated on five antibedy-anttggen binding affinity datasets (Shanker et al., 2024), reported with Spearman correla- tion. Antibody is shown in bold below the plot, and correspouding antigen target shown on x-axis (c) Prediction of protein thermosta- bility, ie. AAG of folding, across three experimental datasets of diverse proteins, reported with Spearman correlation.
5.4. Sidechain context improves performance
To evaluate whether full-atomic modeling improves se- quence design performance, we ablated the two sources of full-atom information during model training: (1) full-atom conditioning, and (2) sidechain packing. We report perfor- mance of our implementation of MPNN and MPNN-GVP, with additions of (1) and/or (2) to MPNN-GVP (FAMPNN is the addition of both [1,2]). We report mean sequence recovery on the CATH 4.2 test set, training with OÅ and 0.3 noise. Shown in Figure 6a, we see incorporation of the sidechain packing objective improves sequence accu- racy. Second, we see full-atomic conditioning also increases sequence accuracy, even more so when trained with 0.3A noise. Both FAMPNN and the baseline model improve with additional sequence context, but interestingly, FAMPNN shows markedly stronger scaling (6% and 15% improve- ments at 90% context versus 2% and 8% at 0% context) from access to sidechain structural information unavailable to backbone-only models.
Widatalla et al., 2023), which are trained with similar MLM or AR objectives, but with no structural conditioning. We hypothesize that inverse folding models, using backbone geometry, gain a limited, but nonetheless effective, induc- tive bias for predicting binding affinity and stability changes upon mutation. From this, we aimed to evaluate if the ad- dition of full-atomic context, and explicit modeling of the crucial sidechain interactions which often determine stabil- ity and binding affinity, can improve performance.
We evaluate FAMPNN (0.3A) on three stability datasets,
7

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
We also benchmark FAMPNN's s ability leverage both partial sequence and sidechain context improve both se- quence design and sidechain packing performance (Figure 6b, c). In Figure 6b, we find that along protein-protein interfaces, where modeling sidechain interactions are more critical, providing partial sidechain context with sequence improves accuracy over providing partial sequence context alone. Moreover. FAMPNN can more effectively leverage sidechain context compared to LigandMPNN, which can also optionally condition on sidechain atoms context. Furthermore, we show that FAMPNN can pack sidechains while conditioning on varying amounts of partial sequence or sidechain conformation context, with increasing context leading to better packing accuracy (Figure 6c).
Noise Level: DA
Notse Level: 0.SA
Conqustoe sucuracyy (%
Ban srAaFrSE
64-
40
28
26
10 20
50
03
10
40
28
Partial Sequenoo sidochain (%)
Partial sequence sidechain (%)
auquance accurazyy (5) 0
Sieechan pooking RMSD (/i
47A-
toiir als iitecaa
2.1
2.0
425
Ligitiiii 2.14
376-
350
13
Partial contoxt
Partial context (25)
Lastly, we analyze the effect of full-atom conditioning on prediction of protein fitness. Shown in Figure 7, we com- pare the average performance on fitness categories from Figure 5 of FAMPNN and a version of the model trained without full-atom conditioning and sidechain packing, but scaled to the same number of parameters. Here, we see improvements in fitness evaluation upon incorporation of full-atomic context in model training, specifically for the SKEMPIv2 dataset. Surprisingly, this does not translate to antibody-antigen binding prediction. This is likely due to the use of inaccurate input structures in this benchmark which potentially introduce adversarial structural context. (see Appendix G.2.5) Together, these results demonstrate that full-atom conditioning can serve as salient information for improved prediction of protein fitness.
Figure 6. (a) Mean sequence accuracy on CATH 4.2 test set for various ablations of model training, conditioning and architecture with OÅ (left) and 0.3 (right) of structural noise added during training. (b) Sequence accuracy at protein-protedn protein interfaces vs. provided partial sequence context for FAMPNN and LigandMPNN. Solid lines use both sequence and sidechain context; dashed lines use sequence context only. (c) Average sidechain packing RMSD on CASP targets vs. partial context. Black: packing sidechains for known partial sequence context. Blue: packing remaining sidechains given partial sidechain context.
Effect of full-atom context fitness evaluation
FAMPNN
MFNN-GVP
0.5
0.46
0.45
0.45
0.32
6. Discussion
Average
FAMPNN is full-atom protein sequence design method, which models the per-token distribution of a residue's 's dis- crete amino acid identity and its continuous sidechain con- formation. This distribution is learned with a combined cate- gorical cross-entropy and diffusion loss, and we demonstrate that learning to model 3D sidechain coordinates improves se- quence design performance. We outline an iterative process to jointly sample protein sequence and sidechain conforma- tion, demonstrating its utility in designing sequences with higher self- confisesst than in a single pass. We achieve sate--ff-the-art accuracy in sidechain packing RMSD, along with an accurate estimate of its own error, PSCE. Finally, we find that incorporation of full-atomic context improves se- quence design performance, notably more so the amount of structural context increases. This result underlies the significant improvement of FAMPNN over backbone-only models in protein fitness prediction, where we are often evaluating only single residue mutations, utilizing the rest of the full-atom structure as salient context.
0.2
0.1
SKEMPI
Stability
Antibadies
Figure 7. Ablation of full-atomic context on fitness evaluation per- formance. We report performance as spearman p correlation with experimental measurements averaged across 10 checkpoints from 100-190k steps, for cach category. We evaluate with the same FAMPNN (0.3A) model, but with and without sidechain the coor- dinates of non-mutated residues
to condition on ligands (Dauparas et al., 2023) and model their conformation (Anishchenko et al., 2024), would ex- pand the biological applications of this work to the design of sidechain ligand interaction networks present in many functional proteins and enzymes. Second, following previ- ous work that has demonstrated success in aligning inverse folding models to experimental data (Widatalla et al., 2024), FAMPNN can be similarly aligned to desired properties to further enrich design performance and fitness prediction. Lastly, because there are biological applications where a starting backbone structure is not available, future work will
While FAMPNN demonstrated the utility of sidechain con- ditioning and conformation modeling, adding the capacity
8

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
investigate how FAMPNN be modified to concurrently design protein backbone as well as sequence and sidechain conformation.
References
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T.,
Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Software
Code for FAMPNN available at https://github. com/ Com/richardshuail fampnn.
Ahdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O' O'Donnell, T. J., Berenberg, D., Fisk, I., Zanichelli, N., et al. Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and ca- pacity for generalization. Nature Methods, pp. 1-11, 2024.
Acknowledgements
We thank Tianyu Lu, Garyk Brixi, and Alex Chu for insight- ful discussions early in the project. We also thank Jin Sub Lee for providing the CASP datasets used for evaluation in (Lee et al., 2023). T.W acknowledges funding by the Stan- ford Graduate Fellowship. T.W and R.W.S acknowledge funding support from the NSF Graduate Research Fellow- ship (DGE-2146755), B.L.H. acknowledges funding sup- port from Are Institute. This research was funded in part by the Advanced Research Projects Agency for Health (ARPA- H) Agreement No. 1AY2AX000054 and D24AC00412- 00. Additional support to P.-S.H are from Merck Research Laboratories (MRL) Scientific Engagement and Emerging Discovery Science (SEEDS) Program, Stanford Medicine Catalyst, and NIH (R01GM147893) The views and conclu- sions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. We are grateful to Justine Yuan for help with illustrations and visual presentation.
A Akpinaroglu, D., Seki, K., Zhu, E., and Kortemme, T. Frame2seq: cruuctre-conditionns masked language mod- eling for protein sequence design.
Anand, N., Eguchi, R., Mathews, I. I., Perez, C. P., Derry, A., Altman, R. B., and Huang, P.-S. Protein sequence design with learned potential. Nature communications, 13(1):746, 2022.
Anishchenko, I., Kipnis, Y., Kalvet, I., Zhou, G., Krishna, R., Pellock, S. J., Lauko, A., Lee. G. R., An, L., Dauparas, J., DiMaio, F., and Baker, D. Modeling protein-small molecule conformational ensembles with chemnet. bioRviv, 2024. doi: 10.1101/2024.099 www.:4.......6....... 64868.
http://wwwwi.xxx....////.o wwtp/////////ooeeee//ee//ee/eeee ear1y/2024/093
URL 111/2222///2/2222//2/ 22/////2222222 09. 25, 614868.
Bennett, N. R., Coventry, B., Goreshnik, I., Huang, B.
Competing Interests
Allen, A., Vafeados, D., Peng. Y. P., Dauparas, J., Baek, M., Stewart, L., et al. Improving de novo protein binder design with deep learning. Nature Communications, 14 (1):2625, 2023.
B.L.H. acknowledges outside interest in Prox Biosciences as a scientific cofounder. All other authors declare no com-
peting interests.
Berman, H. M., Westbrook, J., Feng, Z., Gilliland, G., Bhat, T. N., Weissig, H., Shindyalov. I. N., and Bourne, P. E. The Protein Data Bank. Nucleic Acids Research, 28(1): 235-242, 01 2000. ISSN 0305-1048, doi: 10.1093/
Impact Statement
This work has direct implications for discovery and opti- mization of therapeutics, specifically antibodies engineered to have high binding affinity to drug targets. Furthermore, full-atom sequence design can be particularly useful for the design of enzymes with improved activity and stability, with potential beneficial environmental use cases such as plastic degradation (Tournier et al., 2023) and heavy metal sequestration (Luo et al., 2024). However, similar to other advancements, there are risks that this technology can be used to harm human health, specifically via production of hazardous biomolecules. Hence, we strongly support adher- ence to ethical and biosafety guidelines for machine learning in biology as those set forth by Responsible AI X Biodesign.
nar/28.1.22 1.235. URL https: /to///oiioo// http:/do..oogo///////// wrg. 1093/ nar 1.235...35..
Bryant, P., Pozzati, G., and Elofsson, A. Improved pre- diction of protein protein interactions using alphafold2. Nature Communications, 13(1): 2022. doi: 1031/ $41467-002-28865- URL https: //wii org/10. 1038/s41467-0222 228--65-....
Bushuiev, A., Bushuiev, R., Kouba, P., Filkin, A., Gabrielova, M., Gabriel, M., Sedlar, J., Pluskal, T. Damborsky, J., Mazurenko. S., and Sivic, J. Learning to design protein-protee interactions with enhanced gen- eralization, 2024. URL https: /arxiv. org/ /bbs/ 2310.18515.
9

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Chang, H., Zhang, H., Jiang, L.,, Liu, C.,, and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11315-1122, 2022.
Frank, C., Khoshouei, A., Fu, L.., Schiwietz, D., Putz, D., Weber, I.., Zhao, Z., Hattori, M., Feng, S., de Stigter, Y., Ovchinnikov, S., and Dietz, H. Scalable protein design using optimization in relaxed sequence space. Science, 386(6720):439-445 2024. doi: 10.1126/ science.adq1741. URL org/doi aal//bbs///0 10. 1126/1126/8 http://cienncccccccoe
Chu, A. E., Kim, J., Cheng, L., El Nesr, G., Xu, M., Shuai, R. W., and Huang, P.-.. An all-atom protein generative model. Proceedings of the National Academy of Sciences, 121(27):e231 5001200111222 2024.
Gao, Z., Tan, C., Chacón, P., and Li, S. Z. Pifold: To- ward effective and efficient protein inverse folding. arXiv preprint arXiv:2209. 2243,, 2022a.
Dahiyat, B. I. and Mayo, S. L. De novo protein design: fully automated sequence selection. Science, 278(5335); 82-87, 1997.
Gao, Z., Tan, C., and Li, S. Z. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXiv:2202.01077 2022b.
Dauparas, J., Anishchenko, I., Bennett, N., Bai, H., Ragotte, R. J. Milles, L. F., Wicky, B. I,, Courbet, A., de Haas, J., Bethel, N., et al. Robust deep learning-based pro- tein sequence design using proteinmpnn. Science, 378 (6615):49-56, 2022.
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q.. Deaton, J., Wiggert, M., et al. Simulating 500 million years of evolution with a language model. bioRxiv, pp. 2024 07, 2024.
Dauparas, J., Lee, G. R., Pecoraro, R., An, L., Anishchenko, I., Glasscock, C., and Baker, D. Atomic context- conditioned protein sequence design using ligandmpnn. Biorxiv, pp. 2023-12, 2023.
Holm, L. and Sander, C. Fast and simple monte carlo algo- rithm for side chain optimization in proteins: application to model building by homology. Proteins: Structure, Function, and Bioinformaties, 14(2):213-223, 1992.
Devlin, Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arX1v:1810.04805, 2018.
Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., and Rives, A. Learning inverse folding from millions of predicted structures. International conference on machine learning, pp. 8946-8970. PMLR, 2022.
Dieckhaus, H., Brocidiacono, M., Randolph, N. Z., and Kuhlman, B. Transfer learning to leverage larger datasets for improved prediction of protein stability changes. Proceedings of the National Academy of Sei- ences, (((6))e2314853221 2024, doi: 10.1073/pnas. 2314853121. URL https: http:/wwwwppppppp...oo..ap pnas.nns.......... wwooooogooo.oooooooooooooooooooo doi/ abs 111111001101......3.355122.
Ingraham, J., Garg, V., Barzilay, R., and Jaakkola, T. Gener- ative models for graph-based protein design. Advances in neural information processing systems, 32, 2019.
Ingraham. J. B., Baranov, M., Costello, Z., Barber, K. W.,
Wang. W., Ismail, A., Frappier, V., Lord, D. M., Ng- Thow-Hing, C., Van Vlack, E. R., al. Illuminating protein space with a programmable generative model. Nature, 623(7989):1070-10773 2023.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 12873-12883, 2021.
Jankauskaité, J., Jarcnaz-García, B., Dapkūnas, J., Fernández-Recio. J., and Moal, I. H. SKEMPI 2.0: an updated benchmark of changes in pro- tein-protein binding energy, kinetics and thermody- namics upon mutation. Bioinformatics, 35(3):462- 469, 07 2018. ISSN 1367-4803. doi: 10.1093/ bioinformatics/ty6655. URL https: http::::///ii..r/1.../ 0093/bioinfooo
Evans, R., O'Neill, M., Pritzel, A., Antropova, N., Senior, A., Green, T., Zidek, A., Bates, R., Blackwell, S., Yim, Ronneberger, O., Bodenstein, S., Zielinski, M., Bridg- land, A., Potapenko, A., Cowie, A., Tunyasuvunakool, K., Jain, R., Clancy, E., Kohli, P., Jumper, J., and Hassabis,
/bty635.
D. Protein complex prediction with alphafold-multiner bioRxiv, 2022. doi:
Jing, B., Eismann, S., Suriana, P., Townshend, R. J., and
URL https: http://wwwwww// wwwp::/wwwwwwvv.o.....o..o...e. www.://wwwwwwoooooooooeooo cooteett/////////////ooc/ce.c
Dror, R. Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.0144 0014011999 2020.
early/2022/03/11222 http//////11/2222222 10. 04. 10.4..46334....... 463034.
Fan, L., T., Qin. S., Li, Y. Sun, C., Rubinstein, M., Sun, D., He, K., and Tian, Fluid: Scaling autoregressive text- to-image generative models with continuous tokens, 2024.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. Highly accurate protein structure
URL https:ttp///c.c.....m / / /wwiiv.vivvv.////v//.../a/iwvvvw www.v..rg//.... org/ abs/ 2410. 13863.3863..
13863.
10

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
prediction with alphafold. nature, 596(7873):533-58, 2021.
Proceedings of the National Academy of Sciences, 120 (23):2216438112)) 2023.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of ceffusiin-bbsed generative models. Advances in neural information processing systems, 35: 26565-26577, 2022.
Miao, Z., Cao, Y., and Jiang, T. Rasp: rapid modeling of protein side chain conformations. Bioinformatics, 27(2)):
3117-3122, 2011.
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L.., Ovehin- nikov, S., and Steinegger, M. Colabfold: making protein folding accessible to all. Nature methods, 19(6)679-682, 2022.
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 24174-2418. 2024.
Misiura, M., Shroff, R., Thyer, R., and Kolomeisky, A. B, Dipacker: Deep learning for prediction of amino acid side chain conformations in proteins. Proteins: Structure, Function, and Bioinformatics, 90(6): 2022.
Knudsen, M. and Wiuf, C. The cath database. Human Genomics, 4(3):207, 2010. ISSN 1479-7364. doi: 10. 1186/1479-733444---220
Norn, C., Wicky, B. I., Juergens, D., Liu, S., Kim, D., Tischer, D., Koepnick, B., Anishchenko, I., Players, F,, Baker, D., Protein sequence design by confor- mational landscape optimization. Proceedings of the National Academy of Sciences, 118(11):2201222811 2021.
Koenig, P., Lee, C. V., Walters, B. T., Janakiraman, V., Stin- son, J.. Patapoff, T. W., and Fuh, G. Mutational landscape of antibody variable domains reveals a switch modulat- ing the interdomain conformational dynamics and anti- gen binding. Proceedings of the National Academy of Sciences, 114(4):F486-F495, 2017. doi: 10. 10.1033/pnas. 1613231114.
Notin, P., Kollasch, A. W., Ritter, D., van Niekerk, L.,
Paul, S., Spinner, H., Rollins, N., Shaw, A., Weitzman, R., Frazer, J., Dias, M., Franceschi, D., Orenbuch, R., Gal, Y., and Marks, D. S. Proteingym: Large-scale benchmarks for protein design and fitness prediction. bioRxiv, 2023. doi: 10.1101/2023.12....577222
Krivov, G. G., Shapovalov, M. V., and Dunbrack Jr, R. L. Improved prediction of protein side-chain conformations with sewrl4. Proteins: Structure, Function, and Bioinfor- matics, 77(4)777--795 2009.
URL https: http::/wwww.booxiz. wwtp://te///.t.oeo//eo/o./oo...e early her1ly222231222
Lee, J. S. and Kim, P. M. Flowpacker: Protein side-chain packing with torsional flow matching. bioRxiv, pp. 2024 07, 2024.
.2..3......0 02.07.577777
Pancotti, C., Benevenuta, S., Birolo, G., Alberini, V. Repetto, V., Sanavia, T., Capriotti, E., and Fariselli, P. Predicting protein stability changes upon single- point mutation: a thorough comparison of the avail- able tools on new dataset. Briefings in Bioinformat- ics, 23(2):bbab555, 01 2022. ISSN 1477-4054. doi: 10. 10.1093/bbb/bbab555. URL https: http://oo....g/...////////////// org/10. 1 1093/bib/bbabe
Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text- to-image models using human feedback. arXiv preprint arXiv:2302. /2192, 2023.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint 4406v 11838, 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, pp. 4195 4205, 2023.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary-scall prediction of atomic-level protein structure with a language model. Science, 379(6637); 1123-1130, 2023.
Phillips, A. M., Lawrence, K. R., Moulana, A., Dupic, T., Chang, J., Johnson, M. S., Cvijovic, I., Mora, T, Wal- czak, A. M., and Desai, M. M. Binding affinity land- scapes constrain the evolution of broadly neutralizing anti-influenza antibodies. eLife, 10:e7 1393, 2021. doi: 10.7554/eLife.7133
Luo, Y., Zhang, Y., Xiong, Z., Chen, X., Sha, A., Xiao, W., Peng, I.., Zou, I.., Han, J., and Li, Q. Peptides used for heavy metal remediation: A promising approach. Inter- national Journal of Molecular Sciences, 25(12):6717 2024. doi: 10.3 10.3390/jms251267111
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831. Pmlr, 2021.
McPartlon, M. and Xu, J. An end-to-end deep learning method for protein side-chain packing and inverse folding.

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Randolph, N. and Kuhlman, B. Invariant point message pass- ing for protein side chain packing and design. bioRxiv, pp. 2023-08, 2023.
Xu, J. and Berger, B. Fast and accurate algorithms for protein side-chain packing. Journal of the ACM (JACM), 53(4):533-537, 2006.
Ruffolo, J. A., Bhatnagar, A., Beazer, J., Nayfach, S., Russ, J. Hill, E., Hussain, R., Gallagher, J., and Madani, A. Adapting protein language models for structure- conditioned design. bioRxiv, pp. 2024-08, 2024.
Ye, F., Zheng. Z.,, Xue, D., Shen, Y., Wang. L., Ma, Y., Wang, Y., Wang, X., Zhou, X., and Gu, Q. Proteinbench: A holistic evaluation of protein foundation models, 2024a. URL https: arxiv org/ abs /2409.06744.
Shanker, V. R., Bruun, T. U. J., Hie, B. L., and Kim, P. S. Unsupervised evolution of protein and anti- body complexes with structure-informed language model. Science, 385(6704)446-33, 2024. doi: 10.1 science.adk89466. URL httttp::/www.scinnee / 10. http///cieec..a.ee..c...
Ye, F., Zheng, Z.,, Xue, D., Shen, Y., Wang. I.., Ma, Y., Wang, Y., Wang, X., Zhou, X., and Gu, Proteinbench: A holistic evaluation of protein foundation models. arXiv preprint arXiv:2409.0m 067444996644494 2024b.
Zhang, and Skolnick, J. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302-2309 2005.
Stourac, J. Dubrava, J., Musil, M., Horackova, J., Damborsky. J., Mazurenko, S., and Bednar, D. Fire- ProtDB: database of manually curated protein stability data. Nucleic Acids Research, 19911)4331--3324, 11 2020. ISSN 1305-1048. doi: 10. 10.1099/aar/gkaa0911.. URL https: / wwt.oooooroll..... 1093 nar gkaa981.
Zhang, Y., Zhang, Z., Zhong, B., Misra, S., and Tang, J. Diffpack: torsional diffusion model for autoregressive protein side-chain packing. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Sumida, K. H., Núñez-Franco, R., Kalvet, I., Pellock, S. J., Wicky, B. I. Milles, L. F., Dauparas. J., Wang. J., Kipnis, Y., Jameson, N., et al. Improving protein expression, stability, and function with proteinmpan. Journal of the American Chemical Society, 146(3):2054-23)) 2061, 2024.
Zheng, Z., Deng, Y., Xue, D., Zhou, Y., Ye, F., and Gu, Q- Structure-informed language models are protein design- ers. In International conference on machine learning. pp. 42317-42338, PMLR, 2023.
Sun, Y., Bao, H., Wang, W., Peng. Z., Dong. L., Huang, S., Wang, J., and Wei, F. Multimodal latent language modeling with next-token diffusion. arXiv preprint arXiv:2412.086333 2024.
Tournier, V. Duquesne, S., Guillamot, F., Cramail, H., Taton, D., Marty, A., and André, I. Enzymes' power for plastics degradation. Chemical Reviews, 1230))5511-5501, 2023. doi: 0..1021/acscchemre 2c00644. URL https::////... /ttpi///dioo/ooo//ooo/ org/ wtt.:/...122///11 1021/acs. PMID: 36916764.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL /tatts://arxiv.... abs /1766 03762.
Wang, J., Lisanza, S., Juergens, D., Tischer, D., Watson, L., Castro, M., Ragotte, R., Saragovi, A., Milles, L. F., Back, M., al. Scaffolding protein functional sites using deep learning. Science, 3776604))387-394 2022.
Widatalla, T., Rollins, 7.,, Chen, M.-T., Waight, A., and Cheng, A. C. Abprop: Language and graph deep learn- ing for antibody property prediction. The 2023 ICML Workshop on Computational Biology, 2023.
Widatalla, T., Rafailov, R., and Hie, B. Aligning protein generative models with experimental fitness via direct preference optimization. bioRxiv, pp. 2024-05. 05, 2024.
12

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
A. Benchmarks
A.1. Fitness Evaluation Benchmarks
Table 3. Prediction of protein-protein binding affinity and stability. Bold values indicate the best results. Underlined values indicate
second best.
Dataset
Method
Spearman Correlation
SKEMPI
ProteinMPNN
0.27
ESM-IFF
0.29
FAMPNN
0.39
SKEMPI-PPI
ProteinMPNN
0.41
ESM-IF1
0.18
FAMPNN
0.58
PPIFormer
0,44
GEMME
0.30
RDE Net
0.24
MSA Transformer
0.32
Megascale
ProteinMPNN
0.53
ESM-IF1
0.54
FAMPNN
0.55
FireProt
ProteinMPNN
0.50
ESM-IF1
0.47
FAMPNN
0.55
S669
ProteinMPNN
0.30
ESM-IF1
0.31
FAMPNN
0.41
CR9114-H3
ProteinMPNN
0.33
ESM-IF1
0.37
FAMPNN
0.42
CR9114-H1
ProteinMPNN
0,44
ESM-IFI
0.22
FAMPNN
0.53
CR6261-H9
ProteinMPNN
0.47
ESM-IF1
0.45
FAMPNN
0.55
CR6261-HI
ProteinMPNN
0.46
ESM-IF1
0.43
FAMPNN
0.56
G6
ProteinMPNN
0.13
ESM-IF1
0.28
FAMPNN
0.29
13

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Table 4. Comparison with supervised baselines for stability prediction.
Dataset
Method
Spearman Correlation
Pearson Correlation
Unsupervised FAMPNN
0.53
0.50
Megascale (Test Set)
Supervised
ThermoMPNN
0.73
0.75
Unsupervised
FAMPNN
0.55
0.53
FireProt
Supervised
ThermoMPNN
0.66
0.65
ProteinDPO
0.6
0.6
Unsupervised
FAMPNN
0.41
0.39
Supervised
MSAesm
0.54
MAESTRO
0.50
PROSTATA
0,49
ProteinDPO
0.47
ACDC-NN
0.46
ThermoMPNN
0.43
DDGun3D
0.43
INPS3D
0.43
INPS-Seq
0,43
S669
ACDC-NN-Seq
0.42
DDGun
0.41
PremPS
0,41
Dynamut
0.41
SDM
0.41
PoPMuSiC
0.41
DUET
0.41
RaSP
0.39
ThermoNet
0.39
I-Mutant3.0 .0
0.36
Dynamut2
0.34
B. Model training
B.1. Structural noise
Previous work (Dauparas et al., 2022) has shown that the addition of random Gaussian noise protein structure examples during training improves the self-consistency of model designs. In line with this, trained two versions of FAMPNN, trained with 0 and 0.3A noise, respectively. We add independent noise to each æ, and z coordinate by sampling from normal distribution scaled by factor of 0.3. Specifically, for each coordinate, the perturbation is drawn from (00 o²), with standard deviation a = 0.3.
B.2. Training details
The CATH trained FAMPNN models were trained on single NVIDIA H100 GPU with 80GB of RAM, with a batch size of 64 and fixed example size of 256 residues. The models were trained until 100k steps which required approximately 6-8 hours. The PDB trained models were trained on NVIDIA H100 GPUs with 80GB of RAM per GPU, with a batch size of per GPU and fixed example size of 1024 residues. To increase the effective batch size, 4 gradient accumulation steps
14

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
taken for each backpropagation step, bringing the effective batch size 128 examples. The models were trained until 300k steps which required approximately 3 days. For the PDB-trained models, we used the post-hoe EMA procedure from Karras et al. to choose EMA lengths after training was completed (Karras al., 2024). Based on combination of sequence accuracy and sidechain packing RMSD on the validation set, we chose an EMA length of I % for the 0.3À noise model and 25% for the 0.0A noise model.
B.3. Datasets
B.3.1. CATH
We utilized the CATH 4.2 (Knudsen & Wiuf, 2010) $40 dataset which is curation of domains extracted from the PDB with redundant domains (those with >40% homology) removed, with training, validation and test splits identical to Ingraham et al. (Ingraham et al., 2019). We set all examples to fixed length by either cropping examples larger than the given fixed size, or padding examples which are smaller than the given fixed size. Cropping done during training time by randomly sampling a single continuous span of the protein sequence. The start position of this crop selected uniformly random between the start of the sequence and the fixed size subtracted from the total sequence.
B.3.2. PDB
We used publicly available (Wang et al., 2022) reproduced training splits curated from the entire Protein Data Bank (Berman et al., 2000) used to train AlphaFold3 (Abramson et al., 2024). This dataset includes a cutoff date of 2021-09-30, such that no structures released after this date are included. This dataset was clustered on the chain level at 40% sequence homology for proteins. Then, interface-based clustering is performed as a join on cluster IDs of the corresponding chains in the interface, such that interfaces i and are in the same interface cluster C? interface only their constituent chain pairs {T, 12), J2] have the same chain cluster pairs {Crhain C.hain). Additional details regarding curation of this dataset have been previously described (Abramson al., 2024). To (1) prioritize training of the PDB model to learning to design multichain proteins and (2) ensure protein structure was provided with correct context, we removed every single-chain example which is present as a chain within a multichain example. Second, to maximize the amount of training data whilst preventing redundant training examples, we randomly sample a single example from each training cluster at each epoch. In line with training of AlphaFold- Multimer,,, we supplement our contiguous cropping strategy on the CATH dataset, with a previously described (Evans al., 2022) interface spatial cropping strategy to maximize interface coverage for multichain examples. Our strategy, described in Algorithm chains. Our spatial cropping strategy is identical to Alphafold- Multimerr. In line with (Evans et al., 2022) we randomly sample with equal probability whether to apply spatial cropping or contiguous cropping to given multichain example.
slightly simpler, given that our multichain examples contain only two
Algorithm 1 Multimer Contiguous Crop
Notation: 4, f2: chain lengths; k: desired crop size; m: binary mask; n: total length; cú crop size for chain é; sy start position for chain i
1: function MULTIMERCROP((,.
2: n < 11
3:
m + [0]"
Initialize mask
4:
C1 -- uniform(min(1;, maximiinnnnmmm(ikkk 12)), min(4, k) 1)
5:
e +- A --
6:
81 < uniform(0. n k)
7:
m|s1 s1 + +c] +1
8:
if c2 > then
9:
82 < uniform(s; + C1N n - 02)
10:
m[s2 : 82 + 02] 6-
end if
12:
return m
Binary mask m for selected residues
13: end function
15

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
C. Masked language modeling scheme
Because our method generates both sequence and sidechain coordinates in a per-token manner, we can view the full-atom sequence design procedure through generative masked language modeling framework. Each token is both a residue's sequence identity and its sidechain conformation. Similar MaskGIT and ESM3 (Chang et al., 2022; Hayes et al., 2024), train with randomly selected masking rates. Here, draw from concave schedule:
t= t= u Uniform (0,
where each residue in the sequence is kept with probability L. This prioritizes mostly unmasked tokens during training, focusing learning filling in missing tokens given surrounding context.
C.1. Per-token modeling of sequence and sidechains
Here, we describe how model residue's sequence and its sidechain conformation single token. Following MaskGIT's notation, let Y [wl be sequence of N tokens, with corresponding binary mask M = fr. where my if ys is masked. Define the masked set M = {i m; = 1) and its complement M of unmasked indices. The model learns to predict masked tokens conditioned on unmasked tokens:
Pe (YM M Y M-)-
For categorical tokens, this corresponds to standard masked language modeling (MLM) objective:
[MLM EY.M
Σ
log (y(: Y)))
iEM
In our problem, each token y, consists of both sequence identity 8 (categorical) and sidechain conformation x5; (continuous). We therefore decompose the token distribution
p(u Y M 2(
Y = p(s:
Y N pp(p 8i, Ya))
This factorization requires learning both:
1. Sequence prediction: p(s; Y xz) > trained with the MLM loss LMLM (Devlin, 2018).
2. Sidechain prediction: p(#; 8i, Y trained with the diffusion loss Lan from Li et al. (Li et al., 2024), while conditioning on 8;-
To learn both objectives simultaneously, we simply sum the losses, yielding total loss for the main model (excluding the confidence module):
Crocal LMLM + Lditt
We did not experiment with relative weightings of the losses on each objective.
D. FAMPNN architecture
D.1. Full-atom representation
In FAMPNN we represent residues in combination of the similar atom37 and atoml4 formats.
In the atom37 format, each residue is represented as fixed size matrix of size 37 x 3. Each row corresponds the 3D coordinates of all 37 possible atom types. 4 of these rows correspond to the backbone atoms N, Co, C, and O, while the remaining 33 rows correspond to the residue's 's sidechain atoms. For sidechains where particular atom type is not present, the row for that atom type is a "ghost atom" and is set to the residue's 's Ca position.
The at om14 4 format is very similar to at om37 but is condensed to only the maximum number of atom types a single amino-acid can possess. Thus, other than the backbone atoms N, Co., C, and O which are shared between all amino-acids,
16

---

end for Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Algorithm 2 Full-atom Encoder
Notation: X: full-atom coordinates; S: sequence; M: sequence mask; R: residue indices; C: chain indices; h: hidden dimension; kz number of neighbors, Fiaz: indices of connected nodes, Tine, Leg' number of invariant and equivariant layers respectively, Vnode: vector node features, Vedge: vector edge features, Snodei scalar node features, Sedge: scalar edge features.
1: function FULLATOMENCODERXX, 5, M, R, C,h. Linus Leg)
X ERRIIXX SSNN", S M E {0,
# Backhone structure featurization
2: Sedges Eide f InvariantBackoone Xbs, R, C)
3: hy + 0 nxh
4: bgg < Linear(Sedge)
Invariant backbone encoder layers
5: for - to Lins do
6: hv, he + MPNNEncoder(hyy, Eiaz, M)
7:
Full-atom structure and sequence featurization
8:
be + Concat(hu, aamnarr[vvarannuullAAeeeeeeeee R, C, Eida)]
9:
hg < EmbedSeq(S)
hES < ConcatNeighbors((s: hg, Eiaz)
11:
BESv + ConcatNeighbors (hv, [has: heal. Erdx)
Invariant full-atom encoder layers
12:
for / 1 Linn do
13:
hy, RESV MPNNDecoder(hy,,ha hesv M, E(de)
14:
end for
# Equivariant full-atom featurization
15:
Vnode GetUnitVectorss(> Vedge GetEdge Vectors(X, (X, Eida)
16:
Invariant full-atom featurization
17:
Snode GetDihedrals( X)
18:
Sedge <- GetDistances (X, Fidz)
# Embed GVP features
19:
han h/ + GVP V Vonee Snode) han l eq
20:
+- GVP Vedge: Sodge)
# Merge invariant features
21:
him Linear( (h + hv)
22:
E + Linear( (h + hv
Equivariant full-atom encoder layers
23:
for | = 1 to Leq do
24:
hinu hi hiww h + GVP(hi hig hin hag)
25:
end for
26: by Concat(hy R1 haro
27: L ---iearr(hr))
28: return L, hv
Sequence logits L 12 node embeddings hv
29:
end function
17

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
the atom type of the ith position in one residue's s atom 14 representation may not be the same atom type as another residue. Specifically, residue is represented maximum number of possible atom types single residue can possess. 4 of these rows correspond to the backbone atoms N, Ca, C, and O, while the remaining 10 rows correspond to the residue's 's sidechain atoms. As with at at.m37, for sidechains where a particular atom type is not present, the row for that atom type is set to the residue's Ca position.
fixed size matrix of size 14 x 3. Each row corresponds the 3D coordinates of the
We opt to use at atom14 4 for full-atom conditioning in the sequence design module, and atom37 for Euclidean sidechain diffusion.
D.2. MPNN-GVP full-atom encoder
We represent full-atom protein structure as a graph, encoded with a graph neural network (GNN) using a hybrid MPNN-GVP architecture (Dauparas et al., 2022; Jing et al., 2020). This architecture consists of three primary components, which are an invariant backbone encoder, an invariant full-atom encoder, and an equivariant full-atom encoder, with respect to any arbitrary composition of global rotations and reflections in protein coordinates.
The first two components build off the architecture of ProteinMPNN (Dauparas et al., 2022). ProteinMPNN featurizes backbone structure as a k-NN graph, with set of nodes V that represent each protein residue, and edges & = { which are defined for the 48 nearest neighbors of each residue by Co distance. All node representations V are initialized with Os, and edges in ع = { } are initialized with an encoding of distances between the C, Co, N, O and C,3, atoms of residues i, j in terms of Gaussian radial basis functions (RBF). ProteinMPNN includes an initial "structure encoder" trunk consisting of three previously described (Ingraham et al., 2019) invariant MPNN layers, which update both edge and node representations MPNNEncoder(U;, Ciy) (u', dgy). This output is passed sequence decoder comprised of three additional invariant MPNN layers. In addition to the output of the structure encoder, causally-masked one-hot encodings of sequence identity are concatenated to the edge representation E". The sequence decoder MPNNDecoder(e', Jely- s:]) = (u"",, then updates only node representations, which are passed into a final output head for sequence prediction.
Our initial component, the invariant backbone encoder, is identical to MPNNEncoder, encoding the backbone structure only. However, for our second component, the invariant full-atom encoder, we replace MPNNDecoder with a full-atom encoder, which is identical to the backbone encoder, but with expanded featurization to all atoms. To do this, for all & {e:- J. we concatenate an RBF encodings of the distances between the backbone atoms (C, Ca, N. O) of residue i and all neighboring atoms in residue j. Here, we compress the composition of neighboring atoms to the at 4 representation. Similar to MPNNDecoder, we concatenate one-hot encodings of sequence identity, but remove the causal mask, as FAMPNN is trained with an MLM objective, and sequence positions are randomly masked with a corresponding token.
The third and final component of the model, the equivariant full-atom encoder, consists of modified Geometric Vector Perceptrons (GVP) from Jing et al. GVP layers consist of an equivariant track for learning vector features and an invariant track for learning scalar features. Concretely, given tuple (s, V) of scalar features E R. and vector features V E R GVP computes new features (s', V') E ET x [8x33 In FAMPNN, in addition to the forward and reverse unit vectors in the directions of Ca... - Ca, and Ca.-1 Ca, used in Jing et al. as vector valued node features, we use the equivariant track to encode unit vectors from Co: to all other atoms in residue i. For the equivariant edge features eij. in addition to a unit vector in the direction Cay - C.,, we include unit vectors from C., to all atoms in residue j. again, in the compressed at om14 4 representation. The invariant track embeds the sine and cosine of the backbone dihedral angles of residue as node features, and sinusoidal encodings of the value j - i, as in Vaswani al. (Vaswani et al., 2023), and RBFs of the distance ||Cn, Ca,, ll edge features, previously described (Jing et al., 2020). We additionally incorporate distances from Ca, to all atoms in residue j.
We initialize the invariant- track of GVP with node and edge representation output from the invariant full-atom encoder. Given the sequence identity of a protein is invariant to global rotations and reflections, the final GVP layers return an invariant output by multiplying all vector features with the inverse rotation matrix R E g which defines the frame of the input structure. We then concatenate these outputs with the invariant scalar hidden state. This representation is then passed to both a final output head for sequence prediction, and the sidechain diffusion module for invariant Euclidean denoising of sidechain atoms.
18

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
D.3. Sidechain diffusion
D.3.1. INVARIANT SIDECHAIN REPRESENTATION
During sidechain diffusion, transformed the sidechain each residue into a local reference frame defined by its backbone atoms. To construct frames, used the OpenFold implementation of Algorithm 21 from AlphaFold2, which defines frame from 3 points (Ahdritz et al., 2024; Jumper et al., 2021). The pseudocode is reproduced here in Algorithm 3 for the reader's reference. To construct frame for each backbone, we use N as 3T, Ca as T2, and C as T3. Transforming each sidechain into its local backbone frame allows the sidechain diffusion process for each residue to be invariant to global rotations and translations of the input backbone.
Algorithm 3 Rigid from 3 points using the Gram-Schmidt process function RIGIDEROMS 72, T3)
2:
0 -3- F2
3:
022 - Ł1 32
4:
él - /
5:
12 + 1/2 6 (c
6:
é + ú2 T
7:
é - ı x 62
8:
R c 622 E3)
9:
ft 22
10: return (R, i)
1I: end function
D.3.2. DIFFUSION SCHEME
As discussed in Section 4.3.1, diffusion process uses the variance-exploding EDM scheme from Karras et al. (Karras et al., 2022). We computed data = 0.66 by taking the standard deviation of sidechain coordinates in their local frame from a random batch of 1000 examples from the training dataset. An overview of the parameters we used can be found in Table D.3.2. For sampling, we run a full trajectory using 50 steps diffusion with a step scale 1j = 1.5.
Parameter
Value
mmin
0.01
O muxx
80
data
0.66
7
P Pmean
1.5
Psdd
1.0
Table 5. Overview of EDM parameters used in this work.
Algorithm 5 Conditioned MLP Block
Notation: x: input tensor; c per-token conditioning;
1: function CONDTIONEDML.PBBOCCCC e)
# Pointwise feedforward with AdaLN conditioning
2: (3) Linear (inea)())LU(c)) (i)
Split into shift B, scale 7 gate > No learnable bias or scaling
3: Tskip <- LayerNorm(#4 (z)
4: Tskip <- Tskip x (1 + 7) +
5: Takip + Dropout Linear
MLP block
6:
+
Takip
7: return
8: end function
19

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Algorithm 4 Sidechain diffusion MLP
Notation: X noisy local sidechain coordinates; ey: node embeddings; 8j predicted one-hot encoded sequence;
noise level;
1: function SIDECHAINMLL((XS) U; 81, ())
2: Tj f- reshape( (X
Flatten atom and xyz dimension
Embed inputs
3: 2; < concat (Zi; 8i)
4: 2; < -ineer((z))
# Embed conditioning inputs
5: ambed f NoiseEmbedder((o)
6: Membed < Linear(e;) (4:)
7: G4 < 7embed + Cembed
Run blocks
8:
for = 1 to 4 do
9:
24 c ConditioncdMLPPlookiia q.)
Apply AdaLN pointwise feedforward
10:
end for
# Project back into coordinate space
11:
4 FFinalaayee c)
Unflatten atom and xyz dimension
12: X +- reehapp((z;)
13:
return Xpred
son
14: end function
Algorithm Final Layer
Notation: x: input tensor, conditioning tensor;
1: function FINALLAYER(1, c)
# Final projection with AdaLN conditioning
2:
(3,7) + Linear (S))
>> Split into shift B, scale No learnable bias or scaling
3:
x + LayerNorm (z)
4:
+ (1 + 7) 3
5:
+ Linear(x))
Project to final output
6: return
7: end function
D.3.3. SIDECHAIN DIFFUSION MLP
We parametrize the denoiser Do with multilayer perceptron (MLP) that uses adaptive layer normalization (AdaLN) for conditioning, following Li et al. (Li et al., 2024; Peebles & Xie, 2023). As conditioning input for residue i, we use node embedding i, the current noise level, and the predicted amino acid type for residue (Algorithm 4). To embed noise levels, we use the implementation the timestep embedder from Peebles et al., which uses 256-dimensionia frequency embedding followed by 2-layer MI. (Algorithm 4, Line 5).
Crucially, each of the conditioning inputs the sidechain diffusion MLP is invariant to global rotations and translations of the backbone. Furthermore, the sidechain coordinates fed into the MLP represented in local reference frame defined by the backbone atoms. Therefore, any global SE(3) transformation of the backbone leaves all inputs to the MLP unchanged. As result, each denoising step made with Dg is invariant to global rotations and translations of the input backbone.
D.4. Confidence module
As discussed in Section 4.3.2, at train time, we trained confidence module to predict per-atom sidechain errors given packed structure. This module is trained to take in an input sequence, predicted sidechains, and node and edge embeddings from the full-atom encoder to predict a per-atom error for all sidechains (Algorithm 7). Similar to AlphaFold3 (Abramson et al., 2024), during training, we run a diffusion rollout to generate sidechain coordinates, and we apply stop gradient to all confidence module inputs so that the confidence loss does not affect the training of the main model. For computational
20

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
efficiency, found it sufficient to train the confidence module roughly every training steps: in practice, each training iteration, sample Uniform(O,) 1) and only train the confidence module if S 1/8.
Algorithm 7 Sidechain Confidence Prediction
Notation: Xsen: predicted local sidechain coordinates; v;: node embeddings; eij: edge embeddings; s; predicted sequence;
1: function PREDICTPSCE(X Eij, s;)
Embed full-atom encoder outputs
2: for = 1 to 3 do
3: Vi, Gij < MPNNDecoder(e;; (eaj. s;])
4: end for
# Embed aatype and sidechain coordinates
5:
U, + + (s)
6:
Uj + + Linear Xwen)
Predict per-atom confidence
7:
Pt + Softmax (innear(SLLL(LLLiaa(()))
Predict per-atom bin probabilities Get bin centers D. Compute expected confidence per atom
8:
+ Linspace(0.06255 4.0625, 33)
PSCE + p, b
9:
10: return PSCE
11:
end function
D.4.1. PREDICTED SIDECHAIN ERROR (PSCE)
During training, we binned the true per-atom sidechain error in 33 evenly spaced bins between OÅ to 4A. The confidence module loss then computed categorical cross entropy loss between the confidence model output and the binned error.. At inference, we obtain single value for the predicted sidechain error by computing expectation across all bins using the bin centers (Algorithm 7, Line 8 and 9).
D.5. Sampling Speed
Regarding inference costs, when evaluating CATH trained model on the CATH 4.2 test set, achieve single-step and five-step sampling speed of 0.03s and 0. is per sample on single H100 GPU,
E. Self-consistency evaluation
E.1. Dataset
We follow after ProteinBench (Ye al., 2024b) and benchmark seff-consistency on a of RFdiffusion de novo backbones from lengths 100 500. For each length in {100, 200, 300, 400, 500), generated 100 samples from REdiffusion using the default parameters. For structure prediction, ColabDesign (Mirdita et al., 2022).
single-sequence AlphaFold2 with 3 recycles implemented in
E.2. Seff-consistency benchmarking details
A full table of sequence design methods that benchmarked de novo backbone dataset be found at Appendix Table 6. Following ProteinBench, use sampling temperature of 0.1 for all methods unless otherwise stated. We compared the proposed method against several sequence design baselines, using the official recommended hyperparameters wherever possible:
FAMPNN (0.3A, 0.0A): We used the PDB-trained FAMPNN model to sample sequences using 100 iterative unmasking steps. To minimize the risk of exposure bias during iterative sampling of sidechains, at each step, condition the model only on previously generated sidechains where the predicted sidechain error is less than 0.3A.
ChromaDesign: We used the publicly available weights from the official GitHub repository with the default sampling
21

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
Table Median self-consistency achieved by sequence design methods on 100 nove backbones generated by RFdiffusion each
length in {100, 200, 300, 400, 500 Bolded values indicate the best results.
Length 100
Length 200
Length 300
Length 400
Length 500
scTM (f)
PLDDT (1)
scTM (?)
PLDDT (1)
scTM (1)
PLDDT (1)
scTM (f)
pLDDT (f)
scTM (*)
pLDDT (1)
FAMPNN (0.3A)
0.968
93.00
0.967
91.27
0,938
83.45
0.760
74.73
0.545
61.73
FAMPNN (0.0A)
0.896
88.99
0.890
81.87
0.703
67.86
0.602
63.80
0.471
55.49
ProteinMPNN
0.964
92.45
0,960
89.65
0.935
85.66
0.784
76.81
0.578
62.36
ESM3 (LAB)
0.941
0.829
77.21
0.620
63.70
0.481
$8.89
0.419
$2.76
ESM-IFI
0.605
85.57
0,678
68.20
0.468
60.04
0.493
58.21
0.348
52.71
LM-Design
0.662
82.48
0,487
64.55
0.407
57.08
0.419
53.72
0.356
52.62
AttnPacker
0.586
85.43
0.563
59.06
0.463
50.95
0.414
48.00
0.371
44.65
settings (t=0.5).
ProteinMPNN: We used the 0.2À checkpoint @varilla.mddel.weeieee.. pling parameters their defaults.
with all other sam-
We used the publicly available ESM3-open model with 1.4B parameters, with number unmasking steps equal to the length of the target backbone.
ESM3:
ESM-IFI: We used the publicly available model weights and ran wes.quencessseeeeo py script from the official GitHub repository.
LM-Desigg: We used the CATH-trained ESM-2 650M version of the model from the official GitHub repository (Im_designneem2266500
AttnPacker:
We used the AttnPacker+Desigg model variant that can also condition sidechain rotamers from the official GitHub repository. Following (McPartlon & Xu, 2023), we sampled sequences from the model using Gibbs sampling with 20 transitions per design, a 15% re-sampling rate, and a linear temperature decay from 0,5 to 0.1.
E.3. Diversity
In addition to the self-consistency of designed sequences for the de novo generated backbones, we compared their diversity by reporting the average pairwise sequence similarity of the designs.
E.4. Sampling Time
Finally, regarding inference costs, when evaluating our CATH model on the test-set, achieve single-step sampling speed of 0.03s and 0. per sample on single H100 GPU.
Table Average pairwise similarity of PDB trained model on 100 de novo backbones generated by REdiffusion for each length in ( (100, 200, 300, 400, 500).
Length 100 Sim (L)
Length 200 Sim (L)
Length 300 Sim (L)
Length 400 Sim (L)
Length 500 Sim (L)
FAMPNN (0.3A)
13.4%
9.1%
8.4%
7.9%
7.8%
FAMPNN (0Å)
14.4%
10.1%
9.3%
8.8%
8.6%
ProtieinMPNN
13.9%
9.7%
9.0%
8.6%
8.4%
ChromaDesign
11.1% 1
9.2%
8.7%
8.4%
8.3%
ESM-IF1
16.9%
10.9%
11.0%
10.0%
9.5
ESM3
20.2%
15.4%
12.3%
12.6%
12.1%
F. Sidechain packing evaluation details
As mentioned in Section 5.2, following FlowPacker (Lee & Kim, 2024), we compared our sidechain packing performance to other methods on CASP13, 14, and targets and used MMseqs2 easy-sea rch to remove all CASP13-15 homologues
22

---

0.8 Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
from training and validation dataset with cutoff of 40% similarity. The CASPI3 and CASP14 test sets obtained directly from the official AttnPacker GitHub repository (McPartlon & Xu, 2023), and the CASP15 targets were downloaded from the CASP data archive.
For computing the average sidechain packing RMSD on a given dataset, we first computed the per-residue RMSD for each residue in a protein. Then, we averaged these per-residue RMSDs over all residues in the protein to obtain per-protein RMSD, Finally, we took the mean across all proteins followed AtunPcckers definitions: core residues had least 20 CB atoms within 10Å, while surface residues had most 15 CB atoms within 10Å.
dataset. For core and surface residue RMSDs (Table 2), we
5.5
4.5
4.0
RMSD (A)
3.5
3.0
2.5
2.0
1.5
1.0
0.0
H
K
Residue type
Figure Sidechain packing RMSD distributions per residue type, computed from the CASP15 test set.
Per-atom confidence
Per-residue confidence
Predicted sidechain error (A)
0.0
0.8
0.8
0.4
0.0%
0.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.8
1.0
True sidechain error (A)
True sidechain error (A)
Figure Predicted sidechain enor compared with true sidechain error,, both per-atom (left) and per-residue (right), zoomed in to emors within I
23

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
G. Fitness evaluation
G.I. Model comparisons
G.I.I. FAMPNN
We evaluate mutational fitness with FAMPNN by supplying the entire complex full-atom structure and sequence as imput. However, we mask the sidechain and sequence identity of a given mutated residue. Because FAMPNN can additionally perform sidechain packing, when evaluating a single-mutant within of simultaneous mutations, we pack the sidechains of accompanying mutants. We then sum the log likelihood of each mutant, and subtract from the summed wild-type likelihoods.
G. ESM-IF1
To run ESM-IF1, we followed instructions and code from the ESM-IFI Github (Hsu et al., 2022), summing the log- log-likelihood of simultaneous mutations given the entire complex backbone-structure and sequence as input. However we took the additional step of normalizing with the summed wild-type likelihood of said mutated residues.
G.1.3. PROTEINMPNN
For rumning ProteinMPNN we similarly provided the the entire complex backbone-structtur and sequence as input, using the score function provided in the LigandMPNN Github to attain log-likelihoods (Dauparas al., 2023). We then perform the same strategy as ESM-IFI which is summation across mutated positions, and normalization with summed wild-type likelihood.
G.2. Evaluation datasets
Unless specifically indicated, we report performance using an all vs. all calculation of the Spearman correlation between model predictions and dataset labels.
G.2.1. MEGASCALE
We used a curated, de-duplicated, version the entire Megascale dataset from Diekhaus (Dieckhaus al., 2024), combining the train, validation and test splits into single dataset given only unsupervised methods were evaluated. This resulted in final dataset size of 272,712 experimental AAG data points across 298 unique proteins.
G.2.2. SKEMPIV2
SKEMPIv2 is database of binding free energy changes upon single point mutations within a variety of protein complex interfaces (Jankauskaité et al., 2018). Due to inability of either ProteinMPNN, ESM-IF1, or all three models (FAMPNN included) to handle some examples with non-canonical amino-acids, examples were removed for a final dataset size of 6,649 data points. We additionally evaluate on a test subset of SKEMPIv2 recently curated in Bushuiev data leakage issues in supervised models trained on previously proposed data splits (Bushuiev et al., 2024). We also report performance of supervised and MSA-based models this dataset originally reported in Bushiev et al, with the exception of ProteinMPNN, which we independently evaluated, as this baseline was not included.
to address significant
G.2.3. FIREPROTDB
The FireProt database a curated dataset of changes in free energy (A AGG for 3,438 single mutations for 100 unique proteins (Stourac et al., 2020). Due to inability of either ProteinMPNN, ESM-IFI, or all three models (FAMPNN included) handle some examples with non-canonical amino-acids, 18 examples were removed for a total of 3420 examples.
G.2.4. S669
The S669 dataset contains experimentally measured AAG values of 669 single mutations of 94 proteins (Pancotti et al., 2022). Due to the presence of non-canonical amino acids, 4 variants were removed from this dataset.
24

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
G.2.5. ANTIBODY-ANTIGEN BINDING AFFINITY
The CR9114 (HI,H3), CR6261 (HI,H9) and G6 (VEGF-A) antibody-antigen binding affinity datasets were sourced from Shanker et al, from which details on dataset availability and the imput structures used can be found (Shanker et al., 2024). The CR9114 dataset includes all possible combinations of 16 amino-acid substitutions, whereas CR6261 includes all possible combinations of 11 amino-acid substitutions, totaling 65,536 and 2,048 sequences respectively. Each of these libraries were screened against two subtypes of Influenza Hemagluttinin producing 4 total datasets (Phillips et al., 2021). The third dataset, referred to as G6, assesses all possible single-aming amino acid substitutions to the variable region of antibody G6.31, totaling 4,275 data points of binding with VEGF-A (Koenig et al., 2017). Due to experimental structures being unavailable for each -nntiggenn pair, homologous protein structures were used for some predictions, following Shanker et. al (Shanker al., 2024). For the CR91 14 dataset, the input structure used is CR9114 14 bound to the H5 antigen, despite being used to predict binding to HI and H3 antigen subtypes. Additionally, for the CR6261 dataset, the input structure for predicting binding with both H1 and H9 subtypes, is actually structure of CR6261 complexed with H1.
25

---

Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN
CASP datasets, measured by absolute (MAE) of each chi angle. Bold values indicate the best results. Underlined values indicate second best results. Asterisks denote that the training dataset does not explicitly hold out CASP proteins and homologues. Results for Rosetta, DiffPack (Zhang al., 2024), Attpacker and AttmPacker-pp (McPartlon & Xu, 2023) are from Lee and Kim (Lec & Kim, 2024). All other models were evaluated in this work.
Table Sidechain packing performance comparison
Dataset
Method
Angle MAE (*) &
RMSD (Å)
X1
X2
X3
X4
All
CASPI3
Rosetta
24.84
30.96
45,35
58,28
0.822
DiffPack
22.14
28.80
45.20
52.28
0.789
AttnPacker
17.82
33.41
67.31
48.89
0.745
AttnPacker-pp
16.33
26.00
51.18
49.40
0.676
3DCNN
24.74
33.03
46.96
56.73
0.787
ChromaDesignt
17.61
26.14
43.76
55.05
0.677
LigandMPNN#
16.19
23.96
38.46
49.02
0.680
FlowPacker
16.58
25.04
43.09
51.41
0.683
FAMPNN (0.3A)
20.61
26.01
45.31
50.33
0.667
FAMPNN (0.0A)
14.94
25.07
41.05
48.89
0.579
CASP14
Rosetta
32.32
35.47
49.19
54.27
1.001
DiffPack
31.00
34.43
51.72
57.50
0.994
AttnPacker
27.29
39.26
67.94
49.99
0.955
AttnPacker-pp
26.06
32.75
55.06
50.59
0.900
3DCNN
31.56
36.50
51.29
60.57
0.952
ChromaDesign' *
25.24
31.47
47.27
55.94
0.851
LigandMPNN*
22.06
28.98
43.52
47.36
0.825
FlowPacker
23.09
30.26
47.46
51.14
0.838
FAMPNN (0.3A)
26.02
31.24
47.78
53.08
0.821
FAMPNN (0.0A)
21.53
30.72
47.96
52.21
0.745
CASPI5
Rosetta
32.37
35.22
45.29
58.92
0.938
DiffPack
31.86
34.46
48.06
61.01
0.921
AunPacker
28.16
41.90
69.90
53.22
0.925
AttnPacker-pp
26.80
32.74
56.78
53.98
0.851
3DCNN*
32.12
36.53
50.43
62.37
0.897
ChromaDesign
26.34
31.72
44.44
59.83
0.810
LigandMPNN*
23.32
28.97
41.16
51.87
0.788
FlowPacker
23.63
29.26
42.97
53.35
0.765
FAMPNN (0.3A)
27.96
31.19
45.33
54.92
0.785
FAMPNN (0.0Å)
21.75
29.49
43.07
55.41
0.690
26


Reviews:
[{"review_id": "IGpfrgggDx", "reviewer": "Reviewer_GPV6", "summary": "The paper introduces FAMPNN for protein sequence design that explicitly models both the sequence identity and sidechain conformation of each residue. Unlike existing methods that rely solely on backbone geometry, FAMPNN uses a combined categorical cross-entropy and diffusion loss objective to jointly learn the distribution of amino acid identities and sidechain conformations. The authors demonstrate that this approach improves sequence recovery and achieves state-of-the-art sidechain packing accuracy. Additionally, FAMPNN shows promise in practical applications, such as zero-shot prediction of protein stability and binding affinity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their suggestions to strengthen our baselines and evaluations. Below, we address the raised concerns:\n\n> Inclusion of more recent sequence recovery baselines\n\nWe thank the reviewer for pointing out this omission. To this end, we report Frame2Seq (Dec. 2023), a structure-conditioned masked protein language model achieve 46.53% sequence recovery as a single model, and 49.11% as an ensemble of three models on the CATH 4.2 test set with a single step (Akpinaroglu et. al), while FAMPNN achieves a 49.66% and 50% recovery as a single model with  single and 5-step sampling respectively. Second, some other baselines could not be included due to differences in training data: e.g. certain methods do not train on CATH and elect to train on non-standardized versions of the PDB, thus we cannot compare sequence recovery on a common test set. Third, certain recent baselines for sequence recovery, particularly hybrid models which utilize pre-trained protein language models such as ESM2 (e.g. LM-Design, KW-Design), may have data-leakage issues with the CATH 4.2 test set, as pretrained protein language models likely have trained on sequences in the CATH test set. We will include a discussion of this in the final version of the paper if accepted. Finally, in this paper, we note that we do not claim state-of-the-art performance in terms of sequence recovery, but rather that sidechain packing and sequence design are synergistic tasks to learn, where we can achieve self-consistency competitive with ProteinMPNN while achieving state-of-the-art packing.\n\n> High sequence recovery ≠ good design\n\nWe completely agree that high sequence recovery does not equal good design. To this end we use the de novo benchmark, popularized by ProteinBench (Ye et. al), as a way of measuring the ability to recapitulate structures on structures that are structurally distinct and diverse from those found in nature and report self-consistency to capture if designs “fold into the target structure” as you mention. As these proteins are not found in nature, we hope this demonstrates generalization beyond memorization. In fact, we find very low average pairwise similarity between sequences generated by FAMPNN on this benchmark with **[13.4%, 9.1%, 8.4%, 7.9%, 7.8%]** average pairwise similarity for de novo backbones of length 100, 200, 300, 400, and 500 respectively. We will conduct a more robust benchmark of diversity on de novo backbones and compare to other methods in the final version of the paper if accepted.\n\n> Training and inference costs\n\nWe completely agree that compute costs are important for a useful protein design tool. Our CATH and PDB models are competitive with state-of-the-art models at **8.6M** and **10.7M** parameters respectively, within magnitude of models such as ProteinMPNN (1.9M), Frame2Seq (10M), and PiFold (6.6M). Notably, our model is much smaller compared to models such as ESM-IF (142M), LM-Design (664M), and ESM-3 (1.4B). For training, we mention in Section B.2 that we trained our CATH model for **~8 hours** on a single H100 GPU with 80GB of memory. The PDB model was trained with 4 H100s for 72 hours, but in practice we notice that the model can reach similar performance in **~24 hours** (96 GPU hours). Regarding example size, training of our PDB dataset was conducted with examples cropped or padded to a large fixed size of 1024 residues, with 82.91% of examples being multi-chain complexes. Finally, regarding inference costs, when evaluating our CATH model on the test-set, we achieve a single-step sampling speed of **0.03s** and **0.11s** per sample on a single H100 GPU. Additionally, we are enthusiastic to add a detailed comparison of inference costs to other methods to the final version of the paper. We appreciate this point being, as we have found that because we use a MLM procedure, FAMPNN can sample equally high quality sequences with much fewer steps, making our method much faster for longer proteins than other methods, including ProteinMPNN.", "claims_and_evidence": "yes.", "methods_and_evaluation_criteria": "yes.", "theoretical_claims": "no theoretical contributions.", "experimental_designs_or_analyses": "yes.", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The explicit modeling of sidechain conformations during sequence design is a significant advancement over existing methods that only consider backbone geometry. This is a clear improvement, as sidechain interactions are crucial for protein stability and function.\n2. The use of a diffusion loss for sidechain conformation prediction is well-justified. It allows the model to handle continuous data (sidechain coordinates) effectively. \n3. The paper provides a thorough evaluation of FAMPNN across multiple benchmarks, including sequence recovery, sidechain packing, and protein fitness prediction. The results are strong. The inclusion of zero-shot prediction tasks (e.g., protein stability and binding affinity) is particularly compelling.\n\n**Weaknesses**\n1. In terms of sequence recovery, Table 1 lacks of recent and stronger baselines. This omission weakens the claim of state-of-the-art performance. Include 2023–2024 methods or clarify why they were excluded. \n2. High sequence recovery ≠ good design. The field prioritizes novel, stable sequences that fold into the target structure, not just matching native sequences. The lack of diversity and novelty metrics leaves it unclear if FAMPNN avoids overfitting. This is not only a practical desire for protein design but also an important measure about how and what your model learns, capable of generalization or simply through memorization. Authors however lack of related results and discussions.\n3. Training/inference costs (e.g., GPU hours, memory) are not quantified, raising concerns for scaling to large proteins or multi-chain complexes.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JJfa7I2FzQ", "reviewer": "Reviewer_CPZt", "summary": "This paper presents FAMPNN, an iterative inverse folding algorithm capable of co-generating sidechain conformations and sequences. \nSuch design allows the model to condition on the currently known sidechain atoms in addition to the fixed backbone and sequence.\nFAMPNN models the per-residue sequence type and sidechain structure with a combined cross-entropy and diffusion loss objective.\nExperimental results show that FAMPNN can achieve promising results in full-atom sequence design, sidechain packing, and full-atom conditioned protein fitness evaluation. They also validate the effectiveness of the sidechain context.\n\n## update after rebuttal\nI've read the authors' replies and other reviewers' comments. I will keep my positive score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We'd like to thank the review for their positive evaluation and interest in the methodological choices behind our model. Below, we clarify these points:\n\n> Ghost atoms for masked tokens\n\nTo encode the ghost atom for masked tokens, we set them at the position of the central CA atom by default. Because the model receives a mask sequence token for this position, it is able to understand that these ghost atoms refer to a masked position.\n\n> MLM vs. AR\n\nIn early versions of the model, we experimented with both AR and MLM, finding that they perform similarly in terms of both sequence recovery and self-consistency. We noticed that high-quality sequences could be predicted in relatively few steps, so viewing the generative process through an MLM framework could allow for faster inference. We also chose MLM because it gives us more flexibility in choosing train-time masking schedules, allowing the model to prioritize learning on certain masking levels.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "Yes, the experimental designs are sound as they benchmark the proposed method on 3 related tasks and also perform an additional ablation study.", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to next-token modeling for continuous data.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n1. sound experiment design and results\n2. They show that the FAMPNN can perform protein fitness evaluation quite well. We may scale up the training data (with some af2 distillation data) and model size to get even better results.", "other_comments_or_suggestions": "NO", "questions_for_authors": "1. In section 4.2.1, the authors mention the definition of the ghost atom. However, it is not clear how they encode the ghost atoms for masked tokens.\n\n2. FAMPNN adopts an MLM framework for inverse folding instead of random permutation AR or diffusion. I'm curious whether the MLM framework is optimal for the inverse folding task. Have you ever tried using a random permutation AR (proteinmpnn) for FAMPNN?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I9dv2yhTx6", "reviewer": "Reviewer_MU8A", "summary": "This paper introduces FAMPNN (Full-Atom MPNN), a model that explicitly incorporates sidechain conformation modeling for fixed-backbone protein sequence design. While existing deep learning methods implicitly reason about sidechain interactions based solely on backbone geometry and amino acid sequence, FAMPNN jointly models both sequence identity (discrete) and sidechain conformation (continuous) for each residue using a combined categorical cross-entropy and diffusion loss objective, respectively.\n\nBuilt on a hybrid MPNN-GVP architecture that conditions on both backbone and available sidechain information, FAMPNN employs an iterative sampling strategy to efficiently generate samples from the joint distribution of sequence and structure. The model achieves competitive sequence recovery on CATH 4.2 and strong self-consistency results compared to state-of-the-art methods like ProteinMPNN when evaluated on de novo backbones. FAMPNN also demonstrates superior sidechain packing accuracy on CASP13/14/15 datasets and provides per-atom predicted sidechain error estimates that strongly rank-correlates with true errors. Additionally, the authors show FAMPNN's effectiveness for unsupervised fitness prediction on experimental datasets for antibody-antigen binding, protein-protein binding affinity, and protein stability.\n\nThrough comprehensive analysis, the authors demonstrate that increasing sidechain context (both sequence identity and conformation) leads to better model performance. Their ablation studies reveal that a similarly sized model without sidechain context performs worse for protein-protein binding affinity prediction, though it closely matches FAMPNN's performance for antibody-antigen binding affinity and protein stability prediction.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comprehensive evaluation and suggestions to improve clarity and benchmark comparisons in our paper. Below, we provide detailed responses and clarifications:\n\n> Inclusion of perplexity in addition to sequence recovery for evaluation\n\nThis is a great suggestion. With 1 step recovery, FAMPNN achieves a perplexity of 4.99, and we will update this with the perplexities of all methods in Table 1 in a revised version.\n\n> Evaluation on ProteinGym\n\nWe agree with the reviewers point and will include ProteinGym in our evaluations if accepted. We would also like to note that many of the assays included in our fitness evaluations make up a great amount (but of course not all) of ProteinGym data (eg. Megascale, SKEMPIv2).\n\n> Interval estimates\n\nWe thank the reviewer for pointing this out and believe it would be a valuable addition. We intend to add interval estimates for performance metrics in the final version if accepted. \n\n> Expected monotonic relationship between context and packing accuracy\n\nWe believe that the slight deviations from this trend are due to random chance, and we plan to update this plot with error bars as well as clarify this claim in the text.\n\n> Selection of benchmark models\n\nIn general, we chose to benchmark models that are commonly used for their respective tasks. We agree that ESM3 should be included in the zero-shot fitness prediction evaluations, and we will include this in the final version if accepted.\n\n> Novelty of our work in the context of Chroma\n\nAs correctly pointed out, Chroma does in fact have the capability to return an all-atom structure given an input backbone via its ChromaDesign module. However, the sequence design module does not have the ability to encode a full-atom protein structure and therefore cannot design sequences conditioned on neighboring sidechain context. As a consequence, this means that **Chroma's sequence predictions are unable to leverage either experimentally determined sidechain conformations or previously predicted sidechains**. By contrast, FAMPNN **explicitly conditions on sidechain context**, which allows users to provide known sidechain conformations as conditioning input during sequence generation. This allows for more fine-grained control. Despite this, we very much agree it would be good to discuss Chroma and benchmark ChromaDesign along with other methods. We will provide updates in a revised version and have included an extension to Table 5 below:\n\n| Method        | Length 100     | Length 200     | Length 300     | Length 400     | Length 500     |\n|---------------|----------------|----------------|----------------|----------------|----------------|\n|               | scTM \\| pLDDT  | scTM \\| pLDDT  | scTM \\| pLDDT  | scTM \\| pLDDT  | scTM \\| pLDDT  |\n| FAMPNN (0.3Å) | 0.968 \\| 93.00 | 0.967 \\| 91.27 | 0.938 \\| 83.45 | 0.760 \\| 74.73 | 0.545 \\| 61.73 |\n| FAMPNN (0.0Å) | 0.896 \\| 88.99 | 0.890 \\| 81.87 | 0.703 \\| 67.86 | 0.602 \\| 63.80 | 0.471 \\| 55.49 |\n| Chroma        | 0.940 \\| 90.94 | 0.949 \\| 88.04 | 0.946 \\| 86.73 | 0.914 \\| 80.02 | 0.751 \\| 71.50 |\n\nWe also provide an extension to Table 2 for sidechain packing evaluation below, noting that Chroma has not been trained on a dataset to explicitly hold out CASP homologues:\n\n| Dataset | Method        | Atom RMSD             |\n|---------|---------------|-----------------------|\n|         |               | All / Core / Surface  |\n| CASP13  | FAMPNN (0.3Å) | 0.667 / 0.362 / 0.775 |\n|         | FAMPNN (0.0Å) | 0.579 / 0.345 / 0.659 |\n|         | Chroma*        | 0.677 / 0.392 / 0.770 |\n| CASP14  | FAMPNN (0.3Å) | 0.821 / 0.534 / 0.937 |\n|         | FAMPNN (0.0Å) | 0.745 / 0.430 / 0.858 |\n|         | Chroma*        | 0.851 / 0.550 / 0.964 |\n| CASP15  | FAMPNN (0.3Å) | 0.785 / 0.417 / 0.888 |\n|         | FAMPNN (0.0Å) | 0.690 / 0.350 / 0.789 |\n|         | Chroma*        | 0.810 / 0.434 / 0.917 |\n\n\n> Potential mismatch for antibody-antigen binding dataset evaluation\n\nYes this is true. As described in Shanker et. al, experimental structures matching the exact sequence are not available for each assay, however as the sequences of the different HA variants (H1,3,9 etc.) are quite close, models are still able to perform well using the backbone of the homologous proteins. While this can bring about errors, and we do acknowledge this, this is an extremely common occurrence in practical protein design, where homologous structures must be used due to a lack of experimental structure (homology modelling). Thus, we see this as an opportunity to showcase the model's performance in a structurally data limited situation which is even more prudent to demonstrate given our method uses “more” structure than its backbone-only counterparts.\n\n> Other comments or suggestions\n\nWe thank the reviewer for suggestions on improving the clarity of our figures and will make these changes if accepted.", "claims_and_evidence": "- **Claim**: pSCE serves as an effective confidence metric for sidechain packing\n  While this claim is supported by high Spearman correlation in Figure 3(a), there is miscalibration between predicted and true sidechain error. In the per-residue setting, the maximum pSCE is around 2.0 in contrast to the maximum true error of 4.0. This limits its utility as an absolute error predictor, though it remains useful for ranking/relative confidence.\n\n- **Claim**: FAMPNN outperforms unsupervised methods for antibody-antigen binding affinity and protein stability\n  This claim is supported by Figure 4(a) and 4(c) when FAMPNN is compared to other structure-conditioned models. The authors should also provide performance comparison with leading sequence-only foundation models or qualify in text that the comparison is limited to structure-conditioned models. It's also unclear why the comparison doesn't include other state-of-the-art methods like ESM3.\n\n- **Claim**: Increasing context leads to better sequence packing accuracy \n  This is more or less true but the relationship between context and packing accuracy is not monotic, strictly speaking. There is a nominal increase in RMSD around 40% context for partial sidechain context and a similar increase around 50% context for partial sequence context which should be clarified or explained in the text.", "methods_and_evaluation_criteria": "**Strengths:**\n- The formulation of the training objective considering both discrete sequence tokens and continuous sidechain coordinates is appropriate for the problem at hand.\n- The proposed method re-uses existing components (MPNN, GVP) that have been proven to be effective.\n- The selected datasets and metrics are well-suited to the problem being addressed. In particular, evaluating sequence recovery and self consistency on CATH 4.2 is a standard practice in the field.\n- A diverse set of datasets covering antibody-antigen binding, protein-protein binding affinity, and protein stability are used for zero shot evaluation of fitness prediction.\n\n**Weaknesses:**\n- Sequence recovery aggregates binary decisions for correct sequence identity without taking into account the precise confidence of the model for the correct residue. Perplexity addresses this shortcoming, however, it is not included in the evaluation.\n- ProteinGym (https://proteingym.org/) is a standard benchmark in the field for zero-shot fitness prediction that is not included in the evaluation making it difficult to compare performance with other methods not part of authors' evaluation.", "theoretical_claims": "No theoretical claims are made in this paper.", "experimental_designs_or_analyses": "**Strengths:**\n- Comprehensive benchmarking and analysis of sequence recovery, self-consistency, sidechain packing, zero-shot fitness prediction, and the impact of available sidechain context on performance.\n- Selection of a wide range of appropriate baseline models for comparison.\n- Rigorous ablation study is performed to investigate the primary claim of the paper about the utility of sidechain context.\n- The authors ensure there is no data leakage in the evaluation by holding out validation/test data from training and use updated splits for datasets like Skempi to ensure fair comparison.\n  \n**Weaknesses:**\n- The authors do not provide interval estimates for performance metrics which makes it harder to evaluate significance.\n- The selection of benchmark models varies across evaluations without clear justification. For instance, ESM3 is part of the self-consistency evaluation but not the zero-shot fitness prediction evaluation.\n- A notable exclusion from the paper is the Chroma generative model (https://www.nature.com/articles/s41586-023-06728-8) which also addresses the problem of incorporating sidechain conformation into fixed-backbone sequence design using similar architectural components. It would have been nice to see performance comparison with this model.\n- For antibody-antigen binding datasets, the authors note they used homologous structures when exact experimental structures weren't available (e.g., using CR9114 bound to H5 to predict binding to H1 and H3 subtypes). This potential mismatch might introduce errors in evaluation.", "relation_to_broader_scientific_literature": "The primary contribution of FAMPNN is the explicit modeling of sidechain conformation during fixed-backbone sequence design. This is a known limitation of existing methods widely used in the field such as ESM-IF1, ProteinMPNN, etc. The authors show that addressing this limitation leads to improved performance along various axes. However, the novelty of the work is unclear in the context of Chroma (https://www.nature.com/articles/s41586-023-06728-8) which also addresses the problem of incorporating sidechain conformation into fixed-backbone sequence design and has overlapping ideas.", "essential_references_not_discussed": "The most notable exclusion from the paper is the Chroma generative model (https://www.nature.com/articles/s41586-023-06728-8; Nature 2023) that addresses the problem of incorporating sidechain conformation into fixed-backbone sequence design amongst other things and has overlapping ideas.", "other_strengths_and_weaknesses": "The paper is well written and easy to follow. The authors provide the key implementation details, descriptions of datasets along with preprocessing details, and describe the evaluation criteria in detail.", "other_comments_or_suggestions": "Typos:\n- Missing year of publication for Akpinaroglu et al. in the citation in text. (e.g. page 1, first column, line 49)\n- The citation for RFdiffusion is missing in text.\n- Page 7, second column, lines 341-362: The references to Figure 5 sub-figures in the paragraph appear incorrect.\n- Page 23, line 1251: \"provided the the\"\n\nSuggestions:\n- Please mention the number of unmasking steps used for generating Figure 2a.\n- Please consider remaking Figure 3a to include Pearson correlation as well as marginal distribution of true and predicted sidechain errors.\n- Please mention the size of different datasets used for evaluation in section 5.3 in the main text and the kind of structure (predicted, crystal, etc.) available for each.\n- Please consider remaking Figure 6a to share the y-axis range for the two subplots.", "questions_for_authors": "1. Could the authors clarify the novelty of their work in the context of Chroma (https://www.nature.com/articles/s41586-023-06728-8; Nature 2023) and provide a more detailed performance comparison? This would help in understanding the unique contributions of FAMPNN and allow for a more accurate assessment of the paper's impact.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LE7ZMxgyFD", "reviewer": "Reviewer_mv2D", "summary": "The paper presents a new model FAMPNN for fixed-backbone protein sequence design that models both sequence and sidechains. FAMPNN addresses the limitations of existing methods that rely solely on backbone and sequence identity. The authors demonstrate that FAMPNN improves sequence recovery, achieves state-of-the-art sidechain packing, and can be used for zero-shot prediction of binding and stability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments and questions. Below, we address these questions in detail:\n\n> FAMPNN sampling time comparisons\n\nPlease refer to the last section of the response to reviewer GPV6\n\n> Standard deviation in the predictions\n\nPlease refer to the first section of the response to reviewer MU8A\n\n> Supervised baselines for Figure 5\n\nThis is an excellent suggestion, for **5c** there are baselines available with formal test splits for stability datasets on which we can evaluate FAMPNN to compare to supervised methods. Supervised baselines and formal test splits do not exist for **5a**, but we can run our own baseline (eg. one-hot-encoding of sequence with a ML-based regressors). If accepted, we will include these additional results.\n\n> ProteinMPNN self-consistency on longer lengths\n\nThis is an interesting hypothesis that we also explored. We believe that side-chain prediction error may indeed be compounding for longer-length proteins. At each sampling step, we condition only on previously-predicted side chains with predicted error < 0.3Å, and notice a larger performance degradation without this filtering. \n\n> Why explictily predict sequence?\n\nWe agree that given a full-atom representation, it is trivial to predict the sequence as it is already defined by the atom composition. In early versions of the model, we experimented with removing explicit sequence prediction and calling sequence from the implicit full-atom predictions, but we found this to not work as well in practice. We believe that the cross-entropy loss objective is important for the model to learn to predict high-quality sequences. Additionally, in addition to careful tuning of our masking schedule, and occasionally completely dropping all side-chain coordinates, FAMPNN is able to predict sequences with no full-atom representation, allowing use in situations such as sequence design for de novo backbones. \n> Multi-chain sequence design with FAMPNN\n\nPlease refer to the rebuttal to reviewer GPV6 for details on the training of the PDB model and the respective dataset. Regarding multi-chain design, we showcase the capability of the PDB model for multi-chain design in Figure **6b**, where we demonstrate the utility of full-atom context for higher sequence recovery in protein-protein interfaces. Here, FAMPNN achieves higher sequence recovery, and higher sequence recovery scaling with context than LigandMPNN. Additionally, we note FAMPNN performs state-of-the-art in binding-affinity fitness evaluation in large multi-chain protein-protein complexes greater 3000 residues (SKEMPIv2 dataset), which are shown in figures **5a** and **5c**. Together these results demonstrate robust utility of FAMPNN in multi-chain design.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "no theoretical claims", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "By integrating sequence and sidechain modeling in a single model, the paper extends prior work (e.g. ProteinMPNN which focuses on predicting protein sequences based on backbone structure without sidechain modelling) for improve protein design.", "essential_references_not_discussed": "the authors could expand more on recent all-atom approaches for protein generation (e.g. Protpardelle, ProteinGenerator etc.) and co-folding (AF3).", "other_strengths_and_weaknesses": "Strengths:\n* The paper is well written and handles an interesting and challenging problem of joint sequence and sidechain conformation prediction. The proposed model is novel and well motivated.\n* The authors provide extensive experimental results, demonstrating the effectiveness of FAMPNN across multiple benchmarks (zero-shot prediction of protein fitness, side chain packing, sequence design). The inclusion of the sidechain confidence module adds interpretability to the model's predictions. \n\nWeakness:\n* The authors could include a comparison in sampling time of FAMPNN compared to backbone-only models like ProteinMPNN to measure the computational cost of predicting sidechains.", "other_comments_or_suggestions": "/", "questions_for_authors": "* What is the standard deviation in the predictions ? Given that the process is the outcome of diffusion process, there might be some variance in the prediction.\n* Section 5.3; is it possible to compare on Figure 5.a and 5.c. these results with a supervised baseline to provide a reference point ? \n* Can the authors comment why in Table 5. ProteinMPNN is working better for higher lengths (400, 500); is this due to the error in side chain coordinate predictions ? \n* more generally, why is there a need for predicting sequence in the model given that the full atom representation contains implicitely information on the sequence ? \n* could the authors comment on the application of FAMPNN to multi chain sequence design problems ? I gather from Section 4.3 that the model can be applied to complexes.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a new protein sequence design method called FAMPNN (Full-Atom MPNN) by jointly generate both the sequence identity as well as the side chain conformations, which is very novel compared with existing sequence-only generation method. Experimental results show that the proposed approach improves both sequence recovery and also achieves state-of-the-art performance for sidechain packing. All reviewers agree that the paper presents a sound contribution to the field of protein sequence design, and hence the AC recommends the paper for acceptance.
Decision: Accept (poster)