SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We consider the zero-mean multivariate Gaussian whose inverse covariance matrix  $\Sigma^{-1} = L + aI$  corresponds to a sparse weighted Laplacian matrix  $L$ . Intuitively,  $L_{ij}$  should be zero if the position of the point  $i$  is independent of that of  $j$  given the positions of the neighbors of  $i$  being fixed. The term  $a$  is a small constant whose role is to shift the spectrum of  $L$  and thus make the Laplacian invertible.
We follow the framework of [39] and optimize  $L$  by minimizing the following objective:
$$
\min  _ {L \in \mathcal {L}} \operatorname {t r} \left(\left(X ^ {\text {p o s}}\right) ^ {T} L X ^ {\text {p o s}}\right) + f (L),
$$
where the  $m \times 3$  matrix  $X$  contains the 3D positions of the  $m$  points modeled and  $f(L)$  are some optional constraints we can enforce on the graph Laplacian (and thus also to the adjacency matrix  $A$ ).
The log-likelihood of vectors  $X$  on the graph defined by  $L$  can be directly expressed as a function of the graph adjacency matrix and the pairwise distance matrix  $Z$ :
$$
\operatorname {t r} \left(\left(X ^ {\text {p o s}}\right) ^ {T} L X ^ {\text {p o s}}\right) = A \odot Z = \frac {1}{2} \sum_ {i, j} A _ {i j} \| X _ {i} ^ {\text {p o s}} - X _ {j} ^ {\text {p o s}} \| _ {2} ^ {2}.
$$
Kalofolias [39] propose the following optimization objective, to recover the optimal adjacency, which with a logarithmic term also ensures that every node has at least one neighbor, which is something we also desire:
$$
\min _ {A \in \mathcal {A}} \| A \odot Z \| _ {1} - \mathbf {1} ^ {T} \log (A \mathbf {1}) + \frac {1}{2} \| A \| _ {2} ^ {2}.
$$
An efficient primal-dual algorithm exists for solving this optimization problem [39]. The adjacency matrix that we recover using the mean distance matrix over all of the proteins in paired OAS can be seen in Figure 4. Our use of AHo numbering allows us to directly estimate such mean distances, as with our representation every antibody effectively possesses the same number of residues.
To increase the stability of the Cholesky decomposition  $L = KK^T$ , which is needed for sampling co-variate noise  $X_{t}^{\mathrm{pos}} = \alpha_{t}X_{0}^{\mathrm{pos}} + \sigma_{t}K\epsilon, \epsilon \sim \mathcal{N}(0,1)$ , we instead use the generalized Laplacian  $L = \operatorname{diag}(A1) - A + I$  for the diffusion process.
Figure 4: We define a Gaussian prior on the atom positions by learning an adjacency (conditional dependence) matrix for antibody backbone atom positions from all of the folded paired heavy and light chains in the Observed Antibody Space (OAS) [61]. Dependencies between framework residue atom positions and even correlations between heavy (top-left) and light (bottom-right) chain atom positions are distinctly captured.
G Proof of SE(3) universality
In the following, we prove the SE(3) universality of an MLP-mixer backbone model combined with frame-averaging.
Concretely, let  $X$  be an  $n \times m$  matrix with bounded entries and consider the backbone model:  $\phi(X) = c_{L} \circ r_{L-1} \circ \dots \circ c_{3} \circ r_{2} \circ c_{1}(X)$ , where the subscript denotes layer,  $c_{l}$  is an MLP operating independently on each column of its input matrix, and similarly,  $r_{l}$  is an MLP operating independently on each row of its input.
It is a consequence of Theorem 4 in [68] that if  $\mathcal{F}$  is a bounded  $G$ -equivariant frame and if  $\phi$  is a backbone model that is universal over any compact set, then combining  $\phi$  with frame-averaging over a frame-finite domain (i.e., a domain within which the frame cardinality is finite) yields a  $G$ -equivariant universal model  $\langle \phi \rangle_{\mathcal{F}}$ . Since the  $\mathrm{SE}(3)$  frame is bounded and thus the domain is frame-finite (see also Proposition 1 [68]), to show that  $\langle \phi \rangle_{\mathcal{F}}$  is a universal approximator of any continuous  $\mathrm{SE}(3)$  function, it suffices to show that the MLP-mixer is a universal approximator over continuous functions over any compact set (or simply universal).
To proceed, we will show that there exists a parameterization so that  $\phi(X) = c_L([z; \mathrm{vec}(X)])$  for some vector  $z$ . We can then rely on the universal approximation property [23] of MLPs to conclude that  $\phi$  is also universal.
We first provide an exact constructive argument on how this can be achieved:
1. Let  $V$  be a  $(n + 1) \times (n + 1)$  unitary matrix whose first column is the constant vector  $\mathbf{1}$  and set  $U = V_{\cdot,1}$ : to be the  $n + 1 \times n$  submatrix. The first MLP embeds each column vector  $x$  isometrically into  $n + 1$  as follows:  $c_1(x) = Ux$ . By construction, every column of the output  $X'$  is now orthogonal to  $\mathbf{1}$ .
2. The network appends  $U$  to the left of  $X'$  and  $(UX; \dots; UX)$  to the right. Since appending to the right involves replicating each row multiple times, it can be done by a linear row operation. Appending to the left is slightly more involved and can be achieved by iteratively building  $U$  column by column, with each column added by using a row MLP to add one dimension with a constant value and a column MLP to project the new constant vector introduced to the appropriate column of  $U$ . The key here is that, since we have guaranteed that the columns are always orthogonal to the constant vector, the column MLP can always distinguish the newly introduced column from the ones already at play. Let  $X''$  be the result of this process.
3. A column MLP projects each column of the  $(n + 1)\times (n + nm)$  matrix to  $n$  dimensions by multiplying by  $U^{\top}$ . The output of this layer is  $X^{\prime \prime \prime} = [I;X;\dots ;X]$ .
4. A row MLP zeros out all elements except those indexed by the one-hot encoding in the first  $n$  entries. The latter can be done by a ReLU MLP with one hidden layer: if all entries of our input are smaller than  $b$  and the input is (onehot(i), x), the first linear layer adds  $b$  to only the entries  $[n + in : n + in + n)$  and shifts every element by  $-b$  using its bias. The latter lets the correct elements pass the ReLU with an unchanged value while setting all other elements to zero, as required. Let the output of this layer be  $X^{\prime \prime \prime \prime}$  with
$$
X _ {i,:} ^ {\prime \prime \prime \prime} = (I _ {i,:}; \overbrace {0 ; \cdots 0} ^ {\times i - 1}; X _ {i,:}; \overbrace {0 ; \cdots 0} ^ {\times n - i}).
$$
5. A column MLP sums up all entries. The output is now a row vector  $(\mathbf{1};X_{1,:}^{\top};\dots ;X_{n,:}^{\top})^{\top}$  of length  $n + nm$ , as required.
Steps 1 and 2 of the above constructive argument are inefficient because they rely on very simple layers. This can be avoided if we utilize more powerful MLPs (i.e., functions  $c_{l}$  and  $r_{l}$ ) and an existence argument. Specifically, as there exists a continuous function that achieves step 2, by the universality of MLPs there also exists an MLP that approximates the step arbitrarily well. Similarly, rather than increasing the dimension to  $n + 1$  at step 1, one may argue that the MLP can assign some vector outside of the bounded domain where the input data resides, and thus the dimension can remain  $n$ . Therefore, in this setting, only a constant number of MLPmixer blocks are needed to obtain the desired output.
H Noise Schedule
In our denoising model, we use the cosine-like schedule proposed by Hoogeboom et al. [28], with a 1000 denoising steps:
$$
\alpha_ {t} = (1 - 2 s) \cdot \left(1 - \left(\frac {t}{T}\right) ^ {2}\right) + s,
$$
where a precision  $s = 1e^{-4}$  is used to avoid numerical instability. As  $\sigma_t = \sqrt{1 - \alpha_t^2}$  and  $\beta_{t} = \alpha_{t}^{2}$ , it is sufficient to define just  $\alpha_{t}$ . Following Nichol and Dhariwal [59] we clip  $\alpha_{t|t-1}$  to be at least 0.001, to avoid instability during sampling and then recompute  $\alpha_{t}$  as a cumulative product. To further ensure numerical stability, as recommended by Kingma et al. [41] we compute the negative log SNR curve  $\gamma(t) = -\log \alpha_t^2 + \log \sigma_t^2$ . This parameterization allows for numerically stable computation of all important values [41], for example,  $\alpha_t^2 = \text{sigmoid}(-\gamma(t))$ ,  $\sigma_t^2 = \text{sigmoid}(\gamma(t))$ ,  $\sigma_{t|t-1}^2 = -\text{expm1}(\text{softplus}(\gamma(t-1)) - \text{softplus}(\gamma(t))) = -\text{expm1}(\gamma(t-1) - \gamma(t))\sigma_t^2$ , and  $\text{SNR} = \exp(-\gamma(t))$ . Where  $\text{expm1}(x) = \exp(x) - 1$  and  $\text{softplus}(x) = \log(1 + \exp(x))$ .
I Full Training and Sampling Algorithms
For convenience, in Algorithms 1 and 2 we describe the full training and sampling procedures.
Algorithm 1 Training
1: repeat  
2: Get a data point:  
3:  $X_0^{\mathrm{pos}}, X_0^{\mathrm{seq}} \sim q(X_0^{\mathrm{pos}}, X_0^{\mathrm{seq}})$   
4: Sample some time step:  
5:  $t \sim \mathrm{Uniform}(1, \dots, T)$   
6:  $X_t^{\mathrm{pos}} \sim \mathcal{N}(\alpha_t X_0^{\mathrm{pos}}, \sigma_t^2 \Sigma)$   
7:  $X_t^{\mathrm{seq}} \sim \mathrm{Categorical}(X_0^{\mathrm{res}} Q_t)$ , here  $Q_t = \beta_t I + (1 - \beta_t) Q$   
8: Predict:  
9:  $\hat{X}_0^{\mathrm{pos}}, \hat{X}_0^{\mathrm{seq}} = f_\theta$ . (Project  $(X_t^{\mathrm{pos}}), X_t^{\mathrm{seq}}, t$ )  
10:  $\hat{X}_0^{\mathrm{pos}} = \mathrm{Project}\left(\hat{X}_0^{\mathrm{pos}}\right)$   
11: Compute losses:  
12:  $L_t(\hat{X}_0^{\mathrm{pos}}) = \frac{1}{2} \mathrm{SNR}(t) \| \hat{X}_0^{\mathrm{pos}} - X_0^{\mathrm{pos}} \|_{\Sigma^{-1}}^2$   
13:  $L_t(\hat{X}_0^{\mathrm{res}}) = \beta_t \text{CrossEntropy}\left(\hat{X}_0^{\mathrm{res}}, X_0^{\mathrm{res}}\right)$   
14: Take a gradient step on:  $\nabla_\theta\left(L_t(\hat{X}_0^{\mathrm{pos}}) + L_t(\hat{X}_0^{\mathrm{seq}})\right)$   
15: until converged
Algorithm 2 Sampling
1: Initialize:  
2:  $X_{T}^{\mathrm{pos}} \sim \mathcal{N}(0, \Sigma)$   
3:  $X_{T}^{\mathrm{seq}} \sim \mathrm{Categorical}(Q)$   
4: Loop over the time steps:  
5: for  $t = T, \dots, 1$  do  
6: Predict:  
7:  $\hat{X}_{0}^{\mathrm{pos}}, \hat{X}_{0}^{\mathrm{seq}} = f_{\theta}\left(\mathrm{Project}\left(X_{t}^{\mathrm{pos}}\right), X_{t}^{\mathrm{seq}}, t\right)$   
8:  $\hat{X}_{0}^{\mathrm{pos}} = \mathrm{Project}\left(\hat{X}_{0}^{\mathrm{pos}}\right)$   
9: Sample next state:  
10: if  $t > 1$  then  
11:  $X_{t-1}^{\mathrm{pos}} \sim \mathcal{N}\left(\frac{\alpha_{t|t-1}\sigma_{t-1}^{2}X_{t}^{\mathrm{pos}} + \alpha_{t-1}\sigma_{t|t-1}^{2}\hat{X}_{0}^{\mathrm{pos}}}{\sigma_{t}^{2}}, \frac{\sigma_{t|t-1}\sigma_{t-1}}{\sigma_{t}}\Sigma\right)$   
12:  $X_{t-1}^{\mathrm{seq}} \sim \mathrm{Categorical}\left(\frac{X_{t}^{\mathrm{res}}Q_{t|t-1}^{T} \odot \hat{X}_{0}^{\mathrm{res}}Q_{t-1}}{\mathcal{Z}}\right)$   
13: else  
14:  $X_{0}^{\mathrm{pos}} \sim \mathcal{N}(\hat{X}_{0}^{\mathrm{pos}}, \frac{\sigma_{1}^{2}}{\alpha_{1}^{2}}\Sigma)$   
15:  $X_{0}^{\mathrm{seq}} \sim \mathrm{Categorical}\left(\hat{X}_{0}^{\mathrm{seq}}\right)$   
16: return  $X_{0}^{\mathrm{pos}}, X_{0}^{\mathrm{seq}}$ .
J Metrics Used in the Numerical Experiments
The metrics applied to generated sequences include methods that measure naturalness, similarity to the closest extant antibody, and stability. To examine Naturalness we use the inverse perplexity of the AntiBERTy [74] model trained on a large corpus of antibody chain sequences. As shown by Bachas et al. [6] this score correlates with antibody developability and immunogenicity. To explore Closeness we use fast sequence alignment [9] to determine the closest sequence in the training set. The mean sequence identity (fractional edit distance) to the closest training sequence is then reported as a score. Here Stability is operationalized by estimating the error of the folded structure using IgFold [73]. This error can be used to rank the sequences by how stable we expect their folds to
be. We take the 90th percentile of the residue error as an estimate of the sequence fold stability; the latter typically corresponds to the CDR H3 and L3 loops, which have the most influence over the antibody's functional properties. To further verify that the generated antibody sequences satisfy appropriate biophysical properties, we rely on four additional structure-based metrics Raybould et al. [69]: CDR region hydrophobicity (CDR PSH), patches of positive (CDR PPC), and negative charge (CDR PNC), and symmetry of electrostatic charges of heavy and light chains (SFV CSP). These metrics operate on folded sequences (using IgFold) and take into account distance-aggregated structural properties of CDR regions (and their spatial vicinity), and their significant deviation from reference values is typically associated with bad developability properties of antibodies such as poor expression, aggregation, or non-specific binding [69].
To evaluate the overall similarity of the generated and the test-set distributions, for all of these sequence metrics, we report the Wasserstein distance between the scores of the sequences in the test split and the scores of the generated sequences.
The metrics applied to the generated structures focus primarily on scores known to correlate with free energy and RMSD. a) We use free energy  $\Delta G$  estimated using Rosetta [2] to evaluate the stability of the generated structure. Although lower energy is normally associated with higher stability of the protein structure, one needs to be careful not to reward disproportionally small energies achieved when a miss-formed protein collapses into a morph-less aggregate. Thus, we again report the Wasserstein distance between the generated and test energy distributions. b) RMSD. We also re-fold the generated sequence with IgFold [73] and report the RMSD for the backbone  $N,C_{\alpha},C,C_{\beta}$  and  $O$  atom positions. RMSD is reported as a mean over the generated structures as it captures how well each generated structure matches its sequence.
K Detailed Generated Structure RMSD Evaluation
Model	Full ↓	Fr ↓	Fr. H ↓	CDR H1 ↓	CDR H2 ↓	CDR H3 ↓	Fr. L ↓	CDR L1 ↓	CDR L2 ↓	CDR L3 ↓
EGNN	9.8231	9.3710	9.2929	13.1720	13.0032	10.3360	9.3918	14.6768	10.1584	10.4860
EGNN (AHo)	10.0628	9.4717	9.3552	13.1730	13.4611	12.2434	9.5314	15.3884	10.6975	11.0732
EGNN (AHo & Cov.)	9.4814	8.7581	8.6206	12.9454	13.2237	12.0939	8.8174	15.2841	10.0504	11.1167
FA-GNN	0.8617	0.5748	0.5093	0.6671	0.7438	2.2530	0.6157	0.8199	0.5946	1.1576
FA-GNN (AHo)	0.8321	0.4777	0.4618	0.6881	0.7867	2.2884	0.4860	0.9398	0.5053	1.1165
FA-GNN (AHo & Cov.)	0.8814	0.5934	0.5236	0.5968	0.6213	2.0788	0.5966	0.7907	0.4521	1.3536
AbDiffuser (uniform prior)	0.8398	0.5937	0.5742	0.7623	0.6705	1.8365	0.6095	0.8825	0.4795	1.0698
AbDiffuser (no projection)	11.1431	11.0062	10.8279	13.8692	14.4139	10.4367	11.1709	15.7536	11.5205	11.2404
AbDiffuser (no Cov.)	0.6302	0.4011	0.3826	0.4946	0.5556	1.6553	0.4169	0.5585	0.4321	0.8310
AbDiffuser	0.5230	0.3109	0.2862	0.3568	0.3917	1.5073	0.3322	0.4036	0.3257	0.7599
AbDiffuser (side chains)	0.4962	0.3371	0.3072	0.3415	0.3768	1.3370	0.3637	0.3689	0.3476	0.8173
Table 4: Detailed RMSD for generated antibodies based on Paired OAS dataset [61]. The top three results in each column are highlighted as First, Second, Third.
Model	Full ↓	Fr ↓	Fr. H ↓	CDR H1 ↓	CDR H2 ↓	CDR H3 ↓	Fr. L ↓	CDR L1 ↓	CDR L2 ↓	CDR L3 ↓
MEAN [43]	0.7792	0.3360	0.3045	0.4569	0.3359	2.9053	0.3645	0.4425	0.2490	0.6862
EGNN (AHo & Cov.)	9.2180	8.7818	8.5527	12.0018	12.5770	10.0308	8.9396	14.2269	9.5391	10.4077
FA-GNN (AHo & Cov.)	3.1800	3.3761	1.8529	0.6446	0.5223	2.0202	4.2721	0.5633	0.5376	3.4047
AbDiffuser	0.3822	0.2186	0.1669	0.3611	0.2737	1.1699	0.2610	0.1937	0.2006	0.6648
AbDiffuser (side chains)	0.4046	0.2686	0.2246	0.3861	0.3115	1.1191	0.3073	0.2242	0.2379	0.7122
AbDiffuser (τ = 0.75)	0.3707	0.2138	0.1615	0.3541	0.2709	1.1210	0.2563	0.1830	0.1946	0.6615
AbDiffuser (s.c., τ = 0.75)	0.3982	0.2729	0.2277	0.3914	0.2917	1.0624	0.3127	0.2492	0.2548	0.7131
AbDiffuser (τ = 0.01)	0.3345	0.2000	0.1463	0.3389	0.2723	0.9556	0.2430	0.1530	0.1792	0.6582
AbDiffuser (s.c. τ = 0.01)	0.6795	0.6168	0.5938	0.8161	0.7113	1.1550	0.6396	0.7938	0.7048	0.8395
Table 5: Detailed RMSD for generated antibodies based on Trastuzumab mutant dataset [55]. The top three results in each column are highlighted as First, Second, Third.
Here, in Tables 4 and 5 we provide per-region distances between folded and optimized structures of the generated sequences and the generated structures. As you can see these distances correlate well with the overall RMSD reported in the main text. As the folded structures are optimized, we also investigate how the error changes if similarly to folding models we also use Rosetta [2] to optimize the generated structures in Tables 6 and 7. As can be seen, the optimization slightly reduces the RMSDs, but the relatively modest change for AbDiffuser-generated structures hints at the fact that
the generated structures were relatively physically plausible. In Table 5 we see that while the model that generates the side chains achieves a worse overall RMSD, in most cases it models CDR H3 more precisely, which is the most important part function-wise. This seeming focus on CDR H3 might explain why this model achieved a better predicted binding probability, even while modeling the overall structure slightly worse.
Model	Full ↓	Fr ↓	Fr. H ↓	CDR H1 ↓	CDR H2 ↓	CDR H3 ↓	Fr. L ↓	CDR L1 ↓	CDR L2 ↓	CDR L3 ↓
EGNN	9.8129	9.3487	9.2647	13.2206	13.1699	10.4327	9.3722	14.8368	10.1526	10.6565
EGNN (AHo)	10.1364	9.5233	9.3961	13.3611	13.7014	12.4793	9.5918	15.6919	10.8115	11.3710
EGNN (AHo & Cov.)	9.5411	8.8202	8.6761	13.0186	13.3938	12.1843	8.8849	15.4368	10.1352	11.2356
FA-GNN	0.7817	0.4844	0.4228	0.5558	0.6293	2.1533	0.5205	0.7222	0.4617	1.0457
FA-GNN (AHo)	0.7767	0.4116	0.3918	0.6002	0.7031	2.2311	0.4228	0.8372	0.4054	1.0333
FA-GNN (AHo & Cov.)	0.7798	0.5061	0.4541	0.5485	0.5949	1.9846	0.5195	0.7121	0.3914	1.2032
AbDiffuser (uniform prior)	0.8122	0.5528	0.5300	0.7053	0.6184	1.8129	0.5704	0.8326	0.3914	1.0416
AbDiffuser (no projection layer)	10.9194	10.7255	10.5253	13.6499	15.0346	10.9846	10.9083	15.9310	11.6059	11.7446
AbDiffuser (no Cov.)	0.5867	0.3425	0.3206	0.4272	0.4848	1.6261	0.3606	0.4921	0.3296	0.7801
AbDiffuser	0.5068	0.2896	0.2642	0.3282	0.3708	1.4921	0.3110	0.3871	0.2611	0.7334
AbDiffuser (side chains)	0.4463	0.2751	0.2426	0.2764	0.3266	1.2869	0.3025	0.3187	0.2390	0.7533
Table 6: Detailed RMSD for generated antibodies based on Paired OAS dataset [61] after optimization with Rosetta [2]. The top three results in each column are highlighted as First, Second, Third.
Model	Full ↓	Fr ↓	Fr. H ↓	CDR H1 ↓	CDR H2 ↓	CDR H3 ↓	Fr. L ↓	CDR L1 ↓	CDR L2 ↓	CDR L3 ↓
MEAN [43]	0.7412	0.3004	0.2718	0.5395	0.2909	2.7830	0.3261	0.3758	0.2593	0.6849
EGNN (AHo & Cov.)	9.2535	8.8170	8.5701	12.0330	12.7993	10.1256	8.9911	14.4588	9.7059	10.6565
FA-GNN (AHo & Cov.)	2.1631	2.2522	1.1541	0.6734	0.5783	2.0892	2.9101	1.4517	0.5797	2.1591
AbDiffuser	0.3692	0.2017	0.1415	0.3349	0.2474	1.1464	0.2479	0.1743	0.1589	0.6625
AbDiffuser (side chains)	0.4087	0.2755	0.2304	0.3632	0.3044	1.1065	0.3141	0.2957	0.1920	0.7217
AbDiffuser (τ = 0.75)	0.3584	0.1969	0.1358	0.3283	0.2459	1.1003	0.2434	0.1642	0.1513	0.6599
AbDiffuser (s.c., τ = 0.75)	0.3981	0.2747	0.2267	0.3615	0.2795	1.0497	0.3155	0.2939	0.2050	0.7151
AbDiffuser (τ = 0.01)	0.3210	0.1837	0.1202	0.3023	0.2464	0.9288	0.2306	0.1257	0.1285	0.6583
AbDiffuser (s.c. τ = 0.01)	0.6298	0.5661	0.5450	0.7135	0.6025	1.0870	0.5868	0.6878	0.6012	0.8720
Table 7: Detailed RMSD for generated antibodies based on Trastuzumab mutant dataset [55] after optimization with Rosetta [2]. The top three results in each column are highlighted as First, Second, Third.
L Wet-Lab Validation of Designs
In our wet-lab experiments, we follow the procedure of Hsiao et al. [29]. As we focus on Trastuzumab mutants, the experiments need to account for the fact that the part of the HER2 antigen to which Trastuzumab binds is prone to misfolding in vitro. In the usual experimental procedure for the SPR measurements of Kd values, the antibody is immobilized and the antigen (in this case HER2) is used as a ligand. However, for the domain to which Trastuzumab is binding in HER2, taking into account the problem of misfolding, the standard procedure described by Hsiao et al. [29] is to immobilize the antigen and treat the antibody as a ligand, making the 'misfolded' HER2 essentially invisible in the binding kinetics.
As pointed out by Mason et al. [55], mutating Trastuzumab such that the resulting antibody still binds to HER2 is quite challenging, as some binders in the dataset are just one edit away from non-binders. We also check the similarity of our new binders verified in the wet-lab experiments to both the binder and non-binder sets from the dataset. As can be seen in Table 8 some of the best binders we discovered are equidistant to both sets.
To further highlight the similarity of the non-binder and the binder sets from Mason et al. [55] we include the residue frequency (logo) plots for the CDR H3 positions that were varied in Figure 5.
M Implementation and Training Details
As mentioned in the main text, all of the models are evaluated using the same denoising procedure. They also share the input embedding procedure: we utilize a learned dictionary for each amino acid type and another learned dictionary for the chain type (light or heavy). These initial embeddings are summed together to form the input to the actual model. For all models except the APMixer, an
Generated binder	Binder dist.	Num. closest binders	Non-binder dist.	Num. closest non-binders	KD↓
SRYGSSGFYQFTY	2	2	2	2	5.35e-08
SRWLASGFYTFAY	1	1	2	2	4.72e-09
SRWSGDGFYQFDY	1	1	2	3	4.12e-09
SRWRGSGFYEFDY	1	1	2	3	2.89e-09
SRWRASGFYAYDY	1	2	3	19	2.00e-09
SRYGGFGFYQFDY	2	3	2	2	1.86e-09
SRYGGSGFYTFDY	2	8	2	2	3.17e-10
Table 8: Edit distances of our generated binders to closest binders and non-binders in the dataset by Mason et al. [55], together with the number of such closest (non-)binders.
Figure 5: Logo plots for relative amino acid frequencies in the mutated CDR H3 positions for binders (top) and non-binders (bottom) in the dataset by Mason et al. [55].
additional sinusoidal sequence position embedding is included in the sum. The model also receives the time step  $t$  as an additional input, concatenated with the other inputs. Similar to Hoogeboom et al. [28], we found this simple time-step conditioning to work well. The input and output of the model are always processed with our constraint layer from Section 3.4. The amino acid type predictions are made by computing a vector product with the input amino acid dictionary.
We will cover the model-specific architectural details in the following.
M.1 APMixer Details
We build ab mixer out of blocks as described in Section 3.3, where each block consists of two MLPs, that are applied consecutively on the columns
$$
X _ {. j} = X _ {. j} + W _ {2} \rho \left(W _ {1} \text {L a y e r N o r m} \left(X _ {. j}\right)\right) \text {f o r a l l} j \in [ c ]
$$
and on the rows
$$
X _ {i, \cdot} = X _ {i, \cdot} + W _ {4} \rho \left(W _ {3} \text {L a y e r N o r m} \left(X _ {i, \cdot}\right)\right) \text {f o r a l l} i \in [ r ].
$$
The whole APMixer used 8 such blocks plus an embedding MLP and two readout MLPs, one for predicting residue atom positions and another for the residue type predictions. Their outputs are respectively processed using equivariant and invariant frame averaging formulations.
As we noted earlier, each block is wrapped in frame averaging. We choose to process half of the output equivalently and half invariantly. It is possible to just use the equivariant formulation throughout the inner blocks, but we found this to slightly improve the performance of the model. Possibly because this aligns slightly better with having position-invariant sequence features. In this case, we extend the APMixer block to have a linear layer before the column MLP, to merge the two representations.