SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tianxiao  $\mathbf{L}\mathbf{i}^{1,2*}$ , Hongyu Guo $^{3}$ , Filippo Grazioli $^{4}$ , Mark Gerstein $^{2\dagger}$ , Martin Renqiang Min $^{1\dagger*}$
<sup>1</sup> NEC Laboratories America, <sup>2</sup> Yale University
$^{3}$  National Research Council Canada,  $^{4}$  NEC Laboratories Europe
tianxiao.li@yale.edu, hongyu.guo@uottawa.ca, flpgrz@outlook.com mark.gerstein@yale.edu, renqiang@nec-labs.com
In protein biophysics, the separation between the functionally important residues (forming the active site or binding surface) and those that create the overall structure (the fold) is a well-established and fundamental concept. Identifying and modifying those functional sites is critical for protein engineering but computationally nontrivial, and requires significant domain knowledge. To automate this process from a data-driven perspective, we propose a disentangled Wasserstein autoencoder with an auxiliary classifier, which isolates the function-related patterns from the rest with theoretical guarantees. This enables one-pass protein sequence editing and improves the understanding of the resulting sequences and editing actions involved. To demonstrate its effectiveness, we apply it to T-cell receptors (TCRs), a well-studied structure-function case. We show that our method can be used to alter the function of TCRs without changing the structural backbone, outperforming several competing methods in generation quality and efficiency, and requiring only  $10\%$  of the running time needed by baseline models. To our knowledge, this is the first approach that utilizes disentangled representations for TCR engineering.
Decades of work in protein biology have shown the separation of the overall structure and the smaller "functional" site, such as the generic structure versus the active site in enzymes [1], and the characteristic immunoglobulin fold versus the antigen-binding complementarity-determining region (CDR) in immunoproteins [2, 3]. The latter usually defines the protein's key function, but cannot work on its own without the stabilizing effect of the former. This dichotomy is similar to the content-style separation in computer vision [4] and natural language processing [5]. For efficient protein engineering, it is often desired that the overall structure is preserved while only the functionally relevant sites are modified. Traditional methods for this task require significant domain knowledge and are usually limited to specific scenarios. Several recent studies make use of deep generative models [6] or reinforcement learning [7] to learn from large-scale data the implicit generation and editing policies to alter proteins. Here, we tackle the problem by utilizing explicit functional features through the disentangled representation learning (DRL), where the protein sequence is separately embedded into a "functional" embedding and a "structural" embedding. This approach results in a more interpretable latent space and enables more efficient conditional generation and property manipulation for protein engineering.
DRL has been applied to the separation of "style" and "content" of images [8], or static and dynamic parts of videos [9, 10] for tasks such as style transfer [11, 8] and conditional generation [12, 13].
Attaining the aforementioned disentangled embeddings in discrete sequences such as protein sequences, however, is challenging because the functional residues can vary greatly across different proteins. To this end, several recent works on discrete sequences such as natural languages use adversarial objectives to achieve disentangled embeddings [14, 15]. Cheng, et al. [16] improves the disentanglement with a mutual information (MI) upper bound on the embedding space of a variational autoencoder (VAE). However, this approach relies on a complicated implementation of multiple losses that are approximated through various neural networks, and requires finding a dedicated trade-off among them, making the model difficult to train. To address these challenges, we propose a Wasserstein autoencoder (WAE) [17] framework that achieves disentangled embeddings with a theoretical guarantee, using a simpler loss function. Also, WAE could be trained deterministically, avoiding several practical challenges of VAE in general, especially on sequences [18, 19]. Our approach is proven to simultaneously maximize the mutual information (MI) between the data and the latent embedding space while minimizing the MI between the different parts of the embeddings, through minimizing the Wasserstein loss.
To demonstrate the effectiveness and utility of our method, we apply it to the engineering of T cell receptors (TCRs), which uses a similar structure fold as the immunoglobulin, one of the best-studied protein structures and a good example of separation of structure and functions. TCRs play an important role in the adaptive immune response [20] by specifically binding to peptide antigens [21] (Fig. 1A). Designing TCRs with higher affinity to the target peptide is thus of high interest in immunotherapy [22]. Various data-driven methods have been proposed to enhance the accuracy of TCR binding prediction [23-25, 12, 13, 26, 27]. However, there has been limited research on leveraging machine learning for TCR engineering. A related application is the motif scaffolding problem [28-32] where a structural "scaffold" is generated supporting a fixed functional "motif". Here our goal is the opposite: to modify the functional parts of the sequence instead by directly introducing mutations to it.
We focus on the CDR3 $\beta$  region of TCRs where there is sufficient data and a clearly defined functional role (peptide binding). Using a large TCR-peptide binding dataset, we empirically demonstrate that our method successfully separates key patterns related to binding ("functional" embedding) from generic structural backbones ("structural" embedding). Due to the lack of 3D structural data, we assume that similarity between the "structure-related" parts of the sequence indicates similarity in the structure. Furthermore, by modifying only the functional embedding, our approach is able to generate new TCRs with desired binding properties while preserving the structural backbone, requiring only  $10\%$  of the running time needed by baseline models in comparison. We also note from our TCR engineering results that mutations can be introduced throughout the sequence, which implies that the model learns higher-level functional and structural patterns that could span the whole sequence, instead of looking for a universal clear cut between a "functional segment" and a "structural segment".
We summarize our main contributions as follows:
- To our knowledge, we are the first to formulate computational protein design as a style transfer problem and leverage disentangled embeddings for TCR engineering, resulting in more interpretable and efficient conditional generation and property manipulation.
- We propose a disentangled Wasserstein autoencoder with an auxiliary classifier, which effectively isolates the function-related patterns from the rest with theoretical guarantees.
- We show that by modifying only the functional embedding, we can edit TCR sequences into desired properties while maintaining their backbones, running 10 times faster than baselines.
We define our problem of TCR engineering task as follows: given a TCR sequence and a peptide it could not bind to, introduce a minimal number of mutations to the TCR so that it gains the ability to bind to the peptide. In the meantime, the modified TCR should remain a valid TCR, with no major changes in the structural backbone. Based on the assumption that only certain amino acids within the TCR should be responsible for peptide interactions, we can define two kinds of patterns in the TCR sequence: functional patterns and structural patterns. The former comprises the amino acids
Figure 1: (A) Top: The TCR recognizes antigenic peptides provided by the major histocompatibility complex (MHC) with high specificity; bottom: the 3D structure of the TCR-peptide-MHC binding interface (PDB: 5HHO); the CDRs are highlighted. (B) The disentangled autoencoder framework, where the input  $x$ , i.e., the  $\mathrm{CDR}3\beta$ , is embedded into a functional embedding  $z_{f}$  (orange bar) and structural embedding  $z_{s}$  (green bar). (C) Method for sequence engineering with input  $x$ .  $\mathbf{z}_s$  of the template sequence and a modified  $\mathbf{z}_f'$ , which represents the desired peptide binding property, are fed to the decoder to generate engineered TCRs  $x'$ .
that define the peptide binding property. TCRs that bind to the same peptide should have similar functional patterns. The latter refers to all other patterns that do not relate to the function but could affect the validity. Following [23-25], we here limit our modeling to the CDR3 $\beta$  region since it is the most active region for TCR binding. In the rest of the paper, we use "TCR" and "CDR3 $\beta$  interchangeably.
2.2 Disentangled Wasserstein Autoencoder
Our proposed framework, named TCR-dWAE, leverages a disentangled Wasserstein autoencoder that learns embeddings corresponding to the functional and structural patterns. In this setting, the input data sample for the model is a triplet  $\{\mathbf{x},\mathbf{u},y\}$ , where  $\mathbf{x}$  is the TCR sequence,  $\mathbf{u}$  is the peptide sequence, and  $y$  is the binary label indicating the interaction.
In detail, given an input triplet  $\{\mathbf{x},\mathbf{u},y\}$ , the embedding space of  $\mathbf{x}$  is separated into two parts:  $\mathbf{z} = \operatorname{concat}(\mathbf{z}_f,\mathbf{z}_s)$ , where  $\mathbf{z}_f$  is the functional embedding, and  $\mathbf{z}_s$  is the structural embedding. The schematic of the model is shown in Fig. 1B.
2.2.1 Encoders and Auxiliary Classifier
We use two separate encoders for the embeddings, respectively:
$$
\mathbf {z} _ {e} = \Theta_ {e} (\mathbf {x}),
$$
where  $e\in \{s,f\}$  correspond to "structure" and "function".
First, the functional embedding  $\mathbf{z}_f$  is encoded by the functional encoder  $\Theta_f(\mathbf{x})$ . In order to make sure  $\mathbf{z}_f$  carries information about binding to the given peptide  $\mathbf{u}$ , we introduce an auxiliary classifier  $\Psi(\mathbf{z}_f, \mathbf{u})$  that takes  $\mathbf{z}_f$  and the peptide  $\mathbf{u}$  as input and predicts the probability of positive binding label  $q_{\Psi}(y \mid \mathbf{z}_f, \mathbf{u})$ ,
$$
\hat {y} = q _ {\Psi} (Y = 1 \mid \mathbf {z} _ {f}, \mathbf {u}) = \Psi (\mathbf {z} _ {f}, \mathbf {u}).
$$
We define the binding prediction loss as binary cross entropy:
$$
\mathcal {L} _ {f \_ c l s} (\hat {y}, y) = - y \log \hat {y} - (1 - y) \log (1 - \hat {y}).
$$
Second, the structural embedding  $\mathbf{z}_s$  is encoded by the structural encoder  $\Theta_s(\mathbf{x})$ . To enforce  $\mathbf{z}_s$  to contain all the information other than the peptide binding-related patterns, we leverage a sequence reconstruction loss, as will be discussed in detail in Section 2.2.3.
2.2.2 Disentanglement of the Embeddings
To attain disentanglement between  $z_{f}$  and  $z_{s}$ , we introduce a Wasserstein autoencoder regularization term in our loss function following [17], by minimizing the maximum mean discrepancy (MMD) between the distribution of the embeddings  $\mathbf{Z} \sim Q_{\mathbf{Z}}$  where  $\mathbf{z} = \mathrm{concat}(\mathbf{z}_f, \mathbf{z}_s)$  and an isotropic multivariate Gaussian prior  $\mathbf{Z}_0 \sim P_{\mathbf{Z}}$  where  $P_{\mathbf{Z}} = \mathcal{N}(0, I_d)$ :
$$
\mathcal {L} _ {\text {W a s s}} (\mathbf {Z}) = \operatorname {M M D} \left(P _ {\mathbf {Z}}, Q _ {\mathbf {Z}}\right). \tag {1}
$$
The MMD is estimated as follows: given the embeddings  $\{\mathbf{z}_1, \mathbf{z}_2, \dots, \mathbf{z}_n\}$  of an input batch of size  $n$ , we randomly sample from the Gaussian prior  $\{\tilde{\mathbf{z}}_1, \tilde{\mathbf{z}}_2, \dots, \tilde{\mathbf{z}}_n\}$  with the same sample size. We then use the linear time unbiased estimator proposed by [33] to estimate the MMD:
$$
\operatorname {M M D} \left(P _ {\mathbf {Z}}, Q _ {\mathbf {Z}}\right) = \frac {1}{\lfloor n / 2 \rfloor} \sum_ {i} ^ {\lfloor n / 2 \rfloor} h \left(\left(\mathbf {z} _ {2 i - 1}, \tilde {\mathbf {z}} _ {2 i - 1}\right), \left(\mathbf {z} _ {2 i}, \tilde {\mathbf {z}} _ {2 i}\right)\right),
$$
where  $h((\mathbf{z}_i,\tilde{\mathbf{z}}_i),(\mathbf{z}_j,\tilde{\mathbf{z}}_j)) = k(\mathbf{z}_i,\mathbf{z}_j) + k(\tilde{\mathbf{z}}_i,\tilde{\mathbf{z}}_j) - k(\mathbf{z}_i,\tilde{\mathbf{z}}_j) - k(\mathbf{z}_j,\tilde{\mathbf{z}}_i)$  and  $k$  is the kernel function. Here we use a radial basis function (RBF) [34] with  $\sigma = 1$  as the kernel.
By minimizing this loss, the joint distribution of the embeddings matches  $\mathcal{N}(0,I_d)$ , so that  $\mathbf{z}_f$  and  $\mathbf{z}_s$  are independent. Also, the Gaussian shape of the latent space facilitates generation [35, 17].
2.2.3 Decoder and Overall Training Objective
The decoder  $\Gamma$  takes  $\mathbf{z}_f, \mathbf{z}_s$  and peptide  $\mathbf{u}$  as input and reconstructs the original sequence as  $\mathbf{x}'$ . It also acts as a regularizer to enforce the structural embedding  $\mathbf{z}_s$  to contain all the information other than the peptide binding-related patterns. The reconstruction loss is the position-wise binary cross entropy between  $\mathbf{x}$  and  $\mathbf{x}'$  averaged across all positions of the sequence:
$$
\mathbf {x} ^ {\prime} = \Gamma (\operatorname {c o n c a t} (\mathbf {z} _ {s}, \mathbf {z} _ {f}, \mathbf {u}))
$$
$$
\mathcal {L} _ {r e c o n} (\mathbf {x}, \mathbf {x} ^ {\prime}) = \frac {1}{l} \sum_ {i} ^ {l} - \mathbf {x} ^ {(i)} \log (\mathbf {x} ^ {\prime (i)}) - (1 - \mathbf {x} ^ {(i)}) \log (1 - \mathbf {x} ^ {\prime (i)}),
$$
where  $l$  is the length of the sequence and  $\mathbf{x}^{(i)}$  is the probability distribution over the amino acids at the  $i$ -th position.
Combining all these losses with weights  $\beta_{1}$  and  $\beta_{2}$ , we have the final objective function, which then can be optimized through gradient descent in an end-to-end fashion:
$$
\mathcal {L} = \mathcal {L} _ {\text {r e c o n}} + \beta_ {1} \mathcal {L} _ {f _ {-} c l s} + \beta_ {2} \mathcal {L} _ {\text {W a s s}}.
$$
2.3 Disentanglement Guarantee
To show how our method can guarantee the disentangled embeddings, we provide a novel perspective on the latent space of Wasserstein autoencoders [17] utilizing the variation of information following [16]. We introduce a measurement of disentanglement as follows:
$$
D \left(\mathbf {Z} _ {f}, \mathbf {Z} _ {s}; \mathbf {X} \mid \mathbf {U}\right) = V I \left(\mathbf {Z} _ {s}; \mathbf {X} \mid \mathbf {U}\right) + V I \left(\mathbf {Z} _ {f}; \mathbf {X} \mid \mathbf {U}\right) - V I \left(\mathbf {Z} _ {f}; \mathbf {Z} _ {s} \mid \mathbf {U}\right),
$$
where  $VI$  is the variation of information,  $VI(\mathbf{X};Y) = H(\mathbf{X}) + H(Y) - 2I(\mathbf{X};Y)$ , which is a measure of independence between two random variables. For simplicity, we omit the condition  $\mathbf{U}$  (peptide) in the following parts.
This measurement reaches 0 when  $\mathbf{Z}_f$  and  $\mathbf{Z}_s$  are totally independent, i.e. disentangled. It could further be simplified as:
$$
\begin{array}{l} V I (\mathbf {Z} _ {s}; \mathbf {X}) + V I (\mathbf {Z} _ {f}; \mathbf {X}) - V I (\mathbf {Z} _ {f}; \mathbf {Z} _ {s}) \\ = 2 H (\mathbf {X}) + 2 \left[ I \left(\mathbf {Z} _ {f}; \mathbf {Z} _ {s}\right) - I (\mathbf {X}; \mathbf {Z} _ {s}) - I (\mathbf {X}; \mathbf {Z} _ {f}) \right]. \\ \end{array}
$$
Note that  $H(\mathbf{X})$  is a constant. Also, according to data processing inequality, as  $\mathbf{z}_f \to \mathbf{x} \to y$  forms a Markov chain, we have  $I(x; \mathbf{z}_f) \geq I(y; \mathbf{z}_f)$ . Combining the results above, we have the upper bound of the disentanglement objective:
$$
I \left(\mathbf {Z} _ {f}; \mathbf {Z} _ {s}\right) - I \left(\mathbf {X}; \mathbf {Z} _ {s}\right) - I \left(\mathbf {X}; \mathbf {Z} _ {f}\right) \leq I \left(\mathbf {Z} _ {f}; \mathbf {Z} _ {s}\right) - I \left(\mathbf {X}; \mathbf {Z} _ {s}\right) - I (Y; \mathbf {Z} _ {f}). \tag {2}
$$
Next, we show how our framework could minimize each part of the upper bound in (2).
Maximizing  $I(\mathbf{X};\mathbf{Z}_s)$  Similar to [10], we have the following theorem:
Theorem 2.1 Given the encoder  $Q_{\theta}(\mathbf{Z} \mid \mathbf{X})$ , decoder  $P_{\gamma}(\mathbf{X} \mid \mathbf{Z})$ , prior  $P(\mathbf{Z})$ , and the data distribution  $P_{D}$
$$
\mathbb {D} _ {K L} (Q (\mathbf {Z}) \parallel P (\mathbf {Z})) = \mathbb {E} _ {p _ {D}} \left[ \mathbb {D} _ {K L} (Q _ {\theta} (\mathbf {Z} \mid \mathbf {X}) \parallel P (\mathbf {Z})) \right] - I (\mathbf {X}; \mathbf {Z}),
$$
where  $Q(\mathbf{Z})$  is the marginal distribution of the encoder when  $\mathbf{X} \sim P_D$  and  $\mathbf{Z} \sim Q_{\theta}(\mathbf{Z} \mid \mathbf{X})$ .
Theorem 2.1 shows that by minimizing the KL divergence between the marginal  $Q(\mathbf{Z})$  and the prior  $P(\mathbf{Z})$ , we jointly maximize the mutual information between the data  $\mathbf{X}$  and the embedding  $\mathbf{Z}$ , and minimize the KL divergence between  $Q_{\theta}(\mathbf{Z} \mid \mathbf{X})$  and the prior  $P(\mathbf{Z})$ . Detailed proof of the theorem can be found in the Appendix B.1. This also applies to the two separate parts of  $\mathbf{Z}, \mathbf{Z}_f$  and  $\mathbf{Z}_s$ . In practice, because the marginal cannot be measured directly, we minimize the aforementioned kernel MMD (1) instead.
As a result, there is no need for additional constraints on the information content of  $\mathbf{Z}_s$  because  $I(\mathbf{X};\mathbf{Z}_f)$  is automatically maximized by the objective. We also empirically verify in Section 3.2.3 that supervision on  $\mathbf{Z}_s$  does not improve the model performance.
Maximizing  $I(Y; \mathbf{Z}_f)$  and  $I(Y; \mathbf{Z}_f)$  has an lower bound as follows:
$$
I (Y; \mathbf {Z} _ {f}) \geq H (Y) + \mathbb {E} _ {p (Y, \mathbf {Z} _ {f})} \log q _ {\Psi} (Y \mid \mathbf {Z} _ {f}),
$$
where  $q_{\Psi}(Y \mid \mathbf{Z}_f)$  is the predicted probability by the auxiliary classifier  $\Psi$  (note we omitted the condition  $\mathbf{U}$  here). Thus, maximizing the performance of classifier  $\Psi$  would maximize  $I(Y; \mathbf{Z}_f)$ .
Minimizing  $I(\mathbf{Z}_f; \mathbf{Z}_s)$  Minimization of the Wasserstein loss (1) forces the distribution of the embedding space  $\mathbf{Z}$  to approach an isotropic multivariate Gaussian prior  $P_{\mathbf{Z}} = \mathcal{N}(0, I_d)$ , where all the dimensions are independent. Thus, the dimensions of  $\mathbf{Z}$  will be independent, which also minimizes the mutual information between the two parts of the embedding,  $\mathbf{Z}_f$  and  $\mathbf{Z}_s$ .
As a conclusion, our objective can jointly minimize  $I(\mathbf{Z}_f; \mathbf{Z}_s)$  and maximize  $I(\mathbf{X}; \mathbf{Z}_s)$  and  $I(Y; \mathbf{Z}_f)$ . The former ensures independence between the embeddings, and the latter enforces them to learn separate information, achieving disentanglement.
Datasets Interacting TCR-peptide pairs are obtained from VDJDB [36], merged with experimentally-validated negative pairs from NetTCR [26]. We then expended the negatives to  $5\mathrm{x}$  the size of positives by adding randomly shuffled TCR-peptide pairs and only selected the peptides that could be well-classified by ERGO[23], a state-of-the-art method for TCR-binding prediction (Appendix A.2). The selected samples are then balanced resulting in 26262 positive and 26262 negative pairs with 2918 unique TCRs and 10 peptides. We also performed the same experiments on the McPAS-TCR dataset [37] (Appendix A.1). For each peptide, we select from VDJDB an additional set of 5000 TCR sequences that do not overlap with the training set. This set is used for the TCR engineering experiment.
Implementation Details The TCR-dWAE model uses two transformer encoders [38] for  $\Theta_s, \Theta_f$  and a long short-term memory (LSTM) recurrent neural network decoder [39] for  $\Gamma$ . The auxiliary classifier  $\Psi$  is a 2-layer perceptron. Hyperparameters are selected through grid search. All results are averaged across five random seeds. See Appendix B.2 for more details.
r̅v	r̅b	%valid	#mut/len	%positive valid ↑
TCR-dWAE	1.36±0.03	0.45±0.09	0.61±0.06	0.49±0.05	0.23±0.02
TCR-dVAE	1.44±0.02	0.24±0.02	0.64±0.05	0.4±0.02	0.16±0.01
greedy	0.34±0.0	0.79±0.0	0.02±0.0	0.34±0.0	0.02±0.0
genetic	0.39±0.03	1.0±0.0	0.02±0.0	NA	0.02±0.0
naive rm	0.35±0.0	0.2±0.0	0.03±0.0	0.35±0.01	0.0±0.0
MCTS	-0.1±0.0	0.95±0.0	0.0±0.0	NA	0.0±0.0
TCR-dWAE (null)	1.51±0.01	0.05±0.03	0.85±0.01	0.45±0.07	0.04±0.03
original	1.59	0.01	0.92	NA	0.01
Table 1: Performance comparison. original is the evaluation metrics on the template TCRs without any modification. Only unique sequences are counted. Results are averaged across selected peptides (i.e., SSYRRPVGI, TTPESANL, FRDYVDRFYKTLRAEQASQE, CTPYDINQM) and then averaged across five random seeds.
3.2 TCR Engineering
3.2.1 Manipulating TCR Binding via Functional Embeddings
As shown in Fig. 1C, given any TCR sequence template  $\mathbf{x}$ , we combine its original  $\mathbf{z}_s$  and a new functional embedding  $\mathbf{z}_f'$  that is positive for the target peptide  $\mathbf{u}$  which is then fed to the decoder to generate a new TCR  $\mathbf{x}'$  that could potentially bind to  $\mathbf{u}$  while maintaining the backbone of  $\mathbf{x}$ . For each generation, we obtain the  $\mathbf{z}_f'$  from a random positive sample in the dataset.
3.2.2 Metrics and Baselines
We use the following metrics to evaluate whether the engineered sequence  $\mathbf{x}'$  (1) is a valid TCR sequence and (2) binds to the given peptide, which we denote as validity score and binding score, respectively:
The validity score  $r_v$  evaluates whether the generated TCR follows similar generic patterns as naturally observed TCRs from TCRdb, an independent and much larger dataset [40]. We train another LSTM-based autoencoder on TCRdb. If the generated sequence can be reconstructed successfully by the autoencoder and has a high likelihood in the distribution of the latent space, we consider it as a valid TCR from the same distribution as the known ones (Appendix D.2). We use the sum of the reconstruction accuracy and the likelihood as the validity score and show that this metric separates true TCRs from other protein segments and random sequences (Appendix D.3).
For the binding score, the engineered sequence  $\mathbf{x}'$  and the peptide  $\mathbf{u}$  are fed into the pre-trained ERGO classifier and binding probability  $r_b = \mathrm{ERGO}(\mathbf{x}',\mathbf{u})$  is calculated.
We compare TCR-dwAE with the following baselines (see Appendix C for more details):
Random mutation-based For each iteration, a new random mutation is added to the template. The best mutated sequence (one with the highest ERGO score) is selected on either a sample-level (greedy), population-level (genetic), or without any selection (naive rm), then goes into the next round.
Generation-based This includes Monte Carlo tree search (MCTS), where random residues are iteratively added until the sequence reaches a maximum length. For each generation process, the leaf with the highest ERGO score is added to the results.
TCR-dVAE We also include a VAE baseline inspired by the objectives of IDEL from [16], which minimizes a MI upper bound to achieve disentanglement in the embedding space (Appendix C.3).
3.2.3 Main Results
We use peptides with AUROC  $>0.8$  by the classifier  $\Psi$  for the engineering task (SSYRRPVGI, TTPESANL, FRDYVDRFYKTLRAEQASQE, CTPYDINQM). The results are listed in Table 1. The cutoff for "valid" is  $r_v \geq 1.25$ , as it can successfully separate known TCRs from other protein sequences. The cutoff for "positive" TCR is  $r_b > 0.5$  as the prediction scores by ERGO is rather extreme, either 0 or 1. The vast majority of the generated valid sequences (ranging from  $70\% \sim$