SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compare our model with two types of methods for the generation of the optimized TCR  $\mathbf{x}^{\prime}$ : (1) mutation-based, which iteratively adds random mutations to the template sequence; and (2) generation-based, which generates novel sequences of the pre-determined length range. For both types of methods, the modified/generated sequences are selected by peptide binding scores from the respective pre-trained ERGO. The experiments are performed on each peptide in the dataset independently.
C.1 Mutation-based baselines
Random mutation (naive rm) The TCR is randomly mutated by one amino acid for 8 times progressively. This process is repeated for 10 runs for each TCR and the resulting one with the highest ERGO prediction score is reported.
Greedy mutation (greedy) For each TCR, 10 randomly mutated sequences are generated, each with one amino acid difference from the original sequence. Among the 10 mutated sequences, we select the one that gives the highest binding prediction with the given peptide as the template for the next run. This process is repeated 8 times.
Genetic algorithm (genetic) Let  $M$  be the sample size. For each TCR, 10 randomly mutated sequences are generated, each with one amino acid difference from the original sequence. All mutated sequences along with the original TCRs are then pooled together, and the top  $M$  sequences that give the highest binding prediction are used as the input for the next run. This process is repeated 8 times.
C.2 Generation-based baselines
Monte Carlo tree search (MCTS) TCRs are generated by adding amino acids iteratively, resulting in a search tree. When the TCR length reaches 10, the binding score is estimated by ERGO. For each iteration, a random node is selected for expansion and then evaluated by ERGO, and the scores of all its parent nodes are updated accordingly. The tree expansion ends when the length reaches 20. For every generation process, the highest leaf node is added to the output TCR set.
C.3 TCR-dVAE
TCR-dVAE uses a similar objective as IDEL [16], which is a VAE with a mutual information constraint on the latent space. For training, the loss comprises of the following components:
- The reconstruction loss:  $\mathcal{L}_{\text{recon}}(\mathbf{x}, \mathbf{x}^{\prime})$
The KL divergence term for VAE:  $\mathcal{L}_{KL} = \mathbb{D}_{\mathrm{KL}}(q_{\theta}(\mathbf{z}_s,\mathbf{z}_f|\mathbf{x})\parallel p(\mathbf{z}_s,\mathbf{z}_f))$
- The reconstruction loss given  $\mathbf{z}_s\colon \mathcal{L}_s(\mathbf{x},\Phi (\mathbf{z}_s))$  where  $\Phi$  is an auxiliary decoder.
- The classification loss given  $z_f \colon \mathcal{L}_{f\_cls}(\hat{y}, y)$
- The sample-based MI upper bound between the embeddings:  $\mathcal{L}_{MI}(\mathbf{z}_f, \mathbf{z}_s)$ . This requires an approximation of the conditional distribution  $p(\mathbf{z}_f | \mathbf{z}_s)$ , which is achieved by a separate neural network.
Here we use our own notations, not the ones used in the original paper, for better comparison.
For the hyperparameters, we use 10.0 weight for  $\mathcal{L}_{\text{recon}}$ ,  $\mathcal{L}_s$  and  $\mathcal{L}_{f,\text{cls}}$ , 1.0 for the other terms, and a learning rate of  $1e - 4$ . In practice, we perform annealing [18] on  $\mathcal{L}_{KL}$  and  $\mathcal{L}_{MI}$  where their weights gradually increase from 0 to 1 during the first 100 epochs and remain for the rest of the epochs, to make sure the embeddings are as informative as possible. We observe that other hyperparameter settings would easily lead to severe posterior collapse where the embedding is ignored for the reconstruction.
D Evaluation of the optimized sequences
D.1 Training of the autoencoder
We train an LSTM-based autoencoder, which we denote as TCR-AE0, on the 277 million TCR sequences from TCRdb [40]. TCR-AE0 has a latent space of dimension 16 and is trained for 50,000 steps with a batch size of 256.
D.2 Validity score
The validity score combines two scores calculated from TCR-AE0:
- The reconstruction-based score is calculated as
$$
r _ {r} \left(\mathbf {x} ^ {\prime}\right) = 1 - \operatorname {l e v} \left(\mathbf {x} ^ {\prime}, \operatorname {T C R - A E 0} \left(\mathbf {x} ^ {\prime}\right)\right) / l \left(\mathbf {x} ^ {\prime}\right),
$$
where  $\mathrm{lev}(\mathbf{x}^{\prime},\mathrm{TCR - AE0}(\mathbf{x}^{\prime}))$  is the Levenstein distance between the original sequence and the reconstructed sequence, and  $l(\mathbf{x}^{\prime})$  is the length of the reconstructed sequence. Higher  $r_r$  means  $\mathbf{x}^{\prime}$  is better reconstructed from TCR-AE0 and is thus more likely to be a valid TCR sequence.
- The density-based score calculates whether the embedding of  $\mathbf{x}'$  follows the same distribution as known TCRs. We learn a Gaussian mixture model from the latent embeddings of known TCRs from TCR-AE0. The likelihood of the embedding  $\mathbf{e}'$  of  $\mathbf{x}'$  from TCR-AE0 falling in the same Gaussian mixture distribution is denoted as  $P(\mathbf{e}')$ . The density-based score is calculated as
$$
r _ {d} \left(\mathbf {x} ^ {\prime}\right) = \exp \left(1 + \frac {\log P \left(\mathbf {e} ^ {\prime}\right)}{\tau}\right),
$$
where  $\tau = 10$ . Higher  $r_d$  means the latent embedding of  $\mathbf{x}'$  from TCR-AE0 is more likely to follow the same distribution as other valid TCR sequences.
We then define the validity score as  $r_v = r_r + r_d$ .
D.3 Validation of the metrics
We compare the TCR-AE-derived evaluation metric scores of three different sources:
(1) all unique CDR3 $\beta$  sequences from VDJB.
(2) random segments of length  $8 - 18$  (which is the most frequent lengths of  $\mathrm{CDR3}\beta$  sequences) from random uniprot [63] protein sequences of the same size as (1). The conservative 'C' at the beginning and 'F' at the end are added to the segments.
(3) random shuffling of the sequences from (1), where the first 'C' and the last 'F' are kept
We show in Fig. 4 that for both two scores  $r_d$  and  $r_r$ , as well as their sum, CDR3 $\beta$  sequences score much higher than random proteins or shuffled sequences. This shows these scores could be effectively used for the estimation of TCR sequence validity. We choose  $r_v > 1.25$  as the criteria for valid sequences as it rejects most negatives.
E Extended Results
E.1 Comparison of TCR Engineering Performance
We find consistently improved performance of our method over the baselines in both VDJDB (Table 1) and McPAS-TCR (Table 9). Also, the majority of generated sequences are unique (Table 10) and all are novel (not observed in the original dataset; statistics not shown).
The relative performance between methods in comparison still holds when all peptides, including those that are poorly classified, are taken into account (Table 7). However, when the results are separated by individual peptides (Table 8), we observe that for the poorly classified peptides (AUC<0.8; see Methods), using a positive  $\mathbf{z}_f$  (TCR-dWAE row) has little difference from using a random  $\mathbf{z}_f$  (TCR-dWAE (nu11) row). This further highlights the importance of well-defined and well-classified functional labels. Otherwise,  $\mathbf{z}_f$  does not encode the proper functional information, so it becomes uninformative and does not alter the sequence function better than a random embedding.
E.2 Analysis of the Model
We show extended  $\mathbf{z}_f$  and  $\mathbf{z}_s$  T-SNE patterns in Fig. 5, colored by the ground truth label as well as the predicted label. For the well-classified peptides, there is a clear separation of positives and negatives in the  $\mathbf{z}_f$  space but not  $\mathbf{z}_s$ . There are cases where the true positives are not separable from the true negatives using  $\mathbf{z}_f$ , but the predicted positives and the predicted negatives (by the function classifier  $\Psi$ ) are still separated. We consider the latter as a problem with data quality and classification accuracy, not embedding. Meanwhile, the classifier shows consistent performance over the peptides across random seeds with (Fig. 6, left) and without (Fig. 6, right) the Wasserstein loss.
As a result of the Wasserstein loss, the distribution of the embedding space is closer to a multivariate Gaussian (Fig. 7A. It becomes less regularized without the Wasserstein loss (Fig. 7B). Contrary to  $\mathbf{z}_f$  (Fig. 3B in the main text), the T-SNE of  $\mathbf{z}_s$  and first-layer embedding of the encoder for the positive samples cannot distinguish the binding targets from each other (Fig. 8A).
E.3 Analysis of the Generated Sequences
In addition to the results presented in the main text, we also selected 500 random positive and negative sequences from the training set and replaced their  $\mathbf{z}_f$  with the most positive/negative one in the subset. The generated sequences using their original  $\mathbf{z}_s$  and the new  $\mathbf{z}_f$  have binding scores mostly related to the  $\mathbf{z}_f$ , regardless of whether the  $\mathbf{z}_s$  source is positive or negative. This shows  $\mathbf{z}_f$  can be used to encode and transfer binding information, which lays the foundation for the following TCR engineering experiments (Fig. 8B).
The generated sequences have a similar length distribution as their templates (Fig. 8C), meaning no drastic changes are made. We further find that the  $\mathbf{z}_s$  of the modified TCRs show high cosine similarity with those of their templates, while the  $\mathbf{z}_f$  are more similar to the  $\mathbf{z}_f$  used for their generation (Fig. 8D), but not with that of the template. These show that the modified TCRs preserve the "structural" information from  $\mathbf{z}_s$  and incorporate the new "functional" information from the modified  $\mathbf{z}_f$ .
As in Fig. 8E, there is no significant correlation between the sequence validity and the binding scores. Also, we find that most binding scores predicted by ERGO are rather extreme (either 1 or 0).
E.4 Interpolation
For each peptide, we perform interpolation of  $\mathbf{z}_f$  between 100 random pairs for 10 steps with  $\mathbf{z}_f^{(r)} = r\mathbf{z}_f^{(1)} + (1 - r)\mathbf{z}_f^{(2)}, r \in [0,1]$ , while  $\mathbf{z}_s$  remains the same. Fig. 8F shows that interpolating between positive  $\mathbf{z}_f$  pairs preserves the positive binding score compared to negative pairs. Here we show one example of interpolation, where the first column is the generated sequence and the second column is the predicted binding score:
Seq 1:CASTESDRRSQNTQYF
Seq 2: CASSLSTFTANTAQLFF
Peptide: CTPYDINQM
Interpolated  $\mathbf{z}_f$ , with  $\mathbf{z}_s$  of Seq 1
Sequence  $r_b$
CASTESDRR SQNTQYF 1.0 (original)
CASTESDRRSQNTQYF 1.0
CASTESDRNSNQPQYF 1.0
CASTESTKNSNQPQYF 1.0
CASSESTMNSNQPQYF 1.0
CASSESTTNSNQPQYF 1.0
CASSETTNSNQPQYF 1.0
CASSESTTANTAQLFF 1.0
Interpolated  $\mathbf{z}_f$ , with  $\mathbf{z}_s$  of Seq 2
CASSLSTFTANTAQLFF 1.0 (original)
CASSLSTFTANTAQLFF 1.0
CASSLSTRTANTAQLFF 1.0
CASSLSTRTANNTQLFF 0.16
CASSLSTRTAKNTQLFF 0.97
CASSLSTRTSQNTQYF 1.0
CASSLSTRDSTNTQYF 1.0
These results further prove that  $\mathbf{z}_s$  could be used to transfer binding information independent of the  $\mathbf{z}_s$ , and that the latent space is well-regularized. Also, throughout the interpolation, the hypervariable region only undergoes minor changes. This shows  $\mathbf{z}_s$  could preserve the structural backbone information within both the conservative and hypervariable regions.
E.5 Extended comparison between sampling methods and model architecture
We experiment with several different generation modes and model architecture. The results are shown in Table 11.
E.5.1 Source of positive embeddings
In addition to using a  $\mathbf{z}_f$  from a random positive sample (labeled as random pos), we also try two other means of obtaining positive  $\mathbf{z}_f$ 's: (1) best: we use the  $\mathbf{z}_f$  of the sample with the highest classification score from the auxiliary classifier  $\Psi$  (2) avg: we use the average  $\mathbf{z}_f$  for all positive samples. In both cases, the same  $\mathbf{z}_f$  is used for all generations. We perform the same experiments with TCR-dVAE, by using the mean of the  $\mathbf{z}_f$  of the positive sample. We observe that all three methods have comparable performance for TCR-dWAE, indicating a well-regularized latent space.
E.5.2 Stochastic generation
As TCR-dVAE is a probabilistic model, we also compared with a random sampling-based method. We use both the predicted mean and variance of the positive source to generate 5 random  $\mathbf{z}_f$  's and take the average results of all the generated samples (labeled TCR-dVAE-rand). In this scenario, only the random pos and best modes are applicable. We observe a notable decrease in the performance as the quality of generation varies with the random sampling, probably due to a less regularized latent space.
E.5.3 Transformer decoder
We further test a TCR-dwAE model with a transformer decoder (TCR-dwAE-attn) instead of a LSTM decoder. The performance is comparable with those with LSTM (Table 11) while the autoregressive generation is notably slower (Table 10)
E.5.4 Modeling the hypervariable region only
We train a model only using the hypervariable region and run the sequence modification pipeline as in full CDR3beta (TCR-dWAE (trimmed)). After the hypervariable region is modified, we add back the conservative region of the input template for each generated sequence.
As shown in Table 11, the performance is worse than modeling the full sequence. To our surprise, it seems that the lower performance is caused by a massive drop in the number of positive sequences, while the sequence validity is not affected much by the trimming, as can be seen from the number of valid sequences. This may indicate some interactions between the two regions that are essential for positive binding. Thus, the modeling and generation should be performed on the entire CDR3 $\beta$  region, instead of on manually defined segments.
E.6 Comparison between evaluation metrics
E.6.1 Binding prediction
Besides ERGO [23], NetTCR[26] is another sequence-based TCR-peptide binding model with competitive performance. However, NetTCR is only trained on three peptides and shows the best performance when both the  $\alpha$  and  $\beta$  chain are provided, which is mostly not available in our dataset. Thus, ERGO provides a more efficient evaluation method for single-chain data and a greater variety of peptides. For the comparison between model architectures, we re-train a single-chain NetTCR on our own training set with more peptides. The evaluation result on the generated sequences in Table 1 in the manuscript (reporting the %positive valid only) with the two classifiers is as follows. We show in Table 12 that the conclusions are similar between the two models.
E.6.2 Uniqueness and novelty score
We calculate the novelty score following [64] for each individual sequence from the positive valid subset in Table 1:
$$
\operatorname {N o v} (x) = \min  _ {s _ {i} \in D _ {0}} d (x, s _ {i})
$$
where  $D_0$  is the set of templates and the distance measure  $d$  is the Levenshtein distance relative to the length of the longer sequence. This score can then be used to select the "novel" sequences. We also include a held-out validation set as a reference. We find that around  $1/3 \sim 1/2$  of the sequences remain at cutoff 0.2 and all are excluded at cutoff 0.4 (Table 14). The same pattern is observed in the validation set, which indicates a similar distribution of "novelty" between the generated sequences and the raw data.
We also report the same novelty and diversity metrics for the positive valid subset as in [64]. We would like to point out that:
(1) These metrics evaluate the set of generated sequences as a whole, while we directly select the sequences that meet the validity and binding score criterion, so they are not our primary objective.
(2) The comparisons are only meaningful within different generative models. For the random mutation-based methods, one can always achieve high novelty and diversity by introducing a sufficient number of mutations.
We show that the results are not very different between TCR-dWAE and TCR-dVAE (Table 13). In addition, we include the case of a VAE that suffers from posterior collapse (where the generation is always limited to the same few patterns). The scores for the first two are at a similar level as the validation set, while the diversity score for the collapsed VAE drops a lot. These suggest that the positive valid sequences presented in the paper are sufficiently diverse and novel.
source	#pos	auroc	aupr
AVFDRKSDAK	vdjdb	1641	0.94	0.71
CTPYDINQM	vdjdb	500	0.99	0.81
ELAGIGILTV	vdjdb	1410	0.95	0.79
FRDYVDRFYKTLRAEQASQE	vdjdb	367	0.98	0.85
GILGFVFTL	vdjdb	3408	0.95	0.89
GLCTLVAML	vdjdb	962	0.92	0.73
IVTDFSVIK	vdjdb	548	0.94	0.62
KRWIILGLNK	vdjdb	319	0.95	0.54
NLVPMVATV	vdjdb	4421	0.94	0.85
RAFKQLL	vdjdb	830	0.94	0.75
SSLENFRAYV	vdjdb	322	0.99	0.57
SSYRRPVGI	vdjdb	337	0.99	0.81
STPESANL	vdjdb	234	0.99	0.35
TTPESANL	vdjdb	511	0.99	0.75
ASNENMETM	mcpas	265	0.98	0.63
CRVLCCYVL	mcpas	435	0.95	0.7
EAAGIGILTV	mcpas	272	0.97	0.55
FRCPRRFCF	mcpas	266	0.96	0.58
GILGFVFTL	mcpas	1142	0.96	0.9
GLCTLVAML	mcpas	828	0.95	0.85
LPRRSGAAGA	mcpas	2142	0.96	0.88
NLVPMVATV	mcpas	543	0.93	0.78
RFYKTLRAEQASQ	mcpas	304	0.99	0.91
SSLENFRAYV	mcpas	416	0.99	0.78
SSYRRPVGI	mcpas	337	0.99	0.83
TPRVTGGAM	mcpas	274	0.95	0.52
VTEHDTLLY	mcpas	273	0.95	0.45
WEDLFCDESLSSPEPPSSSE	mcpas	364	0.98	0.93
Table 5: Statistics and ERGO prediction performance for the selected peptides from the first round.
VDJDB
#pos	#all
NLVPMVATV	2880	5478
GLCTLVAML	2880	5478
RAKFKQLL	2880	5478
AVFDRKSDAK	2880	5478
SSYRRPVGI	2268	4934
GILGFVFTL	2880	5478
TTPESANL	2286	4950
FRDYVDRFYKTLRAEQASQE	2034	4726
ELAGIGILTV	2880	5478
CTPYDINQM	2394	5046
MCPAS
#pos	#all
NLVPMVATV	1792	3810
RFYKTLRAEQASQ	1528	3579
WEDLFCDESLSSPEPPSSSE	1928	3929
GILGFVFTL	2560	4482
SSYRRPVGI	1504	3558
SSLENFRAYV	1824	3838
CRVLCCYVL	1680	3712
LPRRSGAAGA	2560	4482
GLCTLVAML	2560	4482
Table 6: Statistics of the training data by peptide.
VDJDB
r̅v	r̅b	%valid	#mut/len	%positive valid
TCR-dWAE	1.47±0.02	0.31±0.04	0.76±0.03	0.47±0.06	0.18±0.01
TCR-dVAE	1.51±0.01	0.2±0.01	0.73±0.02	0.38±0.02	0.15±0.01
greedy	0.31±0.0	0.88±0.0	0.02±0.0	0.32±0.0	0.02±0.0
genetic	0.36±0.02	1.0±0.0	0.02±0.0	1.0±0.0	0.02±0.0
naive rm	0.33±0.0	0.39±0.0	0.02±0.0	0.35±0.0	0.01±0.0
MCTS	-0.1±0.0	0.95±0.0	0.0±0.0	0.0±0.0	0.0±0.0
TCR-dWAE (null)	1.51±0.01	0.08±0.02	0.86±0.01	0.45±0.06	0.07±0.02
original	1.59	0.03	0.92	0.0	0.03
Table 7: Performance comparison for VDJDB, averaged across all peptides (ELAGIGILTV, GLCTLVAML, AVFDRKSDAK, SSYRRPVGI, RAKFKQLL, CTPYDINQM, TTPESANL, NLVPMVATV, FRDYVDRFYKTLRAEQASQE, GILGFVFTL)
r̅v	r̅b	%valid	#mut/len	%positive valid
SSYRRPVGI
TCR-dWAE	1.11±0.14	0.48±0.18	0.37±0.16	0.48±0.06	0.06±0.01
TCR-dVAE	1.35±0.06	0.25±0.07	0.68±0.05	0.38±0.02	0.06±0.02
TCR-dWAE (null)	1.5±0.01	0.03±0.05	0.85±0.01	0.43±0.05	0.02±0.04
original	1.59	0.02	0.92	0.0	0.01
TTPESANL
TCR-dWAE	1.41±0.06	0.64±0.11	0.56±0.06	0.52±0.04	0.34±0.06
TCR-dVAE	1.42±0.02	0.4±0.07	0.74±0.02	0.46±0.01	0.19±0.01
TCR-dWAE (null)	1.5±0.01	0.05±0.04	0.85±0.01	0.46±0.08	0.04±0.03
original	1.59	0.01	0.92	0.0	0.01
FRDYVDRFYKTLRAEQASQE
TCR-dWAE	1.59±0.03	0.28±0.06	0.86±0.03	0.47±0.07	0.22±0.04
TCR-dVAE	1.58±0.02	0.35±0.06	0.91±0.01	0.39±0.02	0.22±0.01
TCR-dWAE (null)	1.51±0.01	0.05±0.04	0.86±0.01	0.45±0.07	0.04±0.03
original	1.59	0.02	0.92	0.0	0.01
CTPYDINQM
TCR-dWAE	1.36±0.03	0.51±0.07	0.66±0.04	0.49±0.05	0.29±0.04
TCR-dVAE	1.41±0.02	0.36±0.06	0.76±0.03	0.43±0.02	0.18±0.02
TCR-dWAE (null)	1.5±0.01	0.08±0.02	0.85±0.01	0.45±0.07	0.06±0.02
original	1.59	0.02	0.92	0.0	0.01
ELAGIGILTV*
TCR-dWAE	1.56±0.03	0.1±0.01	0.9±0.02	0.44±0.06	0.09±0.01
TCR-dVAE	1.58±0.02	0.13±0.02	0.92±0.02	0.32±0.02	0.1±0.01
TCR-dWAE (null)	1.51±0.02	0.08±0.0	0.86±0.02	0.45±0.05	0.07±0.0
original	1.59	0.05	0.92	0.0	0.04
GLCTLVAML*
TCR-dWAE	1.56±0.02	0.11±0.01	0.89±0.02	0.47±0.06	0.1±0.01
TCR-dVAE	1.58±0.02	0.14±0.03	0.91±0.02	0.35±0.03	0.11±0.02
TCR-dWAE (null)	1.51±0.01	0.09±0.0	0.85±0.01	0.46±0.06	0.07±0.01
original	1.59	0.04	0.92	0.0	0.04
AVFDRKSDAK*
TCR-dWAE	1.58±0.03	0.15±0.01	0.9±0.02	0.45±0.06	0.13±0.01
TCR-dVAE	1.59±0.02	0.19±0.02	0.91±0.01	0.33±0.02	0.14±0.01
TCR-dWAE (null)	1.51±0.01	0.15±0.01	0.85±0.01	0.46±0.06	0.12±0.01
original	1.59	0.15	0.92	0.0	0.14
RAFKQLL*
TCR-dWAE	1.58±0.03	0.08±0.0	0.91±0.02	0.43±0.07	0.08±0.0
TCR-dVAE	1.58±0.02	0.13±0.01	0.92±0.01	0.32±0.02	0.1±0.01
TCR-dWAE (null)	1.51±0.01	0.07±0.01	0.86±0.01	0.44±0.05	0.06±0.01
original	1.59	0.04	0.92	0.0	0.04
NLVPMVATV*
TCR-dWAE	1.57±0.02	0.2±0.0	0.9±0.02	0.45±0.07	0.17±0.01
TCR-dVAE	1.59±0.02	0.21±0.0	0.92±0.02	0.33±0.02	0.17±0.0
TCR-dWAE (null)	1.51±0.01	0.18±0.01	0.86±0.01	0.46±0.06	0.15±0.01
original	1.59	0.08	0.92	0.0	0.07
GILGFVFTL*
TCR-dWAE	1.56±0.02	0.19±0.05	0.89±0.02	0.46±0.06	0.17±0.04
TCR-dVAE	1.57±0.02	0.23±0.04	0.91±0.02	0.34±0.02	0.18±0.02
TCR-dWAE (null)	1.51±0.01	0.1±0.0	0.86±0.01	0.44±0.06	0.09±0.0
original	1.59	0.05	0.92	0.0	0.05
Table 8: Performance comparison for VDJDB, separated by peptides. Peptides with * have classification AUC < 0.8.
MCPAS
r̅v	r̅b	%valid	#mut/len	%positive valid
TCR-dWAE	1.39±0.07	0.29±0.03	0.69±0.06	0.47±0.03	0.17±0.01
TCR-dVAE	1.45±0.01	0.31±0.04	0.47±0.04	0.4±0.02	0.1±0.01
greedy	0.33±0.0	0.92±0.0	0.02±0.0	0.33±0.0	0.02±0.0
genetic	0.34±0.03	1.0±0.0	0.02±0.0	NA	0.02±0.0
naive rm	0.31±0.0	0.43±0.0	0.02±0.0	0.35±0.01	0.01±0.0
MCTS	-0.11±0.0	0.94±0.0	0.0±0.0	NA	0.0±0.0
TCR-dWAE (null)	1.45±0.06	0.08±0.0	0.79±0.05	0.41±0.03	0.06±0.01
Table 9: Performance comparison for MCPAS, averaged across selected peptides (SSYRRPVGI, WEDLFCDESLSSPEPPSSSE, SSLENFRAYV, RFYKTLRAEQASQ, GLCTLVAML, CRVLC-CYVL)
VDJDB	MCPAS
valid:all	unique:valid	running time	valid:all	unique:valid
TCR-dWAE	0.79±0.02	0.95±0.01	56	0.72±0.07	0.95±0.01
TCR-dWAE-attn	0.77±0.06	0.97±0.01	5553
TCR-dWAE (null)	0.86±0.01	1.0±0.0	56	0.8±0.05	0.99±0.0
TCR-dVAE	0.84±0.01	0.65±0.02	58	0.8±0.01	0.81±0.06
greedy	0.02±0.0	1.0±0.0	832	0.02±0.0	1.0±0.0
genetic	0.02±0.0	0.73±0.04	1389	0.03±0.0	0.71±0.08
naive rm	0.02±0.0	1.0±0.0	106	0.02±0.0	1.0±0.0
MCTS	0.0±0.0	0.0±0.0	4559	0.0±0.0	0.04±0.08
Table 10: Additional performance comparison. This table shows the ratio of valid sequences and unique valid sequences, as well as the running time per 5000 samples for CTPYDINQM.
r̅v	r̅b	%valid	#mut/len	%positive valid ↑
TCR-dWAE (best)	1.33±0.07	0.53±0.18	0.45±0.14	0.5±0.05	0.18±0.05
TCR-dWAE (avg)	1.37±0.13	0.51±0.21	0.53±0.18	0.46±0.09	0.2±0.06
TCR-dWAE (random pos)	1.36±0.03	0.48±0.09	0.61±0.06	0.49±0.05	0.23±0.02
TCR-dVAE-rand (best)	0.31±0.06	0.01±0.01	0.0±0.0	0.21±0.3	0.0±0.0
TCR-dVAE-rand (random pos)	1.44±0.02	0.35±0.05	0.45±0.04	0.4±0.01	0.1±0.01
TCR-dWAE-attn (random pos)	1.46±0.06	0.32±0.08	0.75±0.06	0.54±0.05	0.19±0.04
TCR-dWAE (trimmed)	1.35±0.08	0.19±0.07	0.68±0.08	0.78±0.0	0.11±0.02
TCR-dWAE (null)	1.51±0.01	0.05±0.03	0.85±0.01	0.45±0.07	0.04±0.03
original	1.59	0.01	0.92	0.0	0.01
Table 11: Comparison between model designs and modes of obtaining positive  $\mathbf{z}_f$ 's.
ERGO	NetTCR
TCR-dWAE	0.24±0.01	0.2±0.01
TCR-dVAE	0.16±0.01	0.15±0.01
greedy	0.02±0.0	0.01±0.0
genetic	0.02±0.0	0.0±0.0
naive rm	0.0±0.0	0.0±0.0
MCTS	0.0±0.0	0.0±0.0
Table 12: Performance using different evaluation metrics (ERGO and NetTCR)
diversity	novelty
TCR-dWAE	0.20±0.0	0.20±0.0
TCR-dVAE	0.23±0.0	0.19±0.0
VAE (collapse)	0.14±0.01	0.17±0.01
Validation	0.26	0.18
Table 13: Diversity and novelty scores for the generated sequences.
0.05	0.1	0.2	0.4
TCR-dWAE	1174.6±38.7	1130.4±34.0	460.1±28.0	0.1±0.1
TCR-dVAE	809.6±49.5	774.6±49.8	290.9±25.6	0.0±0.0
Validation	621	551	191	0
Table 14: Number of "novel" sequences with different novelty score cutoffs.
Figure 4: Distribution of TCR-AE-based evaluation metrics on known CDR3 $\beta$ 's, randomly selected protein segments and randomly shuffled CDR3 $\beta$ 's.
Figure 5: T-SNE of  $\mathbf{z}_f$  and  $\mathbf{z}_s$  embeddings for all peptides in VDJDB (left) and MCPAS (right). Points are colored by the label. "True" means the ground truth label. "Pred" refers to label predcited by the function classifier  $\Psi$ .
Figure 6: ROC of function classifier  $\Psi$  by peptide, with different hyperparameter settings and random seeds.