Scalable Universal T-Cell Receptor Embeddings from Adaptive Immune Repertoires
Authors: Paidamoyo Chapfuwa
Ilker Demirel
Lorenzo Pisani
Javier Zazo
Elon Portugaly
H. Jabran Zahid
Julia Greissl
Venue: ICLR 2025 Poster (2025)
Keywords: Immunomics
T-cell Receptor Embeddings
GloVe
Random Projection Theory
Scaling
Unsupervised Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We employ GloVe and random projection theory to infer scalable universal T-cell receptor embeddings from adaptive immune repertoires.

Abstract: T cells are a key component of the adaptive immune system, targeting infections, cancers, and allergens with specificity encoded by their T cell receptors (TCRs), and retaining a memory of their targets. High-throughput TCR repertoire sequencing captures a cross-section of TCRs that encode the immune history of any subject, though the data are heterogeneous, high dimensional, sparse, and mostly unlabeled. 
Sets of TCRs responding to the same antigen, *i.e.*, a protein fragment, co-occur in subjects sharing immune genetics and exposure history. Here, we leverage TCR co-occurrence across a large set of TCR repertoires and employ the GloVe (Pennington et al., 2014)  algorithm to derive low-dimensional, dense vector representations (embeddings) of TCRs. We then aggregate these TCR embeddings to generate subject-level embeddings based on observed *subject-specific* TCR subsets. Further, we leverage random projection theory to improve GloVe's computational efficiency in terms of memory usage and training time. Extensive experimental results show that TCR embeddings targeting the same pathogen have high cosine similarity, and subject-level embeddings encode both immune genetics and pathogenic exposure history.

Published as a conference paper at ICLR 2025
SCALABLE UNIVERSAL T-CELL RECEPTOR EMBED- DINGS FROM ADAPTIVE IMMUNE REPERTOIRES
Paidamoyo Chapfuwa', Ilker Demirel Lorenzo Pisani"
Javier Zazo', Elon Portugaly', H. Jabran Zahid', Julia Greissl
Microsoff Research, Redmond, USA; 'MIT, USA; Microsoft Research, Cambridge, UK (pchapfuwa, lopisani, javierzazo, e elonp, hzahid, jugrei ss j cmicrooott.coom demire rellrelee @mite edu
ABSTRACT
T cells are a key component of the adaptive immune system, targeting infections, cancers, and allergens with specificity encoded by their T cell receptors (TCRs), and retaining a memory of their targets. High-throughput TCR repertoire sequenc- ing captures cross-section of TCRs that encode the immune history of any sub- ject, though the data are heterogeneous, high dimensional, sparse, and mostly unlabeled. Sets of TCRs responding to the same antigen, i.e., a protein fragment. co-occur in subjects sharing immune genetics and exposure history. Here, we leverage TCR co-occurrence across a large set of TCR repertoires and employ the GloVe (Pennington et al.) 2014) algorithm to derive low-dimensional, dense vector representations (embeddings) of TCRs. We then aggregate these TCR em- beddings to generate subject-leve levell embeddings based on observed
TCR subsets. Further, we leverage random projection theory to improve GloVe's s computational efficiency in terms of memory usage and training time. Extensive experimental results show that TCR embeddings targeting the same pathogen have high cosine similarity, and subject-level embeddings encode both immune genet- ics and pathogenic exposure history.
INTRODUCTION
Low-dimensional representations that capture meaningful qualities of the input data they compress are one of the foundations of modern machine learning. In text and imaging modalities, these em- beddings have reached high level of sophistication and have significantly advanced our ability to build tools such as automatic image captioning or visual questions answering systems (Dosovitskiy al. 202T, Vaswani al. 2017). Recently, significant progress has been made in deriving embed- dings in life science domains as diverse as proteins (Lin et al.) (003)) and the RNA expression levels of cells (Rosen et al., 2023). However, modalities in the life sciences bring their own set of chal- lenges due to a lack of data at scale, data heterogeneity, and data structures that violate assumptions made in text and imaging domains. These challenges require novel approaches to build high-quality representations. Here, focus learning representations of T-cell receptors (TCRs) and TCR repertoires.
The primary function of T cells is identify fragments of foreign proteins, known antigens, derived from viruses, bacteria, and cancerous cells, and to kill the cells in which these proteins are found. These antigens presented by human leukocyte antigens (HLAs) (Zinkernagel & Doherty 1974). Each subject has many HLAs, and HLA diversity is high in the human population (Hughes) & Yeager 1998). HLA frequencies are power-law distributed, not all of their HLAs T cells are specific to both antigens and the presenting HLA, with specificity encoded by the TCR (Davis & Bjorkman 1988). To effectively clear infections, T cells clonally expand (>1000-foll) once they encounter a cognate antigen (Burnet 1976), which significantly increases their likelihood of being sampled and sequenced. As a consequence, subjects sharing HLAs and pathogenic exposure have significantly higher likelihood of sharing subset of TCRs
subjects typically share some but
To whom correspondence should be addressed
Work done principally during an internship Microsoft Research

---

Published as a conference paper at ICLR 2025
(A) HLAS + Rasponse
(D) Measure
Antiigens
Tcell Counts
(C) Randomly Sample
colls and Sequance
[E] Calculate Co-Desureence
2
HERD
Figure 1: A schematic of the data generation process. Each subject has set of HLA types (A) which together with exposure history define their TCR repertoire (B). A subset of these TCRs (0(10)) sequenced (C). Using large number of repertoires, generate TCR co-occurrence matrix that the input to our model (D, E).
than would be expected randomly. Thus, sets of TCRs specific to particular HI.A-pathogenic exposure combination co-oceur in subsets of subjects who share the HLA and pathogenic exposure. It is possible to identify millions HLA-associated TCRs, and previous work has demonstrated that co-clustering yields subsets of TCRs that map to specific pathogens (Zahid et al. 2024 May et al.
2024). Previous work applying deep learning TCRs has focused supervised learning of TCR
protein features for mattgen-biiding prediction and repertoire classification (Zhang et al.) 2021 Zhao
al. 2023 Widrich et al. 2020 Sidhom al. 2021).
Here, we infer TCR embeddings leveraging co-occurrence patterns, which encodes exposure to hundreds of common pathogens and HLA--ssociations (DeWitt III al. 2018). We then aggregate all TCRs observed in a subject's repertoire to derive a subject-level embedded representation that encodes both the subject's HLAs and their immune history. See Figure [ for an overview of the data generation process. TCR repertoires are uniquely positioned to capture which TCRs share an immunological context because they probe deeply into an individual's immune system (Robins) 2013). Moreover, TCR embeddings represent TCRs within an immunological context, such as being part group of TCRS responding to the SARS-CoVa spike protein or group of Cytomegalovirus (CMV) TCRs. When aggregated into repertoire embeddings, this represents low dimensional snapshot of an individual's 's immune history.
The key contributions of this paper are follows:
We show that TCR embeddings can be learned from co-occurrences using the GloVe (Global Vectors for Word Representation) algorithm (Pennington et al. 2014).
We provide a proof-of-concepp demonstrating that a simple aggregation of TCR embeddings into repertoire embeddings captures both the immunc genetics and pathogenic exposure history of individuals.
We demonstrate TCR embeddings get richer when more TCRs and repertoires used.
We leverage random projection theory to improve GloVe's computational complexity in terms of memory usage and training time.
We explore the topology of the embedding space, finding that vectors of TCRs associated with the same antigen and repertoires with similar genetic or exposure profiles have higher cosine similarities.
LEARNING TCR EMBEDDINGS
Co-occurrence of TCRs across repertoires has been employed to discover TCRs targeting the same exposures (Emerson et al. 2017: WeWit III et al. 2018) and HLAs (Zahid et al., 2024). A similar phenomenon has been observed natural language processing, where words with similar meanings co-occur in similar contexts. The GloVe (Pennington et al., 2014) algorithm exploits this property to learn semantically meaningful word embeddings from co-occurrences. Here, use TCR3
2

---

Published as a conference paper at ICLR 2025
TCR
Embedding
TCR
Embedding
Repertoire
Repertoire
X
Co- occuucereene
TCR
C
GloVe
TCR
W
p(x W)
R
Figure 2: Overview of the modeling approach. We compute a TCR-TCR co-occurrence count matrix
Ce xxx from a repertoire- TCR binary matrix X E {0, 1] N>N namely C = X X. Leveraging the GloVe (Pennington et al. 2014) algorithm and the Johnson-Lindenstrauss (JL) transform (Lin- denstrauss & Johnson 1984), we infer low-dimensional TCR embeddings W IR Kxd from TCR
co-occurrence C. We generate repertoire embeddings R via simple pooling function p(æ, W) in Equation 8
quences defined by their gene, gene, and CDR3. The proposed method is general and could also be applied to TCRa-only sequencing paired TCRo+TCR5 sequencing.
We apply a modified version of the GloVe algorithm to learn immunologically meaningful TCR embeddings. Rather than using random weight initialization scheme, we leverage random projec- tion theory to initialize VloVe, leading to improved memory usage and training time while enabling training with only fraction of the data. We calculate TCR co-occurrence statistics from TCR reper- toire measurements of subjects across K TCRs. Let T = {tz, t2 tx| } be the set of K TCRs ær,, (k) means TCR l is present in
consider, and x., E {0, K the n-th TCR repertoire an individual and absent otherwise. We use a binary indicator for TCR presence/absence instead of clonal frequencies for more robust learning. We use TCR repertoire measurements of N individu- als and denote by X E {0, N the sparse and binary repertoire- TCR matrix where K >> N. illustrates our end-to-end modeling approach for (i) learning low-dimensional TCR embed- dings from co-occurrence and (ii) generating repertoire embeddings via simple pooling function p(-) Equation
2.1 USING THE GLOVE ALGORITHM TO LEARN FROM TCR CO-OCCURRENCES
The GloVe algorithm co-occurrence statisties across documents combining global matrix fac- torization methods (e.g., latent semantic analysis (Deerwester et al.) 1990)) and local context window methods (e.g. Word2Vec (Mikolov et al.) 2013a)). We adopt the algorithm for TCRs. Let C denote the TCR-TCR co-occurrence matrix where C(i,.)) is the number of repertoires that contain both TCRs and j. While constructing C for words requires a rather involved scan over the documents, in our case it is a simple matrix product C = x We exploit this difference to develop faster algorithm with lower training time and space complexity. Because X is sparse binary matrix, we leverage distributed techniques for matrix multiplication using only the non-zero elements in X in Spark (Zaharia al., 2010). Refer to Appendix AA.4 for details on the Spark algorithm used to compute C.
We learn TCR embeddings by minimizing the following GloVe loss function L(W, b. W. by C) formulated as:
C.W6bW..B W, S(C w6s) + bb. + by log C1
where (b,6) 6} R are bias terms of the TCR embeddings ( W) E kxd that allow for more flexible modeling of the marginal frequency of occurrence. Previous studies have shown that bias terms are correlated to the marginal frequency occurrence (Shazeer al., 2016). which is consistent with our results. In practice, for a given TCR the final embedding w is the average of the learned embeddings up and w. The weighting function f : 24 - R2+ prevents commonly occurring TCRs from dominating the loss. We deviate from GloVe's weighting function by specifying a monotonic transformation the co-occurrence counts f(.) = 5g (C., > 0) + sCO according to Shazeer et al. (0166), where hyperparameters are set to {sp = 0.1, = 0.25, a = 0.5} and (a) is an indicator function s.t. ((a) = 1 if holds or 1(a) 0 otherwise. This approach yields good performance hyper-parameter TCR-TCR co-occurrence matrix, which is computationally efficient given that C is very sparse
without hyper parameter tuning. Similar to GloVe, we only train on the non-zero elements of the
3

---

Published as a conference paper at ICLR 2025
(approximately 80% of the entries in C' are zero in our case). Note that GloVe optimizes with respect to the log-coo-ccurrencees to preserve the ratios of co-occurrence probabilities, which can also be considered as simple monotonically increasing transform for regularizing the effect of large co-occurrences. Model parameters (W, b, W. b) of the cost function in Equation (1) optimized using stochastic gradient descent on minibatches from C.
2.2 SCALABLE LEARNING VIA JOHNNON--INDENSTTANN
TRANSFORM
Glo Ve does not scale well. To minimize Equation D we first compute the co-occurrence matrix C, which requires calculating Ka dot-product of N -eelement vectors, resulting in O(NK2) time complexity which scales quadratically with the number of TCRs. This calculation becomes very expensive when learning embeddings for millions of TCRs. Additionally, we need to compute and sum O(K2) terms (depending on the sparsity of C) to backpropagate the loss in Equation (1) in single training epoch, which adds to the computational complexity of GloVe. To address these issues, we develop a learning scheme where we initialize GloVe embeddings with carefully con- structed initial TCR embeddings and improve GloVe's computational complexity in training time and memory usage.
Fast random projections can reduce dimensionality while preserving structure. reduce the dimensionality of C' using random projections (Achlioptas, 2003) Roughgarden & Valiant (024).. Precisely, let P be K x d random matrix where d E 2,4 is the desired dimensionality of the TCR embeddings. We compute
1 Vd (X
WJ
where JLJ,, is K x d and we denote by w the i-th row of W for i E {1, .) Here JL transform (see Lindenstrauss & Johnson (1844). XP requires
stands for d x N dot-products of K <<-element vectors, and X '(XP) requires d K dot-products of N-element vectors, resulting in OdNN)) K') time-complexity. Furthermore, when P sampled carefully ac- cording to Theorem 2.1 the transform given by Equation (2) approximately preserves the pairwise L2-distances between the rows C.
Theorem 2.1 (Theorem I in Achlioptas 2003
Let the elements of P be i...d. drawn as follows:
-1,
with probability 1/6
P(a, = v3 x
0, with probability 2/3
(3)
1,
with probability 1/6.
Given e, 3 8> 0, if
4+28 2/2--373
log K, we have, for all i,j E {1, K}? KK),
(1 (1-c) ||e - -c,1l S Ww - w s (1 + ||a - c,1l²
(4)
with probability at least 1 K-8
This means that if the L2-distances between c, E R reliably correlate with the co-oceurrence patterns of the TCRs, one can lower dimensional will IRd TCR embeddings instead, where d O log K) for any distortion 0 < <1 Dasgupta & Gupta 2003 Lindenstrauss & Johnson 1984).
2.3 JL-GLOVE: IMPROVING GLOVE COMPLEXITY WITH JL INITIALIZATION
While the JL transformation has lower complexity than GloVe, i.e., O(dNK) O(NK2). it has 2 key limitations: (i) JL. embeddings are of lower quality than GloVe's ss as shown in Figure and thus they do not perform as well on downstream tasks, discussed in Appendix A.11 (ii) L2 -iddtttancss between co-oceurrence columns c; are preserved when d = O(E log K), which significantly in- creases the TCR embedding dimensions GloVe embeddings in Equation I1 with JL embeddings JI in Equation (6) resulting in faster vergence and allowing us to achieve good performance using only a fraction of the co-occurrence matrix C. shown Figure 3 Note that if TCRs i; and ty have similar co-occurrence patterns,
initialize scale K millions of TCRs. Therefore,
4

---

Published as a conference paper at ICLR 2025
Le., small ||e, - cyll, expect their respective JL embeddings ||w."" w.I'|| L2 di-distasces, to be small. In contrast, the GloVe objective in Equation (I). optimizes for cosine similarity, L.e., TCRs t, and t, will have high cosine similarity (w,, w5) |will ||, ros(,,, wy). To bridge this gap, leverage the geometric relationship
(5) Given Equation (5) it is evident that unit unitnnormalizing JI. embeddings W enables alignment with the GloVe objective function in Equation ( thus resulting in faster convergence, i.e. reduced training time. Moreover, normalized JL transform also accounts for varying marginal occurrences of TCRs. Refer to the Appendix A.2 for a more in-depth discussion on this aspect. Henceforth, we refer to the proposed Ge algorithm initialized with JL as JL-GLOVE.
Iw: wiil = ||,ill + || 2(w,, wg)
Normalizing JL. Transform We aim to normalize WJL without violating the JL. transform prop- erties in Theorem 2.1 or computing C. Naively, one could compute C, unit-normalize it to get C, s.t. é -. see illustration Equation (12). After this transformation, the comparison of pairwise distances ||; é₁ |l will not be affected by marginal occurrences. Importantly, l|², -ēyl should be smaller for co-occurring TCRs and thus preserved after the IL--transform
WJJ
Vd ÖP.
Hence, rows of JL in Equation 66 vilL. E TR.d could be used to initialize GloVe TCR embeddings.
However, to compute C, one needs C = X first, which is quadratic- time in the number of TCRs
K. We instead avoid constructing C' and approximate W. in inear-time:
Proposition 2.2. JL in Equation 66 can be approximated in OdN time when P is constructed
as in Theorem 2.1 by computing W in Equation 2 first and normalizing its rows. See proof in
Appendix 7
Random init. (1%)
The JL transform has been leveraged in various problems in the literature for its ability to reduce computational
JL-Norm Init. (1%)
Random init (100%)
JL-Norm Init. (100%)
complexity (Bingham & Mannila 2001 Akselrod- Ballin et al. [011 Argerich et al.] [016]] Canzar et al.) 2024). However, its normalized version, which can be approximately computed in linear-time as we argue in Proposition 2.2] is, to the best of our knowledge, a novel contribution. The proposed JL-GLOVE algorithm is general and can be re-purposed to other problems with nigh-dimensional and sparse counts data, e-g., B-cell re- ceptor repertoires (Chen et al. 2022). Additionally, con- structing P according Theorem 2.1 further reduces the computational complexity due to the sparsity of P.
10*
Global
10-1
Epoch
Figure 3: Comparison of GloVe loss in Equation (I) over C, when TCR embed- dings are initialized to random vs. JL- Norm JI, and when 100% vs. 1% of C is used (K 65, 751; d 100).
Refer to Appendix A.5 for details on sparse matrix algo- rithm used to compute W.J..
Comparisons of GloVe initialization schemes We initialize GloVe embeddings in Equation | with unit-normalized JL embeddings WJJ in Equation (6). Figure demonstrates that initializing GloVe embeddings with J (JL-Norm) and training on 1% of the co-occurrence data C' results in a loss that approaches the minimum loss achieved when using the entire dataset (100% of C". In- terestingly, the loss starts to increase if one keeps training, since the embeddings start over-fitting to the 1 1% of the co-occurrences. We address this using a separate validation subset of co-occurrences for early stopping and use another 1 % of C' to verify that the corresponding validation loss is almost a perfect proxy for the global GloVe loss which calculated for the entire C matrix. Note that random initialization of GloVe using subset of the co-occurrence data (1%) can never attain the minimum loss of JL-Norm initialization (1%), thus demonstrating that JL contains co-occurrence information beyond what is contained in the 1% of C used in training. Moreover, JL-Norm initial- ization converges faster than random initialization even if the entire matrix C' is feasible to compute and is used for training. This faster convergence becomes critical as the number of TCRs increases. We provide additional comparisons in Appendix A.11
5

---

Published as a conference paper at ICLR 2025
(200)
Poave (889)
Pervo 1586)
Gond 1233)
AM022 0236
ERSSPED (1964)
20
40
Chicase Expouure
MLA
(a) t-SNE Gl
(b) t-SNE Norm
Figure 4: t-SNE comparisons of (a) postprocessed GloVe in Equation 7 and (b) JL-Norm in Equa-
tion 66) TCR embeddings, with an embedding dimension of d = 100, derived from K = 65, 751 TCR measurements across N 31, 938 repertoires. We present the t-SNE plot of subset of 2,800 TCRs, colored by their disease exposure and HLA association.
2.4 ADDRESSING ANISOTROPY TCR EMBEDDINGS
The GloVe objective in Equation optimizes for alignment, i.e., TCR pairs with high co- occurrence are encouraged to have high cosine similarity. However, previous work has shown that GloVe embeddings tend to be anisotropic, occupying a narrow cone in the vector space (Arora al., 2017 Mu & Viswanath 2018). This anisotropy is driven by varying marginal occurrences that fol- low a power law distribution our case, see Figure A similar phenomena has been observed for contextual embeddings (Gao et al. 2019 Ethayarajh, 2019, Li et al. 2020). Inspired by the simple yet effective postprocessing approach by Mu & Viswanath (2018) we (i) center the TCR
embeddings around the origin (zero mean) for each dimension d; and (ti) remove the effect of the
dominant principal components, which are correlated with marginal frequencies. Hence, we specify
the final TCR embeddings W" R kxd following Mu & Viswanath (2018))
i = Wk f
w/ w Σ{ (uey Wk
where the mean vector μ K Ex wn and {ui,... ue} are the dominant principal components from PCA(W) s.t. == [d/100].
3 GENERATING TCR REPERTOIRE EMBEDDINGS
In the previous section, we learned vector embeddings for TCRs. Here, we are interested in doing the same for TCR repertoires, which contain a subset of TCRs in T. We use the postprocessed TCR embeddings W in Equation 7 derive low-dimensional representations of repertoires, which are useful for various downstream tasks, such as HLA and disease predictions. Given an n-th indi- vidual's 's repertoire denoted by binary vector E {0, 1) R. E Rd to the output of the pooling function
we assign the repertoire embeddings
x. TW'
R2 == p(z,, W')
(8)
Σ. 2x
where Σn 2g is the total number of TCRs observed in a repertoire and Equation ( applies a mean pooling set transformation on the dimensions d of the repertoire-specife TCR embeddings, i.e., the non-zero entries of æ. Mean pooling is simple linear transformation across repertoire-specific TCR embedding dimensions, which enables interpretability of the repertoire embeddings. Since co- occurring TCRs cluster in specific directions in Rd, the mean vector shall be skewed towards those directions, and can capture all the clusters when d is large enough (see Section 5 for related em- pirical findings and discussions). While mean pooling has achieved relative success in aggregating sentence embeddings (Shen et al. 2018), alternative advanced set pooling mechanisms, such set transformers (Lee et al. 2019) and optimal transport sunsport-based based kernel (Mialon et al.) 2021), could also be considered.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
We now assess the performance of TCR and repertoire embeddings on 5 disease prediction and 145 HLA inference binary classification tasks. We benchmark the proposed JL-GLOVE algo- rithm against competitive baselines. Additionally, we demonstrate the scalability of the embed- dings on these tasks with respect both the increasing number of TCRs and the growing size of repertoires. PyTorch code to train JL-GLOVE can be found at hhttps://githin olooooe We summarize the training procedure in Appendix A.6
com/
4.1 REPERTOIRE DATASETS
We train JL-GLOVE embeddings using two different training cohorts: i) TDETECT cohort of N = 31, 938 repertoires (May et al. 2024) and úi) PUBLIC cohort of = 3, 996 repertoires that publicly available. Both training datasets downstream tasks we use two further datasets. The MULTIID dataset is a collection of N = 10, 725 repertoires with binary disease and HLA labels. The EMERSON dataset matches that described in Emerson et al. (2017) and has both HLA and CMV labels. See Table[Toor an overview. We consider two TCR sequence selection approaches: HLADB, which is biased for HLA associations, and an unbiased GENPROB. See Appendix A.3 for more details on TCR sequence selection.
unlabeled. To demonstrate the performance
Table 1: Summary of the MULTIID and EMERSON repertoire datasets.
Dalasel
Category
Total
COVID 19
IIVV11
IISV.2
Parvo
CMV
Typed IILA
MULTILD
Train [Diseasc/HLA]
6,136
6,135
847
872
876
1,640
MULTILD
Test [Disease/HI .]]
4,590
4,590
220
225
204
388
EMERSON
Train [Diccase]
666
666
666
EMERSON
Test [Dicass]]
120
120
0
EMERSON
Train [HLA]
466
466
466
EMERSON
Test [HLA]
200
200
200
4.2 QUANTITATIVE RESULTS
We validate that inferred TCR embeddings encode immune genetics and exposure history using repertoire-level classification tasks. We employ L1/L2- disease-specific logistic regres- sion models, parameterized by ut" "b""""" R, Y Rd - y E f0, 1). given shared repertoire embeddings Rn E R. in Equation (8) and binary disease/HLA labels Y 5-disease and 145-HLA classification tasks. We tune the penalty parameter through 5-fold cross- validation.
for m 150;
Classifying HLA types Figures 5a and 5b show the area under the receiver operating charac- teristic curve (AUROC) performance of the repertoire embeddings on predicting 145 HLA types in the MULTIID cohort test data, matching the HLAs modeled in Zahid et al. (0224). Figure [a demonstrates the performance improvement in HLA classification as we scale from R 65, 000 to Ry 4 million HLA-associated TCRs. As expected the performance improves with increasing numbers of TCRs. Figure [5b] repeats this experiment but scaling up the number of repertoires used from the PUBLIC cohort to the TDETECT cohort with Rs 500,000 TCRs chosen in unbiased manner to be enriched for memory TCRs (see Appendix A.3). Again the performance improves with the larger repertoire set, although it is important note that the performance of the TCRs chosen in an unbiased manner is inferior to a set of sequences specifically chosen to predict HLAs, as expected. We observe a similar trend in the EMERSON cohort test data, see Appendix Figures TOaland and T0b
Classifying disease labels We select HLADB sequences with K E [65, 751; 360; 3, 796, 900), such that all sets have 30, 000 TCRs known to be associated with all our disease labels. The remaining sequences are randomly selected from HLAADB to achieve the desired K. Hence, by construction, we expect K = 65, 751 to provide an upper bound for our disease classi- fication tasks, which we wish to maintain repertoire embeddings on three disease classification tasks, COVID-19, Parvo virus and CMV. We compare to two baselines: i) DeepRC, BERT- TCR protein sequence model (Widrich et al.) 2020), and ii) ESLG, which selects disease-asee associated TCRs per endpoint using a case-control setup Emerson al. 2017). In Table 3
we scale K. Table 2 shows the performance of the
and then fits simple linear classifier (Greissl et al. 2021
7

---

Published as a conference paper at ICLR 2025
1.0
0.9
0.8
0.8
AUROC
0.3
0.3
02
0.2
65.55551
K-3,766.000
N-0,996
N-319
(a) varying HLADB K and fixed N 31, 938
(b) varying N and fixed GENPROB K zs 500, 000
Figure 5: JL-GLOVE (d 100) AUROC distribution for the binary classification of 145 common HLAs, given repertoire embeddings from Equation (8) using MULTIID test data. We demonstrate the impact of scaling the number of TCRs (a) and the number of repertoires (b) in measurements of X.
also show the ability of the repertoire embeddings to disentangle two homologous viruses, HSV-1 and HSV-2. Here, we benchmark JL-GLLVE against ESLG and AIRIVA, VAE-based model that uses similar TCR sequence selection to ESLG (Pradier et al. 2023).
Although we expected a drop in JL-GLOVE's disease classification performance as we scale K (at fixed d, see Theorem 2.1), impact varies across disease exposures, with the most signifi- cant impact on Parvo virus and HSV. This is anticipated, given that the simple mean pooling-based repertoire aggregation method becomes more sensitive to noise as we scale K, thus classifiers for discases with stronger overall immunc response, like CMV, remain more robust increasing K. We aim to derive an aggregation function capable classifying a wide range of pathogens and consider improving repertoire aggregation key future work. We also observe that results sensitive to the embeddings dimension as we scale TCRs K (see Figure 13). In general, we find that we need embedding dimensions d > 100 obtain reasonable disease prediction performance. Refer to Tadle[aand selection, which for disease classification results based on unbiased GENPROB sequence as performant HLADB sequences, but whose performance is invariant to the number of repertoires N.
Table 2: Comparison of ESLG, DeepRC, and JL-GloVe (d 100; N = 31, 938; varying HLADB TCRs K) disease-specific models on MULTIID and EMERSON test data sets. We report the median AUROC and sensitivity at 98% specificity, along with the 95% confidence intervals (CI) from 100 bootstrap samples.
Parvo
CMV
COVID-19
Sensitivity
AUROC
Sensitivity
AUROC
Sensitivity
AUROC
ESLG
0.30 = 0.16
0.73 0.06
0.63 + 0.38
0.93 4 0.01
0.70 0.06
0.95 = 0.04
DcepRC Wiidrich ot 2020
+0.002
IL-GloVe (K 751)
0.48 0,26
0.85 : 0.05
0.95 1 0.50
0.99 1: 0.02
0.86 0.03
0.97 2 0.01
JL-GloVe (K 300, 596)
0.30 0.14
0.76
0.07
0.98
+ 0.68
0.98 + 0.03
0.81 0.04
0.96 = 0.01
JL-GloNN (K 33, 7,6, 900)
0.01 0.07
0.53 0.10
0.39 + 0.49
0.92 + 0.05
0.59 0.06
0.92 = 0.01
5 INTERPRETING THE GEOMETRY OF TCR AND REPERTOIRE EMBEDDINGS
Neural word embeddings learned via algorithms such as GloVe and Word2 are known to pre- serve contextual semantic properties through linear vector arithmetic, iee. the word embeddings for "queen king
+ woman" (Mikolov al. 2013b) Levy & Goldberg 2014). Further, the GloVe objective function Equation I encourages TCRs that co-occur across repertoires, relative to their overall occurrence counts to have larger dot products. Essentially, they will have higher co- sine similarity (i.e., point in similar directions to each other in the embedding space) and may have larger embedding norms.
TCR embeddings cluster by antigen Figure 4a illustrates that the TCR embeddings learned by minimizing Equation (I) indeed capture the disease exposure and HLA TCR. Thus we expect that TCRs responding to the same antigen should have high cosine similarity, ie., point in similar
8

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of HSV disease models on MULTIID test data. We report the AUROC and sensitivity at 98% specificity, both overall and stratified by subtype. We present the median and 95% CI from 100 bootstrap samples for the models AIRIVA and ESLG from Pradier et al. (2023). and JL-GloVe (d 100; N = 31, 938; varying HLADB TCRs K).
Overall
HSV-2 negative
HSV-2 positive
HSV-I Model
Sensitivity
AUROC
Sensitivity
AUROC
Sonsitivity
AUROC
ESLG Pradier al.
2023
0.12 0.10
0.15
0.63
0.17
AIRIVA Pradher al
2024
0.30 0.12
0.74 1 0.09
0.35 1 0.20
0.74 0.10
0.32 1 0.22
0.67 1 0.16
IL-GloNe (K 751)
0.11
0.90
092 092-0000
+ 0.12 081#
VL-GloVe (K 390,
0.46 = 0.15
0.87 = 0.05
0.57 0.14
0.91 = 0.05
0.40 0.22
JL-GloVe (K = 7966900)
0.09 0.11
0.64 1 0.09
0.03 1 0.09
0.64 0.12
0.27 1 0.19
0.65 1 0.14
(a) HSV-1 Prediction Task
Overall
HSV-1 negative
HSV-1 positive
HSV-2 Model
Sensitivity
AUROC
Sensitivity
AUROC
Sonsitivity
AUROC
ESLG Pradier et al. 2023
0.11 0.10
0.75 + 0.07
0.16 + 0.21
0.79 + 0.16
0.12 0.15
0.75 + 0.10
AIRIVA
Pradier
0.37 0.18
0.78
0.26
0.86
0.10
JL-GloVe (K 751)
0.37 0.22
0.90 + 0.04
0.21 + 0.73
0.8
0.04
0.30 + 0.18
0.87 1 0.06
IL GlaVe(K 360.5001
0.25 0.16
0.86 1 0.05
0.21 1. 0.71
0.97 0.05
0.23 1 0.13
0,82 1 0.07
JL-GioVe (K 3. 796, 900)
0.08 0.10
0.72 + 0.07
0.07 0.22
0.77 0.12
0.12 0.14
0.70 1: 0.09
(b) HSV-2 Prediction Task
K=1,725,003
Enosediigg and CoreeC Cantiter
796.929
5=250
Sroeune Sepertvee
K-15,751
0.6
-62
Mar roy
Aat sed
Souckia osf
Mat osf
DiWoront Anllges
Sems Artigan
Sems Aniigon
Some Anrgan
Graps
(a) TCR embedding cosine similarity by antigen
(b) COVID-19 classifier logits by subgroup
Figure 6: JL-GLOVE (d = 100; N = 31, 938; varying HLADB TCRs K) distribution of (a) cosine similarity caleulated between TCR embeddings associated to the same antigen vs. different antigens and (b) the dot products (umm Rr,) between the un" weights of the MULTIlD COVID-19 classifier and TDETECT cohort repertoire embedded vectors Rr.
directions in the TCR embedding space. We directly test the claim with TCR antigen associations derived using a multiplexed identification of T cell receptor antigen specificity (MIRA) (Klinger al. 2015) assay, specifically designed to identify antigen specific TCRs. The MIRA assay (see A.7 for details) has been used to associate millions of TCRs to thousands of antigens from known pathogens. Here, we limit the analysis to SARS-CoV-2, Parvo virus, and HSV-1 antigens with associated TCRs. As expected, Figure 6a shows that TCRs associated with the same antigen have significantly higher cosine similarity. i.e,, point similar directions, than TCRs associated with different antigens. Importantly, this property is preserved the number of repertoires (see Figure Tla).
scale the number of TCRs K and
Classifier weights stratify repertoires by antigen Using previously developed models, we can predict with high precision and accuracy whether a subject in our sample has had a natural SARS- CoV-2 infection, vaccination but no natural infection neither (Pradier et al. 2023). Subjects with natural infection respond to a broad range of proteins derived from the SARS-CoV-2 In contrast, vaccines use only the spike protein and therefore vaccinated subjects elicit T cell response to a subset of all SARS-CoV- antigens derived from the spike protein. Figure 6b shows the distribution of the dot product of the weights of the MULTIID COVID-19 classifier and TDETECT cohort repertoire embeddings (u" , Rm). Interestingly, the dot product is largest for subjects with natural infection, followed by subjects who have been vaccinated and smallest for subjects who are neither vaccinated previously infected. This result indicates that the weights of the MULTIID COVID-19 classifier

---

Published as a conference paper at ICLR 2025
(2991)
OW (260))
ON (266))
Pane 1983)
Fore
ANEII 12236
ANG2O 12250
-20
-20
-20
-40
-40
-40
-00
-40
60
-40
By Disaase Senooure
HLA
BY Booss Exposure
Py
(a) S-SNE ESM-2
(b) E-SNE TCRdist
Figure 7: L-SNE comparisons of (a) ESM-2 (Rives al. 2021 and (b) TCRdist (hamming) (Dash) et al. 2017) embeddings derived from a subset of 2,800 disease- and HL.A-associated TCR amino acid sequences. We present the 1-SNE plot of subset of TCRs, colored by their disease exposure and HLA association.
generalize to the TDETECT cohort and represent a superposition of vectors that point in the direction of various antigens derived from the SARS-CoV-2 genome. The broader the immune response in terms of antigens, the greater the dot product with the classifier. However, we notice a smaller separation at K 3, 796, 900, which we attribute to the lack of robustness of the mean pooling function for large K and the need for 100. See Figure Tib for additional results on unbiased GENPROB sequences, demonstrating poorer repertoire stratification than HLADB sequences.
Comparisons to TCR protein sequence embedding approaches Embedding TCRs and reper- toires using TCR co-occurrence yields interpretable and biologically relevant geometric properties which are not captured by other embedding schemes. Figure demonstrates that TCR embeddings derived from protein models such as the pretrained ESM-2 (Rives et al. (02T) and protein distance based approach TCRdist (Dash et al. 2017) fail cluster the embedding space by HLA or disease exposure unlike our proposed JL-GILOVE co-occurrence based TCR embeddings shown in Figure 4 These results are not surprising since TCR amino acid sequences are randomly generated via the random V(D)J recombination (Tonegawa) 1983), which violates evolutionary assumptions made in protein models such as ESM-2. Consistent with findings from Nagano et al. (2024), the quality of the TCRdist embeddings is slightly better than ESM-2. However, TCRdist assumes that TCR proteins binding to the same antigen often share amino acid sequence similarity, i.e, small hamming BLOSUM distance, which does not generalize across all pathogens. Also, the dimensions of TCRdist emheddings increase with the number of TCRs, since = K, and computational com- plexity scales quadratically with the number of TCRs because the embeddings are derived from K2 comparisons.
CONCLUSIONS
Most data are unstructured, sparse, and heterogeneous, and representing such data is primary chal- lenge for modeling. Here, generate low-dimensional representations of TCRs and TCR reper- toires based on the co-occurrence of TCRs at scale. We propose JL.-GLOVE, which employs the GloVe algorithm learn immunologically meaningful TCR embeddings. Moreover, we improve GloVe's computational efficiency in terms of memory usage and training time by leveraging pow- erful initialization based on random projection theory. Further, the proposed JL-GLOVE algorithm is general and can be repurposed to learn embeddings in other data modalities where alignment is derived from co-occurrence statisties.
Extensive experimental results show that the repertoire embeddings summarize the immune genet- ies and pathogenic exposure histories of individuals. Notably, TCR embeddings cluster by antigens, and this property remains invariant as we scale the number of TCRs and repertoires. Moreover, as the amount TCR and repertoire data increases, these embeddings will continue to improve, enabling the quantification of more disease exposures for rarer HLA types. These embeddings can be combined with other modalities, such as single-cell RNA sequencing, to provide more informa- tion for individual T cells, and with clinical modalities, such electronic health records, to offer more subject-level information. Ultimately, personalized medicine and individualized treatments will require careful accounting of immune genetics and pathogenic exposure history.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for their insightful comments. This re- search was conducted as part of joint collaboration between Adaptive Biotechnologies and Mi- crosoft Research.
REFERENCES
Dimitris Achlioptas. Database-friendly random projections: Johnson-lindenstrauss with binary coins. Journal of computer and System Sciences, 2003.
Ayelet Akselrod-Ballin Davi Bock, R Clay Reid, and Simon K Warfield. Accelerating image reg- istration with the with electron microscopy. IEEE transactions medical imaging, 2011.
lindenstrauss lemma: Application to imaging 3-d neural ultrastructure
Luis Argerich, Joaquín Torré Zaffaroni, and Matías J Cano. Hash2vec, feature hashing for word embeddings. arXiv, 2016.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence embeddings. ICLR, 2017.
Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: applications to image and text data. In ACM SIGKDD, 2001.
F. M. Burnet. A modification of jerne's theory of antibody production using the concept of clonal selection. CA: A Cancer Journal for Clinicians, 1976.
Stefan Canzar, Van Hoan Do Slobodan Jelie, Sören Laue, Domagoj Matijević, and Tomislav Prusina. Metric multidimensional scaling for large single-cell datasets using neural networks. Algorithms for Molecular Biology, 2024.
Yuan Chen, Zhiming Ye, Yanfang Zhang, Wenxi Xie, Qingyun Chen, Chunhong Lan, Xiujia Yang, Huikun Zeng, Yan Zhu, Cuiyu Ma, et al. deep learning model for accurate diagnosis of infection using antibody repertoires. The Journal of Immunology, 2022.
Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and linden- strauss. Random Structures & Algorithms, 2003.
Pradyot Dash, Andrew Fiore-Gartland, Tomer Hertz, George Wang, Shalini Sharma, Aisha Souquette, Jeremy Chase Crawford, Bridie Clemens, Thi HO Nguyen, Katherine Kedzierska, et al. Quantifiable predictive features define cpitope-specific t cell receptor repertoires. Nature, 2017.
Mark M Davis and Pamela.] Bjorkman. T-cell antigen receptor genes and t-cell recognition. Nature, 1988.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 1990.
William S DeWitt III, Anajane Smith, Gary Schoch, John A Hansen, Frederick A Matsen IV, and Philip Bradley. Human cell receptor occurrence patterns encode immune history, genetic back- ground, and receptor specificity. Elife. 2018
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 201
11

---

Published as a conference paper at ICLR 2025
Ryan O Emerson, William S DeWitt, Marissa Vignali, Jenna Gravley, Joyce K Hu, Edward J Os- borne Cindy Desmarais, Mark Klinger, Christopher S Carlson, John A Hansen, et al. Immunose- quencing identifies signatures of cytomegalovirus exposure history and hla-mediated effects on the I cell repertoire. Nature genetics, 2017.
Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geom- etry of BERT, EL,o, and GPT-2 embeddings. EMNIP-IJCNLP,, 2019.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem training natural language generation models. ICLR, 2019.
Julia Greissl, Mitch Pesesky, Sudeb C. Dalai, Alison W. Rebman, Mark J. Soloski, Elizabeth J. Horn, Jennifer N. Dines, Rachel M. Gittelman, Thomas M. Snyder, Ryan O. Emerson, Edward Meeds, Thomas Manley, Ian M. Kaplan, Lance Baldo, Jonathan M. Carlson, Harlan S. Robins, and John N. Aucott. Immunosequenccing of the t-cell receptor repertoire reveals signatures specific for diagnosis and characterization of early lyme disease. medRxiv, 2021.
Austin L Hughes and Meredith Yeager. Natural selection at major histocompatibility complex loci of vertebrates. Anmual review of genetics, 1998.
Mark Klinger, Francois Pepin, Jen Wilkins, Thomas Asbury, Tobias Wittkop, Jianbiao Zheng, Mar- tin Moorhead, and Malek Faham. Multiplex identification of antigen-specifie cell receptors using combination of immune assays and immune receptor sequencing. PloS one, 2015.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transforerr: A framework for attention-based permutation invariant neural networks. In ICML, 2019.
Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In CoNLL, 2014.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In EMNLP, 2020.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 2023.
W Johnson Lindenstrauss and J Johnson. Extensions of lipschitz maps into a hilbert space. Con- temp. Math, 1984.
Damon H May, Steven Woodhouse, H Jabran Zahid, Rebecca Elyanow, Kathryn Doroschak, Matthew T Noakes, Ruth Taniguchi, Zheng Yang, John R Grino, Rachel Byron, al. Identi- fying immune signatures of common exposures through co-occurrence t-cell receptors in tens of thousands of donors. bioRxiv, 2024.
Grégoire Mialon, Dexiong Chen, Alexandre d' Aspremont, and Julien Mairal. A trainable optimal transport embedding for feature aggregation and its relationship to attention. In ICLR, 2021.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In ICLR Workshop Papers, 2013a.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. NAACL, 2013b.
Jiaqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In ICLR, 2018.
Yuta Nagano. Andrew Pyo, Martina Milighetti, James Henderson, John Shawe-Taylor, Benny Chain, and Andreas Tiffeau-Mayer. Contrastive learning t cell receptor representations. arXiv, 2024.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP., 2014.
12

---

Published as a conference paper at ICLR 2025
Melanie F Pradier, Niranjani Prasad, Paidamoyo Chapfuwa, Sahra Ghalebikesabi, Maximilian Ilse, Steven Woodhouse, Rebecca Elyanow, Javier Zazo, Javier Gonzalez Hernandez. Julia Greissl, et al. Airiva: deep generative model of adaptive immune repertoires. In MLHC. 2023.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 2021.
Harlan Robins. Immunosequenncing applications of immune repertoire deep sequencing. Current Opinion in Immunology, 2013.
Yanay Rosen, Yusuf Roohani, Ayush Agrawal, Leon Samotorcan, Tabula Sapiens Consortium, Stephen R Quake, and Jure Leskovec. Universal cell embeddings: A foundation model for cell biology. bioRxiv, 2023.
Tim Roughgarden and Gregory Valiant. Cs168: The modern algorithmic tooloxx lecture #4: Dimen- sionality reduction. Lecture Notes, 2024. URL.
061168110 68/1111.
Zachary Sethna, Yuval Elhanati, Curtis G Callan Jr, Aleksandra M Walezak, and Thierry Mora. Olga: fast computation of generation probabilities of b-and t-cell receptor amino acid sequences and motifs. Bioinformaticx, 2019.
Noam Shazeer, Ryan Doherty, Colin Evans, and Chris Waterson. Swivel: Improving embeddings by noticing what's missing. arXiv, 2016.
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. Baseline needs more love: On simple word- embedding-based models and associated pooling mechanisms. In ACL, 2018.
John-William Sidhom, H. Benjamin Larman, Drew M. Pardoll, and Alexander S. Baras. Author correction: DeepTCR is deep learning framework for revealing sequence concepts within t-cell repertoires. Nature Communications, 2021.
Susumu Tonegawa. Somatic generation of antibody diversity. Nature, 1983.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Michael Widrich, Bernhard Schäfl, Milena Pavlovic, Hubert Ramsauer, Lukas Gruber, Markus Hol- zleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al. Modern hopfield networks and attention for immune repertoire classification. In NeurlPS, 2020.
Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scott Shenker, and Ion Stoica. Spark: Cluster computing with working sets. In 2nd USENIX workshop on hot topics in cloud computing (HotCloud 10), 2010.
H Jabran Zahid, Ruth Taniguchi, Peter Ebert, I-Ting Chow, Chris Gooley, Jinpeng Lv, Lorenzo Pisani, Mikaela Rusnak, Rebecca Elyanow, Hiroyuki Takamatsu, et al. Large-scale statistical mapping of t-cell receptor 33 sequences to human leukocyte antigens. BioRxiv, 2024.
Wen Zhang, Peter G. Hawkins, Jing He, Namita T. Gupta, Jinrui Liu, Gabrielle Choonoo, Se W. Jeong, Calvin R. Chen, Ankur Dhanik, Myles Dillon, Raquel Deering, Lynn E. Macdonald, Gavin Thurston, and Gurinder Atwal. A framework for highly multiplexed dextramer mapping and prediction of t cell receptor sequences to antigen specificity. Science Advances, 2021.
Yu Zhao, Bing He, Fan Xu, Chen Li, Zhimeng Xu, Xiaona Su, Haohuai He, Yueshan Huang, Jamie Rossjohn, Jiangning Song, and Jianhua Yao. Deepair: A deep learning framework for effective integration of sequence and 3d structure to enable adaptive immune receptor analysis. Science Advances, 2023.
Rolf M Zinkernagel and Peter C Doherty. Restriction of in vitro cell-mediated cytotoxicity in lymphocytic choriomeningitis within a syngeneic or semiallogeneic system. Nature, 1974.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 JL NORMALIZATION PROOF
Proof.
We adapt the following proof from the discussions in Section 4 of Roughgarden & Valiant
2024).
Let us define
w = Σ c,))P(G,m)
==1
where m & (1,2,...,d and the JL transform / (m) = va 20% see Equation 2 (m) a zero-mean random variable with variance
Note
-a0)==10 ||xl|?
hence [m))) ||cx||². Consequently, we have
IL w
w (m)²
k
w (m)
(9)
m
Note that Equation ( is an average of unbiased estimators of ||cx|l. By central-limit theorem, we have
11 & ||clll
(10)
Finally, we note
1 vd
1 (cx, P) w
w w
IL W
(cx, P)
(11)
Vd cllll ||xx||
The last step Equation is due to Equation (TO) for large d. Specifically, one can first compute W:LL following Equation linear-time, and then normalize the rows of W to ger WJL This approach is approximately equivalent to the quadratic-time time alternative of computing and normalizing C' before projecting via P to obtain WJJ using Equation 66.
A.2 ACCOUNTING FOR VARYING MARGINAL OCCURRENCES OF TCRs
25000
20000
Counts
15000
10000
5000
0
Marginal Occurrence
Figure 8: Marginal occurrence counts of = 3, 796, 900 TCRs computed from = 31, 938 repertoires.
14

---

Published as a conference paper at ICLR 2025
It is reasonable to expect ||c, - -cill to be smaller when TCRs f, and t, frequently co-oceur, and larger otherwise. However, this can be misleading due to the different marginal occurrence frequen- cies of TCRs, which are power-law distributed in case; see Figure g Consider the following illustrative co-occurrence matrix C of TCRs:
0
01
0
0
0
C
8
0
Č
0
0
(12)
75
60
0
0
0.78
0.63
60 80]
Unit-normalize the rows
0
0
0.6
0.8
Note that 1 and 12 never co-occur, while f3 and 1 co-occur frequently. Despite this, we have ||e| - c2ll 10 which is smaller than ||c3 call 25. The main reason that e₁ and co have smaller norms than C3 and C4 because & and &2 are rarer TCRs. This observation suggests that we should remove the effect of the TCR embeddings' norms from the pairwise L2 distances across C; before leveraging the random projection theory for dimensionality reduction (see Theorem 2.1
A.3
TCR SEQUENCE SELECTION
The probability of a specific TCR (amino acid sequence) being generated varies by ~2 orders of magnitude ranging for 10-6 10-30 (Sethna et al., 2019). Thus, TCRs with high likelihood of random generation will necessarily co-occur with one another. We are only interested in model- (í.e., memory TCRs) as these TCRs carry meaningful information about immune genetics and pathogenic exposure history. We employ mul- tiple selection strategies to enrich our sample for memory TCRs. Zahid et al. (0024) use labels of
ing co-occurrence of TCRs that are antigen-experieneee
HLAs identify sequences that have strong statistical association to HLAs, meaning they likely memory TCRs. We refer to these sequences as HLADB. Refer to May et al. (2024) for more details of how the set used here derived.
We also select set of TCRs enriched for memory an unbiased manner requiring no labels using combination of the TCR generation probability and the observed frequency in repertoires. The generation probability provides naive prior on the expected frequency; TCRs that are memory undergo clonal expansion and therefore will likely be observed at a higher frequency in repertoires than the naive expectation from the generation probability (Det III et al. 2018). We empirically determine the observed frequency cutoff function of generation probability using our set of sequences from HLADB. In other words, using HLADB which is set of sequences known to be enriched for memory TCRs, we derive the distribution of observed frequencies for those sequences relative to their naive expectation frequency given by the generation probability and derive cut- off. We then use this cutoff to select from all TCRs in our repertoires which does not require any repertoire level labels. We refer to these sequences GENPROB:
Log Normalized Frequency
10
Model
Mean
-2
-3
-12
-11
-10
-9
-8
Log Generation Probability
Figure 9: Normalized frequency of 4 million HLADB TCRs as a function of generation probability. Red points show the mean normalized frequency equally populated bins generation probability.
Orange and blue shaded regions indicate the central limits containing 68% (1o) and 95% (2x) of the data, respectively. The black line is fit to the data the range indicated according to Equation (33). The deviation of the normalized frequency from a power-law at low generation probability consequence of sample size; given a larger number of repertoires, the power law relationship would extend to lower values of generation probability.
15

---

Published as a conference paper at ICLR 2025
Figure 9 shows the distribution of normalized frequency of TCRs function of generation proba- bility. The mean normalized frequency as a function of generation probability is a power law which is fit by line (black curve in Figure 9
in log-log space given by
log Jobs Pgen) = mmogg0 Pgen + 9) c.
(13)
where m 0.797 is the slope and с -2.213 is the intercept which is defined the mean at a generation probability of 10 (i.e. 1o Pgen + 9); log10 fobs and log10 gen are the logarithm base 10 of the normalized frequency of TCRs (i.e., fraction of repertoires in which the TCR is observed) and logarithm base 10 of the TCR amino acid sequence generation probability calculated using OLGA (Sethna et al. 2019), respectively. Let ((Pgen) == log Sobs()) denote the value of 1og10 sobs a fixed generation probability Pgen- We fit ( gen with Gaussian distribution with mean μ = 1og (Ple((' and standard deviation a 0.473, which is constant and independent of generation probability (see shaded regions in Figure 2 The full distribution is defined by our power-law fit, which defines the Gaussian mean, and the invariant standard deviation. Thus, we use the quantile function to select sequences at fixed percentile probability т, s.t. inf R P (Y(Pgen gen = 10-9) <)) J.. Specifically, to select sequences at a specific percentile probability t, we set с + č in Equation (3). This yields an observed frequency threshold that increases with generation probability and is at fixed percentile č of the empirical distribution of HLADB TCRs, as observed in TDETECT samples. Note, when т = 0.5, then ė = and we recover the mean fit shown by the black line in Figure [ We choose to yield approximately 500, 000 sequences in the TDETECT and PUBLIC cohorts. This selection requires us to set 1 7 to 0.015 and 0.085 for TDETECT and PUBLIC cohorts, respectively.
A.4 COMPUTING TCR CO-OCCURRENCES
Generating C' is computationally expensive, and although C' is sparse, as the number of TCRs K grows, it can become very large. Thus, computing T X in one step can become intractable. For- tunately, this operation is easily parallelized with Spark (Zaharia et al. 2010) by distributing the computation along either rows or columns of X, as shown in Algorithm
Algorithm 1 Computing TCR co-occurrences C
Input: TCR repertoire binary matrix X Output: TCR co- co-occurrences C
1: while $ < K do
2: Given the n-th individual's repertoire Cn. filter w. to generate &n. which represents the
presence or absence of the subset of TCRs ī {ta...., to} Vn s.t. B = 100 and
TC
3: Join & and In on n to generate all TCR pairs t; in T and t, in 7
4: Group by (i,j) across all to create Ć,
5: end while
6: Union all C. to create C
7: return C
A.5 COMPUTING JL EMBEDDINGS
Algorithm 2 provides the sparse matrix algorithm for computing the JL-Norm embeddings 11 which are used to initialize GloVe embeddings Equation (I).
A.6 TRAINING JL-GLOVE
We initialize the TCR embeddings W with the JL-Norm embeddings JL and use the Adagrad op- timizer (Duchi et al.) 201T) with learning rate 0.05 to minimize the GloVe objective in Equation (T)
via stochastic gradient descent minibatches from C. We leverage the distributed Dask framework to load minibatches (partitions) of C, Lee, millions of entries stored as Parquet files Further,
httts::/www.daskkkt
https: wwwp://uut..aaaa......eeea /rgg
16

---

Published as a conference paper at ICLR 2025
Algorithm (Approximately) Computing J.-Norm embeddings W.JL
Input: TCR repertoire binary matrix X
Output: Approximation to W (see Proposition 2.2
1: Construct the projection matrix P by sampling its elements i..dd following equation
2: Compute M1 XP
3: Compute M2 M
4: Compute M = M2 M2 (see equation 2
5: m, f [m. for all rows m, of M
6: return
we use PyTorch Lightning for distributed data parallel training across one node equipped with
NVIDIA A100 80GB GPUs (if K 2 500, 000) or 2 Tesla V100 16GB GPUs (if K < 500, 000).
A.7 MIRA ASSAY DESCRIPTION
In brief, a panel of antigens associated with specific proteins are presented on HLAs in specific wells. T cells taken from blood draw separated and put into the solution. T cells that respond to these antigens are activated. Activated T cells are identified via surface proteins and sorted out for TCR sequencing. Different subsets of antigens are present in different wells and using combinatoric reconstruction, specific T cells are associated with specific antigens.
A.8 SCALING LAWS
Figure 10a and Figure T0b demonstrate that quality of the embeddings improves as we scale number of TCRs K and repertoires N. We quantify the scaling impact with AUROC distribution across 145 HLAs on the EMERSON cohort test data.
0.9
0.8
0.8
AUROC
0.6
0.5
0.5
0.3
0.2
0.2
0.0
K-65,55
366,5996
K-3,796,900
N- 3,996
N N 31,938
(a) varying HLADB and fixed N 31, 938
(b) varying N and fixed GENPROB K RJ 500,
Figure 10: JL-GLOVE HLAs, given repertoire embeddings from Equation (8) using EMERSON test data. We demonstrate impact of scaling the number TCRs (a) and the number of repertoires (b) measurements
100) AUROC distribution of the binary classification of 145 common
A.9 INTERPRETING THE GEOMETRY TCR AND REPERTOIRE EMBEDDINGS
Figure Tla shows that GENPROB TCRs associated with the same antigen have significantly higher cosine similarity, i.e., they point similar direction than TCRs associated with different antigens. However, observe lower cosine similarities for similar antigens when the number of repertoires N is lower.
Next, we inspect the dot products between the COVID-19 logistic regression classifiers' weights and TCR embeddings " RR,). Figure lib does not show the same separation observed Figure 6b in terms of the dot product, between subjects with natural infection, vaccinated individuals, and
https://Tightningg
17

---

Published as a conference paper at ICLR 2025
those who are neither vaccinated nor previously infected. We attribute this discrepancy to: i) the less performant GENPROB TCR sequence selection, resulting in a weak COVID-19 classifier u" (see Table44; and di) the use of a simple mean pooling repertoire aggregation function, which is not robust to noise when K is too large. Additionally, we find that the TCRs known to be associated with disease/HLA labels (i.e, enhanced sequences) have significantly larger dot products with the corresponding classifier's s weights. We depict this in Figure
12
33, ..E
Dor Pooacct of Resuturss
0.30
0.6
0.03
Embsottng
10.00
-0.15
vid
Be Gartiet
Mat
-2.4
Difeerr Artigens
Same Anilgws
Game Antisen
Same HSVI
Coote-15
Parvodiis
(a) TCR embedding cosine similarity by antigen
(b) COVID-19 classifier logits by subgroup
Figure 11: JL-GLOVE (d 100;
500, 000 GENPROB TCRs; varying N repertoires) distri- bution of (a) cosine similarity calculated between TCR embeddings associated to the same antigen vs. different antigens and (b) the dot products (umT Ra) between the ut weights of the MULTIID COVID-19 classifier and TDETECT cohort repertoire embedded vectors R.
[|- [1-Krmm
300k TCRx with 300d emoeddinggs Eine-uning with concrneece matris eppchs
1.0
2.5
2.0
2.0
3.
1.3
10
1.0
0.5
0.8
Randam Iill. 360k TCBs with 300eim, enboddings Full baahing with entire cooccurance matris for 100 epochs
0.5
155
murth
2.0
155
0.4
0.4
02
0.3
022
-25
0.5
25
5.0
embeddings. turing cooccurance matrix epochs
2.0
Give
DR21 BTT3
2.0
0.5
0.4
0.4
2.00
0/
0.3
125
02
00
0.5
1.00
0.0
69
Random Inf: 05k TERs with 300 embecdings..o Full traiting with entire cooccurence matris for 100 epocht
0.5
0.5
and
0.5
3.0
0.5
0/6
02
34
02
-2.5
5.0
-2.5
6.3
-2
dotted wilh TEN crrcoadings
doticl with C1
duttod U TOX onbcoaeeg
defice with TON crocodagg
detiod With TOX onlecciigg
Figure 12: Logistic regression classifier weights dotted with TCR embeddings (u"", wyy), respec- tively. TCRs that are enhanced sequences (ES) for the corresponding disease/HLA labels point in the direction given by the classifier's 's weight, i.e., they yield higher cosine similarity.
A. QUANTITATIVE CLASSIFICATION RESULTS
We present additional quantitative classification results for JL-GloVe (d = 100; GENPROB K Rs 500, 000; varying N) in Table 4 and Table 5 Compared to Table 6 and Table 7
we note that the
18

---

Published as a conference paper at ICLR 2025
importance of TCR selection K is greater than that of repertoire measurements N used to train JL- Glo on downstream classification tasks. Further, Figure [3] highlights that weaker signals, such as the Parvo virus, are highly sensitive to the embedding dimensions d as we scale TCRs K; i.e., we need higher embedding dimensions to capture such signals using a simple mean pooling-based repertoire embedding aggregation function.
Table 4: Comparison of ESLG, DeepRC, and Ve (d = 100; GENPROB K ~ 500, 000; vary- ing N) disease-specific models on MULTIID and EMERSON test data sets. We report the median AUROC, AUPRC, and sensitivity at 98% specificity, along with the 95% confidence intervals (CI) from 100 bootstrap samples.
Parvo
CMV
COVID-19
Model
Sensitivity
AUROC
AUPRC
Sensitiity
AUROC
AUPRC
Sensitivity
AUROC
AUPRC
ESLG
0.30 + 0.16 0.73 + 0.06
0.86 + 0.05
0.63 + 0.38
8:95 + 0.01
0.88 + 0.08 0.70 + 0.05 0:95 + 0.04 0.84 + 0.0
DeeRCC
Wilhish at 2000
0.83 +0002
CV 3, 3,996)
0.08 + 0.08
0.58
+ 0.09
0.77
+ 0.04
0.11 +
0.71 + 0.09
0.66 + 0.07
0.47 + 0.04
0.86 + 0.2
0.64 + 002 0.53 + 0.02
BL-GooNe (N- 31, 9350
0.02 + 0.00
0.40 0.00
0.73
0.04
0.64 + 0.59
0:93 + 0.05
0.92 + 0.03
0.35 + 0.05
0.82 + 0.02
Table 5: Comparison of HSV disease models on MULTIID test data. We report the AUROC, AUPRC, and sensitivity at 98% specificity, both overall and stratified by subtype. We present the median and 95% CI from 100 bootstrap samples for the models AIRIVA and ESLG from Pradier et al. (2023), , and Ve (d = 100; GENPROB K ~ 500, 000; varying N).
HISV-2 negalive
ISSV-2 positive
HSV-1 Model
Senstiviity
AUROC
AUPRC
Senvitivity
AUROC
AUPRC
Sensitivity
AUROC
AUPRC
Pradie 2033
0.12 0.10
0.62 0.00
0.18 0.15
0.63 0.12
0.14+ 0.17
0.19
AIRIVA
0:112
0.74.1
0.35
0.74
0.22
0.16
3,908
0.21
0.70
1004
0.38
0.8
GloVe 31, 635)
0.15
0.09
0.38
1.007
(a) HSV-1 Prediction Task
Oversil
BSV-1 negative
HSV-1 positive
HSV-2 Model
Sensiivity
AUROC
AUPRC
Sensiivvit
AUROC
AUPRC
Senviivity
AUROC
AUPRC
ESLG
Profiee al 2023
0.11 2.10
0.78 0.07
0.16 1 0.21
0.79 = 0.16
0.12 0.15
0.75- 0.15-0.10
AIRIVA Prodtor ar 2023
0.37 0.18
0.78 128-000 0.10
0.57 + 0.26
0.85 - 0.12
0.22+00.00
0.77 177--0.00 0.10
IL-GloVe (N 3, 906
0.07 0.14
0.74 0.07
0.57 a 006
0.14 0.24
0.77 = 0.15
082 2012
002 0.11
0.70
0.08
0.54 1:44::000
GloVe LV 9389
0.17 0.11
0.74 0.06
0.62 à 0.05
0.09 2.17
0.68 = 0.16
0.47 1011
019 20.12
0.78 0.08
0.67 1:00
(b) HSV-2 Prediction Task
Table 6: Comparison of ESLG, DeepRC, and VL = 100; N = 31, 938; varying HLADD TCRs K) disease-specific models on MULTIID and EMERSON test data sets. We report the median AUROC, AUPRC, and sensitivity at 98% specificity, along with the 95% confidence intervals (CI) from 100 bootstrap samples.
Parvo
CMV
COVID-19
Model
Sensitivity
AUROC
AUPRC
Senstivity
AUROC
AUPRC
Sensitivity
AUROC
AUPRC
0.30 + 0.16
0.73 + 0.06
0.05
0.63 0.38
0.93 0.0
0.88 0.08
0.20 0.06
0.95
0.84
DeepRC (Widrch 2020)
2010
0.83
_0,002
Il-Gleee (K 65,781 15
+ 0.26
0.85
0.05
0.94 + 0.01
0.95 0.50
0.90 0.02
0.98
0.86
0.97-
0.92
Glove (K 200, 506)
0,50 1 0.14
0.76
J: 0.07
0.90 J: 0.02
0.9%
0.68
0.98
0.03
0,95 0.03
0.81 1: 0.04
0.96 0.01
0.90 0.01
JL GloVe OK- 3. 706, 900)
0.01 1 0.07
0.53 1. 0.10
0.73 1. 0.04
0.39 1 0.49
0.92 0.05
0.88 0.06
0.59 0.06
0.92 0.01
0.75 0.02
A.11 COMPARISONS OF GLOVE INITIALIZATION SCHEMES
First, we set K = 65, 751 and use a set of TCRs that are known to be associated with the set of labels we are interested in. Figure 14 provides a clear picture of how JL-Norm initialization compares to full GloVe- training with random initialization of TCR embeddings. We see that JL- Norm embeddings already yield non-trivially good performance metrics, which quickly reaches full-training level performance after fine-tuning on a very small portion of C' for only few epochs. Note that same level of training with random initialization performs very poorly. Next, we increase to K = 360, 596 to assess how well can our methods scale and stay robust to noise as we include more and possibly unrelated TCRs for the tasks we consider, see Figure 15 We note that fine- tuning JL vs. full training compares similar to before, and the performance decrease compared to K = 65, 751 is not significant in general. This is critical as we would like to include as many TCRs as possible in our analyses to include more A/expoosre coverage (see Section 4.2.
19

---

Published as a conference paper at ICLR 2025
1.0
1.0
AUROC for Parvo virus
0.9
AUROC for Parvo virus
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
500
Embed Dim (d) for K = 65,751
Embed Dim (d) for K = 360,596
(a) Parvo virus for K = 65, 751
(h) Parvo virus for K = 560,596
596
1.0
1.0
0.9
0.9
AUROC for Covid
0.8
JROC for Covid
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
10 50 100
300
500
0.0
10
50
100
300
Embed Dim (d) for 65,751
Embed Dim (d) for 360,596
(c) COVID-19 for K = 65, 751
(d) COVID-19 for K 360, 596
Figure 13: JL-GLOVE AUROC for COVID-19 and Parvo virus across varying numbers of TCRs K and embedding dimensions d on the MULTIID test dataset. We report the median AUROC with 95% CI from 100 bootstrap samples.
Table 7: Comparison of HSV disease models on MULTIID test data. We report the AUROC, AUPRC, and sensitivity at 98% specificity, both overall and stratified by subtype. We present the median and 95% CI from 100 bootstrap samples for the models AIRIVA and ESLG from Pradier et al. (003)). and VL (d = 100; N = 31, 938; varying HLADB TCRs K).
negative
HSV-2 positive
HSV-1
Senstivity
AUROC
AUPRC
Sensitivity
AUROC
Senstivity
AUPRC
Prasiee
0.10
0.62 0.09
0.8
063
AIRIVA
0.12
0.35
0.74
0.32
H-CGWWa (K 66, 751)
0.90
0.96
0.58
0.92
0.7
IL GleVe 360, 596)
0.16
0.15
0.87
0.08
0.94 0.001
0.37 0.14
091 0.05
0.96
0.0
0,00 0.22
0.1 0.11
0.93 003
BL-GloNe UK 900)
0.09 0.11
0.64
0.09
0.82 0.033
0.03 1 0.09
0.64 1 0.12
0.76
0.0000
0.27 0..19
0.65 0.14
0.88 0.005
(a) HSV-1 Prediction Task
Oversil
HSV-1 negative
HSV-1 pooltive
BSV-2 Model
Sensiiivity
AUROC
AUPRC
Senvilivity
AUROC
AUPRC
Sensilivity
AUROC
AUPRC
ESLG Pradiier al 2013
0.11 0.10
0.75 0.07
0.16 2
0.79 = 0.16
0.15::0:15
0.75 0.10
AIRIVA Prsterm 2025
0.37 0.11 0.15
0.78 -06=00 0.10
0.57 + 0.26
0.86 - 0.12
0.32++0.00
0.77 077-0..0 0.10
TL-GleVe (K 65, 751)
0.37 0.22
0.90 0.04
0.84 1 0.04
0.31 0:73
0.98 004
091 4 0.09
0.30
0.87
0.06
0.80 :
IL. GloVer 350)
0.16
0.00
0.78
097 0.8
0.91 000
2-13
0,07
0.73
Glo 8, 900)
0.08 0.10
0.72 0.07
0.55 0.06
0.07
0.77 0.12
0.54 1 0.11
1.12
0.70 0.09
0.56
(b) HSV-2 Prediction Task
20

---

Published as a conference paper at ICLR 2025
AUROC
JL-Norm (2%, 3ep)
Jo-Norm (0.2%. 5ep)
JL Norm zero shat
RI (2%) 3ep)
AI (0.2%, Sep)
0.4
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
1.0
0.6
0.7
0.8
0.9
1.0
HLA:AM02001
Figure 14: K 65, 751 TCRs used. Area under receiver operator characteristics (AUROC) and precision-recall curves (AUPRC).
RI, 65k TCRs, 100dim. (100%. 100ep)
AUROC AUPAC
300dim. (100%. 100ep)
Norn,, 360k TCRs, 300dim. (2%. Sep) No-r 300dim.
zero-shot
0.6
0.7
0.8
0.9
0.6
0.8 cohort)
1.0
0.90
0.95 HLA-APO2*01
1.00
Figure 15: K 360, 596 TCRs used. Area under receiver operator characteristics (AUROC) and precision-recal curves (AUPRC).
21


Reviews:
[{"review_id": "LtCYkGhMS8", "reviewer": "Reviewer_Qp7p", "summary": "In this study, the authors developed a method to derive low-dimensional representations of T cell receptors and subject-level repertoires in feature space. To enhance computational efficiency, the method employs random projection theory.", "questions": "In the weaknesses section, it would be beneficial to provide further illustrations regarding the biological background, the methodology employed, and a detailed explanation of why the model is effective.\n\nMoreover, in addition to reporting AUC and sensitivity, the authors should also include other relevant metrics such as specificity, positive predictive value (PPV), negative predictive value (NPV), and overall accuracy. It is important for the authors to clarify how the cut-off points for these metrics were determined, as this information is crucial for understanding the model's performance and its clinical applicability.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well-organized and clearly written.\n\nThe proposed method is technically sound.\n\nThe application of random projection theory to enhance computational efficiency, particularly regarding memory usage and training time, is noteworthy.", "weaknesses": "The biological definitions presented in the study are somewhat unclear. For instance, when the authors refer to TCR embedding, it is important to specify whether they mean both the TCR alpha and beta full chains, the CDR3 regions of both chains, or only the CDR3 region of the TCR beta chain. Additionally, do the authors take into account V(D)J gene information when using the CDR3 region?\n\nGiven that TCRs are highly cross-reactive, the authors need to provide further explanation on why using co-occurrence information alone is effective for TCR embedding.\n\nThe repertoires of different subjects contain varying numbers of TCRs. How do the authors address this variability when representing them with a matrix of the same TCR dimensionality?\n\nConsidering the high cross-reactivity of TCRs, how do the authors define the TCR-level ground truth without relying on wet-lab-based experiments?\n\nWhen discussing classification tasks, it would be helpful to clarify whether the focus is on receptor-level classification or repertoire-level classification. Furthermore, given different receptors have clone frequencies within the repertoire, it appears that the authors do not consider clone frequency in their repertoire-level embedding.\n\nThe interpretation of deep learning models is crucial for clinical applications; however, the authors have provided limited results in this area.\n\nLastly, there is a noticeable lack of comprehensive comparisons with state-of-the-art works such as DeepTCR, TCRAI, DeepAIR, and DeepRC, which should be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gwt6qa2Nhj", "reviewer": "Reviewer_ovF4", "summary": "The authors present JL-GLOVE, a scalable algorithm for generating low-dimensional embeddings for T cell receptors (TCRs) and TCR repertoires using TCR co-occurrence data. The main idea is to leverage the co-occurrence patterns of TCRs that target the same antigen to learn meaningful representations. To address the computational challenges of large-scale TCR data, the authors introduce the JL-GLOVE method, which combines GloVe with random projection theory. This approach improves memory efficiency and speeds up the training process. They then aggregate these TCR embeddings to generate subject-level embeddings, providing a low-dimensional representation of an individual's immune history. The embeddings show that TCRs targeting the same antigen exhibit high cosine similarity, and aggregated repertoire embeddings correlate with immune profiles, supporting disease prediction and HLA inference tasks. Results demonstrate the utility of these embeddings for predictive modeling and potential applications in personalized medicine by integrating them with other data modalities.", "questions": "- The paper benchmarks JL-GLOVE against ESLG and AIRIVA for disease classification tasks. The authors can include a more comprehensive comparison with other deep learning models specifically designed for TCR repertoire analysis (DeepTCR, DeepID etc).\n\n- The authors observe that the disease classification performance is sensitive to the embedding dimension (d) and the number of TCRs (K). A more systematic exploration of the impact of these parameters can be done A more detailed analysis of the impact of different embedding dimensions across various dataset sizes would be valuable.This would aid other researchers in configuring JL-GLOVE for datasets of different sizes or resolutions, thereby increasing the framework’s accessibility and practical utility.\n\n- Presenting one or two practical case studies where JL-GLOVE embeddings provide actionable insights in a real-world immunological context (e.g., identifying rare disease signatures) would further emphasize the method’s applicability.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The use of the JL transform significantly improves the computational efficiency of the GloVe algorithm, enabling the analysis of large datasets containing millions of TCRs. The authors demonstrate that JL-GLOVE achieves good performance using only a fraction of the co-occurrence data, making it suitable for handling the increasing scale of TCR repertoire sequencing data.\n\n- The embeddings produced not only capture the co-occurrence patterns among TCRs but also demonstrate clustering by antigen specificity and HLA association. This biologically meaningful structure aligns with immune response patterns and enhances the interpretability of the embeddings, which is valuable for immunological research and practical applications like personalized medicine.\n\n- The paper rigorously validates the embeddings’ effectiveness through multiple downstream tasks, including disease classification and HLA inference. The experiments demonstrate the robustness of the embeddings to scale, supporting their utility in predicting immune response profiles across various pathogens, and showcasing meaningful performance improvements with larger datasets.", "weaknesses": "- The authors compare JL-GLOVE to protein sequence-based embeddings (e.g., ESM-2 and TCRdist), which are structurally different from co-occurrence embeddings. While this comparison is useful, the paper could benefit from a broader comparison with other immunology-focused embedding techniques, such as contrastive learning methods or graph-based embeddings, which may capture additional biological context.\n\n- The paper relies primarily on a mean pooling approach for aggregating TCR embeddings at the repertoire level, which, while straightforward, may be overly simplistic. This method is prone to noise, especially as the number of TCRs (K) increases, potentially limiting classification performance for diseases with more subtle immune signatures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mBYLdrG8Kk", "reviewer": "Reviewer_x8Nf", "summary": "This paper develops JL-GLOVE, a method for creating vector representations/embeddings of T-cell receptors (TCRs) and immune repertoires that capture meaningful biological relationships. The method leverages TCR co-occurrence across patient repertoires, adapting the GloVe algorithm from natural language processing while incorporating the Johnson-Lindenstrauss transform for computational efficiency. TCRs are embedded such that those targeting the same pathogen have similar vector representations, and patient repertoires are represented by averaging their constituent TCR embeddings. The resulting embeddings successfully encode both immune genetics (HLA types) and pathogen exposure history, improving as more data is added, and outperform baseline methods on disease prediction tasks. The authors demonstrate their method's scalability and interpretability, showing it can process millions of TCRs while maintaining performance, though they note that the simple averaging approach for patient-level representations could be improved. By creating these biologically meaningful representations, the work provides a foundation for quantifying immune system similarity between individuals and could assist in personalized medicine applications.", "questions": "- I believe this paper should be in the topic area of applications to physical sciences (biology / immunology) rather than unsupervised, self-supervised, semi-supervised, and supervised representation learning. There are few novelties in terms of methods development, but the application of these methods are extremely impactful and a nice way to show the power of representation learning. \n- You mention other methods for set level representations, why not use those? Average is nice in it's simplicity, but does this simplicity cost performance? Would be nice to see a benchmark against set representation methods. You may also be interested in OTKE method for set level representation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- Originality: the development of TCR embeddings and immune repertoire representations is original and an under studied area in the representation learning community. The application of Glove algorithm here fits nicely and works. \n- Quality: the produced results are of high quality and provide a significant impact to the field\n- Clarity: the paper is very clear to read and understand. The authors give the right amount of biological/immunological background to understand the paper and why it is important. \n- Significance: this is a very significant and meaningful contribution to the field of personalized medicine. The application of representation learning for TCRs and immune repertoires is a great step towards better medicine.", "weaknesses": "- The novelties of the paper are not the representation learning method itself. The paper applies Glove algorithm with a few modifications that help it work better, but are not necessarily innovations in of themselves. \n- As stated, the immune repertoire method of taking the average, is nice and works, it could be further developed by other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYX2d40x1j", "reviewer": "Reviewer_DdmS", "summary": "This paper introduces a scalable approach to generating T-cell receptor (TCR) embeddings by leveraging the GloVe algorithm, adapted with the Johnson-Lindenstrauss (JL) transform for improved computational efficiency. The approach aims to create subject-level embeddings of TCR repertoires, which capture immune genetics and pathogenic exposure history. It employs a co-occurrence-based model to detect immune-related patterns and provides an aggregation of TCR embeddings at the subject level, which the authors claim could serve in predicting diseases and HLA types.", "questions": "- How does the model handle rare but potentially significant TCRs? Given the emphasis on co-occurrence, it is unclear how rare TCRs are represented, as these could provide unique insights in immune responses but may not frequently co-occur with other TCRs.\n- The current validation relies primarily on logistic regression without exploring other classifiers or model interpretability techniques, which would strengthen the paper’s claims on model generalizability and utility.\n- How does the embedding perform across other immunological datasets?\n- Additional clarity on computational scaling challenges would be helpful, especially given the potential high-dimensional space of TCR repertoires.\n- What additional features could improve biological relevance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The approach is somewhat novel in applying co-occurrence modeling, inspired by NLP, to TCR data. Leveraging random projection (JL transform) to enhance GloVe's performance also demonstrates creativity in handling large datasets.\n- The paper is well-organized with clear methodological sections, providing figures and tables to explain the model architecture and performance comparisons.", "weaknesses": "- While the method adapts the GloVe and JL transform for TCR analysis, there is limited advancement in the biological interpretability of embeddings over existing approaches.\n- The paper lacks rigorous benchmarks against established methods beyond simple logistic regression. Disease and HLA classification tasks do not adequately demonstrate the model’s robustness, especially with limited sensitivity for certain conditions (e.g., HSV) at larger embedding scales.\n- While clustering by disease and antigen provides some interpretative insight, the embeddings’ clinical relevance is unclear.\n- Despite using the JL transform to improve scalability, the computational requirements for large-scale TCR data (e.g., 4 million TCRs) are still high, limiting the practical applicability of this approach in settings with constrained computational resources.\n- The method relies heavily on co-occurrence patterns, which may not fully account for complex immunological interactions, such as those involving low-frequency, yet clinically relevant, TCRs. Moreover, the assumption that TCRs responding to the same antigen will necessarily co-occur in similar contexts lacks validation and may oversimplify TCR functional diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)