Scalable Universal T-Cell Receptor Embeddings from Adaptive Immune Repertoires

Authors: Paidamoyo Chapfuwa, Ilker Demirel, Lorenzo Pisani, Javier Zazo, Elon Portugaly, H. Jabran Zahid, Julia Greissl
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Immunomics, T-cell Receptor Embeddings, GloVe, Random Projection Theory, Scaling, Unsupervised Representation Learning

Abstract:
T cells are a key component of the adaptive immune system, targeting infections, cancers, and allergens with specificity encoded by their T cell receptors (TCRs), and retaining a memory of their targets. High-throughput TCR repertoire sequencing captures a cross-section of TCRs that encode the immune history of any subject, though the data are heterogeneous, high dimensional, sparse, and mostly unlabeled. 
Sets of TCRs responding to the same antigen, *i.e.*, a protein fragment, co-occur in subjects sharing immune genetics and exposure history. Here, we leverage TCR co-occurrence across a large set of TCR repertoires and employ the GloVe (Pennington et al., 2014)  algorithm to derive low-dimensional, dense vector representations (embeddings) of TCRs. We then aggregate these TCR embeddings to generate subject-level embeddings based on observed *subject-specific* TCR subsets. Further, we leverage random projection theory to improve GloVe's computational efficiency in terms of memory usage and training time. Extensive experimental results show that TCR embeddings targeting the same pathogen have high cosine similarity, and subject-level embeddings encode both immune genetics and pathogenic exposure history.

Decision: accept (poster)
Reviewer scores: 5, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730619359216

thread_type:
official_review

<thread_object>
creator: reviewer_ddms

<content>
summary:
This paper introduces a scalable approach to generating T-cell receptor (TCR) embeddings by leveraging the GloVe algorithm, adapted with the Johnson-Lindenstrauss (JL) transform for improved computational efficiency. The approach aims to create subject-level embeddings of TCR repertoires, which capture immune genetics and pathogenic exposure history. It employs a co-occurrence-based model to detect immune-related patterns and provides an aggregation of TCR embeddings at the subject level, which the authors claim could serve in predicting diseases and HLA types.

strengths:
- The approach is somewhat novel in applying co-occurrence modeling, inspired by NLP, to TCR data. Leveraging random projection (JL transform) to enhance GloVe's performance also demonstrates creativity in handling large datasets.
- The paper is well-organized with clear methodological sections, providing figures and tables to explain the model architecture and performance comparisons.

weaknesses:
- While the method adapts the GloVe and JL transform for TCR analysis, there is limited advancement in the biological interpretability of embeddings over existing approaches.
- The paper lacks rigorous benchmarks against established methods beyond simple logistic regression. Disease and HLA classification tasks do not adequately demonstrate the model’s robustness, especially with limited sensitivity for certain conditions (e.g., HSV) at larger embedding scales.
- While clustering by disease and antigen provides some interpretative insight, the embeddings’ clinical relevance is unclear.
- Despite using the JL transform to improve scalability, the computational requirements for large-scale TCR data (e.g., 4 million TCRs) are still high, limiting the practical applicability of this approach in settings with constrained computational resources.
- The method relies heavily on co-occurrence patterns, which may not fully account for complex immunological interactions, such as those involving low-frequency, yet clinically relevant, TCRs. Moreover, the assumption that TCRs responding to the same antigen will necessarily co-occur in similar contexts lacks validation and may oversimplify TCR functional diversity.

questions:
- How does the model handle rare but potentially significant TCRs? Given the emphasis on co-occurrence, it is unclear how rare TCRs are represented, as these could provide unique insights in immune responses but may not frequently co-occur with other TCRs.
- The current validation relies primarily on logistic regression without exploring other classifiers or model interpretability techniques, which would strengthen the paper’s claims on model generalizability and utility.
- How does the embedding perform across other immunological datasets?
- Additional clarity on computational scaling challenges would be helpful, especially given the potential high-dimensional space of TCR repertoires.
- What additional features could improve biological relevance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687369108

thread_type:
official_review

<thread_object>
creator: reviewer_x8nf

<content>
summary:
This paper develops JL-GLOVE, a method for creating vector representations/embeddings of T-cell receptors (TCRs) and immune repertoires that capture meaningful biological relationships. The method leverages TCR co-occurrence across patient repertoires, adapting the GloVe algorithm from natural language processing while incorporating the Johnson-Lindenstrauss transform for computational efficiency. TCRs are embedded such that those targeting the same pathogen have similar vector representations, and patient repertoires are represented by averaging their constituent TCR embeddings. The resulting embeddings successfully encode both immune genetics (HLA types) and pathogen exposure history, improving as more data is added, and outperform baseline methods on disease prediction tasks. The authors demonstrate their method's scalability and interpretability, showing it can process millions of TCRs while maintaining performance, though they note that the simple averaging approach for patient-level representations could be improved. By creating these biologically meaningful representations, the work provides a foundation for quantifying immune system similarity between individuals and could assist in personalized medicine applications.

strengths:
- Originality: the development of TCR embeddings and immune repertoire representations is original and an under studied area in the representation learning community. The application of Glove algorithm here fits nicely and works. 
- Quality: the produced results are of high quality and provide a significant impact to the field
- Clarity: the paper is very clear to read and understand. The authors give the right amount of biological/immunological background to understand the paper and why it is important. 
- Significance: this is a very significant and meaningful contribution to the field of personalized medicine. The application of representation learning for TCRs and immune repertoires is a great step towards better medicine.

weaknesses:
- The novelties of the paper are not the representation learning method itself. The paper applies Glove algorithm with a few modifications that help it work better, but are not necessarily innovations in of themselves. 
- As stated, the immune repertoire method of taking the average, is nice and works, it could be further developed by other methods.

questions:
- I believe this paper should be in the topic area of applications to physical sciences (biology / immunology) rather than unsupervised, self-supervised, semi-supervised, and supervised representation learning. There are few novelties in terms of methods development, but the application of these methods are extremely impactful and a nice way to show the power of representation learning. 
- You mention other methods for set level representations, why not use those? Average is nice in it's simplicity, but does this simplicity cost performance? Would be nice to see a benchmark against set representation methods. You may also be interested in OTKE method for set level representation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713981957

thread_type:
official_review

<thread_object>
creator: reviewer_ovf4

<content>
summary:
The authors present JL-GLOVE, a scalable algorithm for generating low-dimensional embeddings for T cell receptors (TCRs) and TCR repertoires using TCR co-occurrence data. The main idea is to leverage the co-occurrence patterns of TCRs that target the same antigen to learn meaningful representations. To address the computational challenges of large-scale TCR data, the authors introduce the JL-GLOVE method, which combines GloVe with random projection theory. This approach improves memory efficiency and speeds up the training process. They then aggregate these TCR embeddings to generate subject-level embeddings, providing a low-dimensional representation of an individual's immune history. The embeddings show that TCRs targeting the same antigen exhibit high cosine similarity, and aggregated repertoire embeddings correlate with immune profiles, supporting disease prediction and HLA inference tasks. Results demonstrate the utility of these embeddings for predictive modeling and potential applications in personalized medicine by integrating them with other data modalities.

strengths:
- The use of the JL transform significantly improves the computational efficiency of the GloVe algorithm, enabling the analysis of large datasets containing millions of TCRs. The authors demonstrate that JL-GLOVE achieves good performance using only a fraction of the co-occurrence data, making it suitable for handling the increasing scale of TCR repertoire sequencing data.

- The embeddings produced not only capture the co-occurrence patterns among TCRs but also demonstrate clustering by antigen specificity and HLA association. This biologically meaningful structure aligns with immune response patterns and enhances the interpretability of the embeddings, which is valuable for immunological research and practical applications like personalized medicine.

- The paper rigorously validates the embeddings’ effectiveness through multiple downstream tasks, including disease classification and HLA inference. The experiments demonstrate the robustness of the embeddings to scale, supporting their utility in predicting immune response profiles across various pathogens, and showcasing meaningful performance improvements with larger datasets.

weaknesses:
- The authors compare JL-GLOVE to protein sequence-based embeddings (e.g., ESM-2 and TCRdist), which are structurally different from co-occurrence embeddings. While this comparison is useful, the paper could benefit from a broader comparison with other immunology-focused embedding techniques, such as contrastive learning methods or graph-based embeddings, which may capture additional biological context.

- The paper relies primarily on a mean pooling approach for aggregating TCR embeddings at the repertoire level, which, while straightforward, may be overly simplistic. This method is prone to noise, especially as the number of TCRs (K) increases, potentially limiting classification performance for diseases with more subtle immune signatures.

questions:
- The paper benchmarks JL-GLOVE against ESLG and AIRIVA for disease classification tasks. The authors can include a more comprehensive comparison with other deep learning models specifically designed for TCR repertoire analysis (DeepTCR, DeepID etc).

- The authors observe that the disease classification performance is sensitive to the embedding dimension (d) and the number of TCRs (K). A more systematic exploration of the impact of these parameters can be done A more detailed analysis of the impact of different embedding dimensions across various dataset sizes would be valuable.This would aid other researchers in configuring JL-GLOVE for datasets of different sizes or resolutions, thereby increasing the framework’s accessibility and practical utility.

- Presenting one or two practical case studies where JL-GLOVE embeddings provide actionable insights in a real-world immunological context (e.g., identifying rare disease signatures) would further emphasize the method’s applicability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720198339

thread_type:
official_review

<thread_object>
creator: reviewer_qp7p

<content>
summary:
In this study, the authors developed a method to derive low-dimensional representations of T cell receptors and subject-level repertoires in feature space. To enhance computational efficiency, the method employs random projection theory.

strengths:
The paper is well-organized and clearly written.

The proposed method is technically sound.

The application of random projection theory to enhance computational efficiency, particularly regarding memory usage and training time, is noteworthy.

weaknesses:
The biological definitions presented in the study are somewhat unclear. For instance, when the authors refer to TCR embedding, it is important to specify whether they mean both the TCR alpha and beta full chains, the CDR3 regions of both chains, or only the CDR3 region of the TCR beta chain. Additionally, do the authors take into account V(D)J gene information when using the CDR3 region?

Given that TCRs are highly cross-reactive, the authors need to provide further explanation on why using co-occurrence information alone is effective for TCR embedding.

The repertoires of different subjects contain varying numbers of TCRs. How do the authors address this variability when representing them with a matrix of the same TCR dimensionality?

Considering the high cross-reactivity of TCRs, how do the authors define the TCR-level ground truth without relying on wet-lab-based experiments?

When discussing classification tasks, it would be helpful to clarify whether the focus is on receptor-level classification or repertoire-level classification. Furthermore, given different receptors have clone frequencies within the repertoire, it appears that the authors do not consider clone frequency in their repertoire-level embedding.

The interpretation of deep learning models is crucial for clinical applications; however, the authors have provided limited results in this area.

Lastly, there is a noticeable lack of comprehensive comparisons with state-of-the-art works such as DeepTCR, TCRAI, DeepAIR, and DeepRC, which should be addressed.

questions:
In the weaknesses section, it would be beneficial to provide further illustrations regarding the biological background, the methodology employed, and a detailed explanation of why the model is effective.

Moreover, in addition to reporting AUC and sensitivity, the authors should also include other relevant metrics such as specificity, positive predictive value (PPV), negative predictive value (NPV), and overall accuracy. It is important for the authors to clarify how the cut-off points for these metrics were determined, as this information is crucial for understanding the model's performance and its clinical applicability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731981451852

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Rebuttal by Authors

comment:
We thank the reviewers for their encouraging and insightful comments. We are grateful for the
largely positive reception of our work and would like to highlight a few key points made by the
reviewers:
* Originality: Leveraging random projection (JL transform) to enhance GloVe’s performance
is a creative and noteworthy approach. *(Qp7p, ovF4, x8Nf, DdmS)*
* Clarity: The paper is well-organized, well-written, and easy to understand, with clear
methodological sections. *(Qp7p, x8Nf, DdmS)*
* Soundness: The proposed method is technically sound. Extensive experimental results
demonstrate the robustness of the embeddings at scale and their effectiveness on down-
stream tasks. *(Qp7p, ovF4, x8Nf)*
* Significance: The embeddings capture co-occurrence patterns among TCRs and show clustering by antigen specificity and HLA association, making a significant contribution to personalized medicine. *(ovF4, x8Nf)*

We are encouraged by the positive evaluation scores (8, 6) and are keen to address the concerns
underlying the lower scores (5, 5) to bridge gaps in understanding. Below are the point-by-point
responses and a summary of changes, which are also highlighted in the updated revision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper is at the intersection of Biology and AI. It develops embedding which will be useful for understanding of how Tcells target various infections. There is consensus that this paper should be of interest to ICLR community.