Tabula: A Tabular Self-Supervised Foundation Model for Single-Cell Transcriptomics
Authors: Jiayuan Ding
Jianhui Lin
Shiyu Jiang
Yixin Wang
Ziyang Miao
Zhaoyu Fang
Jiliang Tang
Min Li
Xiaojie Qiu
Venue: NeurIPS 2025 poster ()
Keywords: Foundation Model
Single-cell
tabular learning
federated learning
Primary area: machine_learning_for_sciences

Abstract: Foundation models (FMs) have shown great promise in single-cell genomics, yet current approaches, such as scGPT, Geneformer, and scFoundation, rely on centralized training and language modeling objectives that overlook the tabular nature of single-cell data and raise significant privacy concerns. We present TABULA, a foundation model designed for single-cell transcriptomics, which integrates a novel tabular modeling objective and federated learning framework to enable privacy-preserving pretraining across decentralized datasets. TABULA directly models the cell-by-gene expression matrix through column-wise gene reconstruction and row-wise cell contrastive learning, capturing both gene-level relationships and cell-level heterogeneity without imposing artificial gene sequence order. Extensive experiments demonstrate the effectiveness of TABULA: despite using only half the pretraining data, TABULA achieves state-of-the-art performance across key tasks, including gene imputation, perturbation prediction, cell type annotation, and multi-omics integration. It is important to note that as public single-cell datasets continue to grow, TABULA provides a scalable and privacy-aware foundation that not only validates the feasibility of federated tabular modeling but also establishes a generalizable framework for training future models under similar privacy-preserving settings.

TABULA: A Tabular Self-Supervised Foundation Model for Single-Cell Transcriptomies
Jiayuan Ding" "t Stanford University
Jianhui Lin" Central South University
Shiyu Jiang' University of Southern California
Yixin Wang Stanford University
Ziyang Miao Central South University
Zhaoyu Fang Central South University
Jiliang Tang" Michigan State University
Min Li Central South University
Xiaojie Qiu Stanford University
Abstract
Foundation models (FMs) have shown great promise in single-cell genomics, yet current approaches, such as scGPT, Geneformer, and scFoundation, rely on centralized training and language modeling objectives that overlook the tabu- lar nature of single-cell data and raise significant privacy concerns. We present TABULA, a foundation model designed for single-cell transcriptomics, which in- tegrates novel tabular modeling objective and federated learning framework to enable directly models the cell-by-gene expression matrix through column- wise gene reconstruction and row-wise cell contrastive learning, capturing both gene-level relationships and cell-level heterogeneity without imposing artificial gene sequence order. Extensive experiments demonstrate the effectiveness of TABULA: despite using only half the pretraining data, TABULA achieves sttteoofthh-an performance across key tasks, including gene imputation, perturbation prediction, cell type annotation, and multi-omics integration. It is important note that as public single-cell datasets continue to grow. TABULA provides scalable and privacy- aware foundation that only validates the feasibility federated tabular mod- eling, but also establishes a generalizable framework for training future mod- els under similar privacy-preserving settings. All resources openly available at https: //g thub. wonm/aristoteeeeeeeet tabula to support broad community adoption and future methodological advances.
pretraining across decentralized datasets. TABULA
1 Introduction
Over the past year, we have witnessed the emergence of single-cell foundation models, notably
Geneformer [I] scFoundation (2]. CellPLM 3 ScGPT [4) and NicheFormer [5]. However, current models fail to explicitly account for the tabular structure of the scRNA-seq data, often naively converting gene expression within a single cell to gene sequences to mimic word sequences used in NLP. While this adapts the NLP paradigm single-cell data, it overlooks critical features of the underlying tabular structure of scRNA-seq. Moreover, has been shown previously that scRNA-ses data is susceptible to privacy breaches through linking attacks 66 7 (More details are provided in Appendix D), where information from one study can linked to another identify private data.
Equal contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

This privacy concern becomes even more pronounced as we begin creating foundation models with tens of millions of single cells and datasets from thousands of individuals.
To address these critical gaps, we introduce TABULA, foundation model specifically designed for single-cell data through the lens of tabular learning and federated training. TABULA introduces four key innovations:
First, TABULA formulates single-cell pretraining self-supervised tabular learning problem, directly modeling the cell-by-gene tabular structure. It is important to note that TABULA uses comupteed gene expression inputs instead of introducing artificial mask values such -1 in traditional FMs to learn robust gene-level representations through column-wiss reconstruction, and cell-level representations through row-wise contrastive learning. This tabular modeling strategy captures both gene dependencies and cell-level heterogeneity, without imposing artificial gene order.
Second, TABULA is trained in federated learning setting not only to preserve data privacy across institutions or tissue datasets, but also to enable tissue-specific embeddings by decoupling the training of a shared tabular transformer from client-specific embedders. This design allows TABULA to capture both globally shared biological patterns and unique transcriptional features of each client tissue.
Third, extensive experiments demonstrate the effectiveness of TABULA. TABULA achieves state- of-the-art performance across a wide range of benchmark tasks, including gene imputation, genetic perturbation prediction, cell type annotation, multi-omics integration, and batch correction while requiring only half the data for pretraining.
Fourth, as public single-cell datasets continue grow, the need for scalable and privacy- preeeeviigee training becomes increasingly important. TABULA provides a generalizable framework that combines federated learning with tabular modeling, enabling collaborative model training without data sharing. This not only validates the feasibility of federated tabular modeling but also lays the foundation for settings.
training future models under similar
Design Principles
Despite recent progress in single-cell foundation models 8 4 2, existing approaches face critical limitations that hinder their biological fidelity, scalability, and privacy. These limitations motivate the design of TABULA, which is guided by the following three principles.
Centralized Learning: Genwlomeer
Autoregressive: ScOPT
Federated Learning
Tabular Learning
gI
R+
De
Apglegsted
X
X1
Xi
X
X,
Data
X
X2
Xx
X
Masked Language Modeling: Censeoomee Couratetion
X
X
XL
Xxx
X
(ndia)
Draiy
Brain
Lung
Heal
X..
X
Xui
1000
Gerain aira srnvin)
mett computee
(a) Existing Foundation Models
(b) TABULA
Figure 1: An illustration of the difference between (a) existing foundation models and (b) TABULA for single-cell modeling.
Principle 1: Realistic Corruption Strategies. Existing FMs such as Geneformer and scFoun- dation [2] often use artificial corruption strategies (c.g., masking gene expression values with -1) for training (Figure Dp, right). These unrealistic sentinel values do not appear during inference and result in training-inference inferenceeeeee mismatch that degrades generalization. TABULA avoids this issue by generating corrupted views through resampling from the empirical marginal distribution of gene values (Figure 2). preserving statistical realism and improving downstream biological performance.
Principle 2: Tabular Modeling for Unordered Mixed-Type Type Single-Cel Data. Unlike natural language, where sequence order is meaningful, gene expression profiles lack an inherent ordering and

---

consist of mixed discrete gene token IDs and continuous gene expression values. Most existing models artificially impose gene sequences via attention masking (scGPT 4 or ranking (Geneformer (I) (Figure D right). which distorts the tabular nature of the data. In contrast, TABULA treats the cell-by- gene matrix as a table and learns both gene-level dependencies via column-wise reconstruction and cell-level variation via row-wise contrastive learning (Figure [T. right). This dual-axis self-supervised learning respects the true tabular structure of the data.
Principle 3: Federated Learning for Trivacy-Preservve Collaboration. Centralized training poses major privacy risks in genomics, where individual-level data is sensitive and protected. Prior work has shown that scRNA-seq data vulnerable to linkage attacks [6] 71. To mitigate these risks, TABULA is trained using a federated learning setup (Figure D left). Each client (e.g., tissue or institution) retains its data locally and contributes to a shared global model by only exchanging weights, enabling collaborative training across heterogeneous datasets without data sharing.
These three design principles, realistic corruption, tabular structure awareness, and federated privacy, form the foundation of TABULA and are reflected in the framework described in Section 3
3 The Proposed TABULA Framework
In this section, we introduce the proposed TABULA framework and each component in TABULA. As illustrated in Figure 2 the tabular modeling in TABULA requires corrupted gene expression inputs, which are detailed in Section 3.1 The process by which the tabular embedder transforms the cell-by-gene matrix into embeddings for the transformer, along with the modeling of cell-by-gene structure from a tabular learning perspective, is detailed in Section 3.2 At a high level, TABULA consists of two stages: pre-training and fine-tuning, which are demonstrated in Section 3.3
3.1 Corrupted View Generation for Tabular Modeling
To handle the intrinsic tabular structure of scRNA-seq dataset, TABULA introduces innovative self-supervised tabular modeling framework that consists of both gene-wise reconstruction learning and cell-wise contrastive learning. Both training components start with generating a corrupted representation of the scRNA-seq data. We followed Xtab [9] to construct corrupted views through random feature resampling. As shown in Figure we randomly selected subset of genes in cells in training batch and then resampled their values from the empirical marginal distribution [T0] of these genes within the same training batch. We set the corrupted ratio at 60%. This means that for the original view of each cell sample e and its corrupted view 2, 60% of the gene values are resampled while 40% remain unchanged. The original and corrupted views of the same cell are treated positive pair in contrastive learning, as they originate from the same underlying biological state but differ through controlled noise.
3.2 TABULA Model Architecture
TABULA innovatively treats single-cell data, represented as a cell-by-gene matrix, tabular data, and models its intrinsic tabular structure through a dual-axis self-supervised learning framework. Additionally, TABULA addresses cross- tooss-tsssee variation by partitioning the model into distinct clients in federated setting, each tissue client equipped with tissue-specific embedders that capture tissue- specific variants, and shared tabular transformer (global or local) that encodes global knowledge across tissues. In this section, we describe how TABULA encodes the raw cell-by-gens matrix into transformer-compatible embeddings in Tabular Embedder, and how models the matrix from tabular learning perspective in Tabular Modeling.
Tabular Embedder scRNA-seq data can be represented as a cell-by-gene matrix, X E N.M. where each entry X E R++ denotes the read count of RNA molecules for gene j E {0, I,... ., M in cell i E {0, 1,..., N). Each row of the cell-by-gene table, X, is considered as input cell sample in a tabular transformer, and each column a gene feature token. The function of the embedder module to convert each cell sample to feature embeddings E E (d Here, M denotes the number of columns, +1 indicates the special [CI.S] token (see below), and d is the embedding dimension. TABuLLa treats each row of the cell-by-gene matrix as an unordered set of genes, where each gene is represented by column feature name embedding (gene token) and feature value embedding
3

---

Corrupted View Generation
Tabular Modeling
Margin dietriuution 60% sedected
senvs corappec within
Cotume reconsrurtioon
2
Row contassive
154
pos
neg
19
g+
g+
Commpied
Gane-aten learning
Pretraining
Gono wise
Leanning
Gene wiso Galcatae
Celieein oe
Tixauw Speeiiic
Fine-tuning
Project i leads
Project Heads
Project Heads
Downstream Taskk
Aggregation
XP%OPR
pertutution
Global Tabuler Transformer
Local Tranfoomer
Twensformr
Tia formar
intinncc
Lung
Updato
Gene Emtwodiigg
emoodd mg
Brain
Embedder
Embeddar
Embedder
Genes
Calle
Distributed
Client (Brain)
Client (Lung)
Client (Heart)
comrectoo
sources
Figure 2: Overview of the TABULA framework. TABULA learns both gene- and cell-level repre- sentations from tabular single-cell data using dual-axis self-supervised tabular learning objectives: column-wise gene reconstruction and row-wise cell contrastive learning. It trained in federated setting with a shared global transformer and client-specific embedders, enabling privac--rresevving and tissue-aware representation learning.
(expression level). While the current implementation models only gene tokens, the framework can naturally incorporate metadata (c.g., cell type, tissue, or platform) in future extensions.
Column feature name embedding. The feature column the cell-by-gene table for each client corresponds individual genes. Our pretraining dataset includes 196 distinct studies, from which we select 1,200 HVGs per study, yielding a total of 23, HVGs across all datasets. Consequently, we have 23,156 unique feature columns learn during the pretraining phase. We assign each column feature name, feature,, a unique integer identifier, id sd(featurey) These identifiers collectively form the column feature Additionally, a special token, [CLS], is used to aggregate all column features into a cell-level representation. The input column feature tokens for cell are therefore represented p(i)
vocabulary utilized in TABULA.
vector
E R.M, where M is a predetermined maximum sequence length in the tabular transformer:
token
(1) tnken id (feature(() id (Seature
id (reature )
=
The embedding representation of input column feature tokens for cell i is denoted E I M xd where d represents the embedding dimension:
ttkken
Loken
embroken
Loken
where emb represents embedding layer to learn feature token embeddings.
Column feature value embedding. Expression values in scRNA-seq data vary due to differ- ences sequencing protocols, sensitivity, and depth, leading to inconsistencies across datasets. To tackle this issue, we adopt scGPT's value binning technique 4 to convert expression counts into relative values. Unlike scGPT, which only focuses on non-zero expression counts each cell, select 1,200 HVGs for modeling in each dataset, including those with zero expression.

---

By incorporating zero-expressed genes, TABULA provides more complete view of gene ex- pression variability, capturing both active and inactive genes. This broader perspective allows for the detection of important biological patterns that could be overlooked when considering only non-zero genes, as zero expression can have biological significance [[]]. For each HVG
expression count in a cell, we compute the raw absolute values and partition them into B consecutive intervals [Bin,, Bin++1), where t E (1, B). The binned expression value (6) for cell i is designated as: "j
= --(0
if X > 0 and X E [Bing, Bin4+1) (0, if X =0
(3)
The column feature values for cell are represented as vector (al [1
M
RM, where M is the predefined maximum sequence length. These values are then embedded via E val emb where E E Mxd and emb is a learnable embedding layer.
Consequently, the embedding E E R Mxd for cell i is represented as:
E (() token
EE())
Finally, the embedding of the [CLS] token is appended to the feature embedding, resulting in a final feature embedding E E R(M++)xxd The final hidden state corresponding to the [CLS] token is used as row (cell) representation, aggregating from column feature embeddings. It is important to note that each client's embedder module is distinct and specialized for learning tissue-specific features.
Tabular Modeling. To effectively model the cell-by-gene table, we design two specialized pretraining losses for tabular modeling: reconstruction loss and contrastive learning to learn gene column features and row cell features, respectively shown in Figure To calculate the reconstruction and contrastive losses, corrupted view must be generated for each cell, which has been discussed in Section [3. To note that the original view and corrupted view ē of the same cell are treated as a positive pair in contrastive learning, as they originate from the same underlying biological state but differ through controlled noise.
Reconstruction loss gene-wise column learning. Reconstruction loss is self sulf-supe training objective. important to note that it aims to recover the original gene-wise view from corrupted view of that gene across all cells in the batch. Taking gene j as example, we take the corrupted view, čÿ E R (N indicates the number of cells in a batch.), for gene j as input and aim to reconstruct its original binned gene expression g) E R. where gj = ( 0 ,....,c [N The reconstructed view for gene j can be represented as g E R. In this study, view and the reconstructed view. The reconstruction loss is denoted as:
Mean Squared Error (MSE) measure the distance between the original
M
N
1
( 2
Lec = M
Σ ==
(5)
The reconstruction loss Liee is applied to all M gene values, enabling the model to learn gene- specific distributions and inter-gene relationships for richer feature representations. Unlike prior models [ [ 4 that use an artificial mask value (e.g., -1), we replace corrupted entries by resampling from the empirical gene distribution, aligning training conditions with inference for improved consistency.
Contrastive loss as cell-wise row learning. Similar to the reconstruction objective, we generate a(i) as a corrupted view of cell c( The original and corrupted views of the same cell form a positive pair, while all other cells (cli) and e(i) within the batch are treated as negative pairs. Contrastive loss encourages the model to minimize the distance between positive pairs and maximize between negatives. Specifically, as shown Figure the gene expression profiles of each cell, including the original view and corrupted view ć₁, are embedded into latent representations (e), 61). ). The loss enforces that positive pairs (e.g., e and é1) ) are close in latent space, while negative pairs (e.g., and ē,)) are far apart.
5

---

In this study, we employ the SimCLR [12
loss for contrastive pretraining, which is denoted as:
exp (sim(e, 0:)/T)
Coontrast
log
2.V
N
Σ
[izi (exp (sim(e, 65)/T) + exp (sim(e;,ey)/r)) ez)/T))
1=1
exp (sim((e.
+ log
(6)
N [s (exp simm + exp (sim éj)/T))
Σ
where e(*) and @(1) represent the cell embedding of cell i from the original view and corrupted view respectively after being processed through the transformer block. L[tezes]] E {0, 1 is an indicator function that takes the value of I if k + i; otherwise, it is O. The similarity function sim(-, :) represents cosine similarity here to measure the similarity between two cell embeddings, denotes the temperature parameter, and N is the number of cells in batch.
The final tabular modeling objective in TABULA is a combination of the two loss functions:
Lub a<rse + Ceontrast
where the scaling factor a used balance the two loss terms to ensure comparable magnitudes. In this study, it is set to 0.03. The combined loss provides holistic approach to learning both gene-level (column-wise) and cell-level (row-wise) features from the cell-by-gene matrix. While gene-wise reconstruction promotes robust gene-level representation learning, cell-wise contrastive learning enforces both intra-cell consistency and inter-cell variability. Together, these objectives enable TABULA to effectively model the tabular structure of single-cell data across both the gene and cell axes.
3.3 TABULA Pre-training & Fine-tuning
Pre-training. To protect data privacy in single-cell data, TABULA is pretrained using a federated learning framework shown in Figure 2 Each client, representing a hospital, research institute, or tissue type (e.g., lung, brain, heart), trains locally without sharing raw data. In this study, TABULA is pretrained on 15 million scRNA-seq profiles across eight clients: Intestine, Panereas, Lung, Heart, Blood, Kidney, Brain, and Others (see Appendix A for pretraining data collection and preprocessing). Each client includes tissue-specific embedder that encodes the cell-by-gen matrix into embeddings. These embeddings are passed through tabular transformer to compute gene-wise and cell-wise self-supervised losses (shown in Equation 7 Training proceeds in alternating rounds of local and global updates: local transformer weights from client-side are uploaded to a central server, aggregated into a global model, and then broadcast back to clients (see Appendix B for the federated learning framework). This iterative communication enables knowledge sharing across heterogeneous datasets while preserving data confidentiality through strict decentralization. Note that this cross-tissue client setup demonstrates federated training of TABULA, the framework is generally applicable to cross-institutional or other settings, enabling collaborative pretraining without sharing raw data and thereby addressing privacy concerns. More pertaining details can be found in Appendix C
Task-specific Fine-tuning, TABULA can be fine-tuned for a variety of downstream tasks, including gene perturbation prediction, cell type annotation, multi-omics integration, batch correction, and gene expression imputation shown in Figure During fine-tuning, first loads the tissue-specific pretrained embedder, followed by the pretrained global tabular transformer. A task-specific head or decoder then attached based on the nature of the task, enabling the model to adapt effectively through task- task-specific optimization.

---

Experiments
4.1 Preliminary Study: Federated vs. Centralized and Tabular vs. MLM
To demonstrate the advantage of TABULA 's tabular learning and its adaptability to federated learning. we compare four pretraining strategies: centralized vs. federated learning, each paired with either tabular modeling or masked language modeling (MLM) (Figure 3 (left). The pretraining dataset includes 1M cells from CELLxGENE (250K per tissue: pancreas, blood, brain, lung). Centralized training uses pooled data; federated training distributes data across four tissue clients with model weights aggregation. MLM uses a 15% masking ratio. We assess performance on two downstream tasks: cell type annotation (hPancreas dataset [13 and genetic perturbation prediction (Adamson [14] and Norman [5] datasets), measuring accuracy and Pearson correlation coefficients. As shown in Figure 3 the results show that tabular modeling consistently outperforms MLM in both training schemes. Notably, federated-tabee pretraining achieved the highest overall performance, matching or exceeding centralized models despite data heterogeneity. More details of the implementation can
be found in the ADpendix C
annotation (hPancreas dataset)
Genetic perturbation prodiction datesel)
Unseen single gene perturbation detasel)
Centratzed Faeeesied
6:14
Campalad Foterolid
Cantmtaee Federated
Lang
Tabular Learning
0.94
0.88
0.73
0.71
0.46
0.35
Contaltaed
0.85
0.78
0.71
0.69
0.46
0.31
(X0A))
1200
(2500)
Tabular
MLM
Tabular
Figure Benchmarking federated vs. centralized pretraining and tabular learning vs. masked language modeling (MLM) on I million cells.
4.2 Pretraining Data Scaling Laws in TABULA
To investigate how pretraining dataset size influences downstream performance, identify clear scaling law exhibited by TABULA. As depicted in Figure 4 increasing the number of pretraining cells from 300,000 to 3 million and then 15 million progressively enhances cell type annotation accuracy in the hPancreas dataset [3]. UMAP visualizations (Figure[a, reveal more coherent clustering of annotated cell populations as data scale increases, especially notable in the alpha and beta cell clusters (highlighted in red). Quantitative analysis (Figure F further confirms consistent improvements across accuracy, precision, recall, and maero F1 score, highlighting that larger-scale pretraining substantially strengthens the generalizability and robustness of learned representations. Importantly, these results also demonstrate that the benefits of scaling laws hold even under federated learning framework, underscoring TABULA is ability to leverage large-scale, decentralized datasets while preserving data privacy.
Annotation
300K
Scaling performance of annotation improves as the pretraining dasaset size increases
0.5
Acourary
Pression
TRC
0.4
Macrc F1
302c
Number pretesining cells
Figure 4: Scaling law in TABULA. (a) Pancreas Speecii--specffic cell type annotation performance the hPancreas dataset [3], visualized via UMAP. (b) Scaling law showing improved cell type annotation performance (accuracy, precision, recall, and macro Fl) the pretraining dataset increases from 300K to 3M and 15M cells.
4.3 Evaluation on Gene-Level Downstream Tasks
Task I: Gene Imputation We first summarize key differences among TABULA and sfate-of-the-art foundation models from training paradigm, modeling objective, and pretraining data scale as shown
7

---

in Figure Sh. Unlike existing centralized models that rely on language modeling objectives (AR or MLM), TABULA adopts a federated training mechanism and explicitly models the tabular structure of single-cell data with only half the pertaining data. We evaluate gene-level imputation performance by masking non-zero expression values in scRNA-seq data and predicting them using fine-tuned decoder. TABULA is benchmarked against scGPT, scBERT, and Geneformer across four datasets (PBMC5K, Jurkat, Melanoma [16], hPancreas ). As shown in Figure 5p. TABULA consistently achieves lower mean squared error (MSE) at both the cell and gene levels, with statistically significant
improvements (Wilcoxon P < 0.05, marked by *). See Appendix for additional results and implementation details.
b
(cell level)
MSE (gene level)
Modeling
2.5
1.25
scGPT
33M
2.0
Genoforer
1.00
Geneformer
Centrolized
MLM
30M
1.5
SCBERT
0.75
adoondaatio
Centralized
MLM
30M
1.0
Tabula
Federated
Tabular
15M
0.25
PBMCSK
Junket
Melanoma
hPancrees
Figure 5: TABULA improves imputation fidelity and accuracy. (a) Differences between TABULA and state-of-the-arr foundation models. (b) MSE comparisons across four datasets at cell and gene levels; TABULA consistently achieves lower errors (Wilcoxon P < 0.05, marked by *).
Task 2: Genetic Perturbation Prediction We evaluate genetic perturbation prediction on three benchmark datasets: Adamson [14), Norman [13], and Replogle (17), covering both single- and double-gene perturbations. Performance measured using Pearsonaelu between predicted and observed gene expression changes, on all genes and the top 20 differentially expressed (DE) genes per condition. We compare TABULA against both specciccc baselines, GEARS [18, CPA [19], and linear regression (LR), well as foundation models including scGPT 44. Geneformer [1] and scFoundation [2]. As shown in Table [ on Adamson, TABULA achieves the highest score on all genes (0.695) and performs comparably the best DE score (0.778) and remains competitive overall. On Replogle, TABULA outperforms scGPT greatly by +0.069 (all genes) and +0.012 (DE genes). To assess generalization, we test unseen Norman perturbations. TABULA outperforms scGPT Pearsonaata for both single- and double-gene settings, especially on DE genes (Figure left).
DE genes (0.787 vs. 0.789 for scGPT), On Norman, it achieves
We further assess reverse perturbation prediction following the setup of scGPT 4 where the task to recover the causal gene perturbation(s) that led to observed expression profile. We evaluate 20-gene benchmark comprising 210 possible single- and double-gene combinations, reporting top-K hit rates. As shown in Figure (middle), TABULA achieves 2/2 hit rate of 48.57% and a 1/2 hit rate of 94.29% at top-1, substantially outperforming ScGPT. The heatmap in Figure (right) visualizes the perturbation pairs correctly identified in the top-1 (red) and top-3 (blue), demonstrating TABULA 's effectiveness in pinpointing causal gene combinations. These results highlight TABULA strong performance in both predictive and reverse perturbation tasks, especially under distribution shifts and combinatorial settings. See Appendix E for additional results and implementation details.
Table 1: (Task 2) The results of genetic perturbation prediction across Adamson, Norman, and Replogle datasets, evaluated by Pearsonasha on all genes and differentially expressed (DE) genes. OOT indicates out-of-time
Adamson
Norman
Replogle
Model
All Genes
Genes
All Genes
DE Genes
All Genes
DE Genes
GEARS
0.531
0.678
0.547
0.715
0.154
0.277
CPA
0.145
0.309
LR
0.532
0.697
0.141
0.298
Geneformer
0.199
0.666
0.316
0.326
0.091
0.362
seFoundation
OOT
OOT
O0T
O0T
OCT
OOT
scGPT
0.615
0.789
0.583
0.742
0.242
0.464
TARULA
0.695
0.787
0.577
0.778
0,311
0.476

---

Unseen gane perturbation prediction
Double gene perturbation reverse prediction (Norman dataset)
Reverse prediction visualtzation
(Norman datawed)
20-go 20-gane perturbation space
Single gane perturbation
Double gene perlutbaton
Idoaa DCGFT
0.4
S:OPT
Accurataly predicted Tap Accunallly predicted Top
0.6-
0.0
0.50
0.6
Tabula 2/2
0.40
Tobulo 1/2 Rate
0.35
0.
sutait 2/2 rit Rate
SCGPT 1/2 Hit Rate
All ganes DE genes
All ganes DE genes
10 Number of top predicted penuurations
6
Figure 6: TABULA outperforms scGPT on unseen gene perturbation and reverse perturbation. Left: Pearson correlations for unseen single and double perturbation prediction on all and DE genes. Middle: Top-K hit rates for reverse prediction of double perturbations (1/2 and 2/2 correct). Right: Heatmap of 20-gene pairwise space with top-1 (red) and top-3 (blue) accurate reverse predictions.
4.4 Evaluation on Cell-Level Downstream Tasks
Task 3: Cell Type Annotation Cell type annotation assigns biological labels to cells based on gene expression profiles. To evaluate TABULA, we fine-tune it with a supervised cross-entropy loss and test on three datasets: hPancreas [33, Myeloid [4, and Cell Lines [20]. We compare against scGPT, Geneformer, and scFoundation, each fine-tuned with their default settings. Performance measured using accuracy, precision, recall, and FI score. TABULA achieves the best overall performance on hPancreas and Cell Lines, and while slightly behind scFoundation in Myeloid accuracy, it outperforms others in remaining metrics- demiiisstdmmmstttttin strong generalization (Table 2). Notably, TABULA achieves this using only half the pretraining data while ensuring privacy protection through federated training. See Appendix [B] for additional results and implementation details. We also report Task 4&5 Multi-Omics and Multi-Bater Batch Integration results with implementation details in Appendix
Table 2: (Task 3) The results of cell type annotation across hPancreas, Myeloid, and Cell Lines datasets, evaluation by test accuracy, precision, recall, and Fl score.
Model
haaaneress
Mycloid
Cell Lines
Accuracy
Precision
Recall
Accuracy
Precision
Accuracy
Precision
geneFormer
0.9395
0.6534
0.6491
0.6425
0.5953
0.3704
0.3300
0.3441
0.9911
0.9918
0.9904
0.9910
scFoundation
0,9492
0.6140
0.6032
0.6003
0.6352
0,4070
0,3764
0,3804
0.9931
0,9932
0.9930
0.9931
sGGTT
0.9680
0.7250
0.6420
0.3460
0.9930
TABULA
6.9810
0.7814
0.7795
0.7708
0.6213
0.3909
0.3634
0,3728
0.9935
0.9939
0.9931
0.9935
In addition to these main results, we provide additional analyses in the appendix to further demonstrate the efficacy of TABULA. Specifically, we demonstrate that tissue-specific embedders can effectively capture biologically meaningful variation across different tissue types, as detailed in Appendix Furthermore, ablation studies presented in Appendix Conffrmm the importance and effectiveness of our proposed tabular modeling objectives, highlighting their critical role in enhancing downstream performance.
5
Related Work
Recent advances in single-cell foundation models (FMs) have adapted techniques from natural lan- guage processing to learn transferable representations from large-scale transcriptomic data. Notable models include Geneformer (.. scFoundation (2), scGPT [4], and CellPLM [3. These models predominantly employ centralized training and language modeling objectives such as autoregressive (AR) 21 or masked language modeling (MLM) [2]. which ignore the intrinsic tabular structure of single-cell data. Moreover, those models reply on centralized training thus raising concerns about data privacy and ethics from public single-cell data. Prior work has shown that scRNA-seg data is vulnerable to linkage attacks 7].
To address these limitations, we propose TABULA, a foundation model that captures the tabular structure of single-cell data through tabular modeling while preserving privacy through federated learning. TABULA employs a dual-axis pretraining objective, gese reconstruction and cell-wise

---

contrastive learning, to model the cell-by-gent matrix without imposing artificial gene order. By decoupling global updates from client-specific embedders, supports across institutions. Extensive experiments demonstrate the effectiveness of TABULAA
learning
6 Conclusion
In this work, we present TABULA, the first ariacyypreserving foundation model for single-cell transcriptomies that explicitly models the cell-by-genc matrix using novel tabular pretraining objective. To our knowledge, TABULA is the first to propose a tabular modeling framework tailored for single-cell data, moving beyond sequence- requence-based approaches that impose artificial gene sequence order. It achieves strong performance across diverse benchmark tasks using only half the pretraining data and captures complex biological signals across gene- and cell-level representations, It is important to note that as public single-cell datasets continue to grow, TABULA offers a scalable and privacy-aware foundation that not only validates the feasibility of federated tabular modeling, but also establishes a generalizable framework for training future models under similar privacy-preserving settings.
10

---

References
[1] Christina Theodoris, Ling Xiao, Anant Chopra, Mark Chaffin, Zeina R Alyed, Matthew Hill, Helene Mantineo. Elizabeth M Brydon, Zexian Zeng, X Shirley Liu, et al. Transfer learning enables predictions in network biology. Nature, 618(7965):616- 624, 2023.
[2] Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Xuegong Zhang, and Le Song. Large-scale foundation model on single-cell transcriptomics. Nature methods,
2024.
[3] Hongzhi Wen, Wenzhuo Tang, Xinnan Dai, Jiayuan Ding, Wei Jin, Yuying Xie, and Jiliang Tang. CellPLM: Pre-training of cell language model beyond single cells. In The Twelfth International Conference on Learning Representations, 2024.
[4] Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. segpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature Methods, 21 1870)147701148) 2024.
[5] Anna C Schaar, Alejandro Tejada- Giovanni Palla, Robert Gutgesell, Lennard Halle, Mariia Minaeva, Larsen Vornholz, Leander Dony, Francesca Drummer, Mojtaba Bahrami, et al. Nicheformer: a foundation model for single-cell and spatial omics. bioRxiv, pages 2024-04 04, 2024.
[6] Conor R Walker, Xiaoting Li, Manav Chakravarthy, William Lounsbery-Scaife Yoolim A Choi, Ritambhara Singh, and Gamze Gürsoy, Private information leakage from single-cell count matrices. Cell, 185723))65776644, 2024.
[7] D' Nae Ferguson. A privacy concern: Bioinformatics and storing biodata. In The ADMI 2021 Symposimm, 2021.
[8] Fan Yang. Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert large-scale pretrained deep language model for cell type annotation of single-cell ma-seq data. Nature Machine Intelligence, 4(10):852-866 2022.
[9] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint 2rXhv:2305.06090 2023.
[10] Vinit Kumar Mishra, Karthik Natarajan, Dhanesh Padmanabhan, Chung-Piaw Teo, and Xiaobo
Li. On theoretical and empirical aspects of marginal distribution choice models. Management Science, (16)::511-15311 2014.
[11] Ruochen Jiang, Tianyi Sun, Dongyuan Song, and Jingyi Jessica Li. Statistics or biology: the inro-inflation controversy about scrna-seq data. Genome biology, 23(1):31, 2022.
[12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PmLR, 2020.
[13] Jiawei Chen, Hao Xu, Wanyu Tao, Zhaoxiong Chen, Yuxuan Zhao, and Jing-Dong J Han. Transformer for one stop interpretable cell type annotation. Nature Communications, 14(1):223, 2023.
[14] Britt Adamson, Thomas M Norman, Marco Jost, Min Y Cho, James K Nuñez, Yuwen Chen, Jacqueline E Villalta, Luke A Gilbert, Max A Horlbeck, Marco Y Hein, et al. A multiplexed single-cell crispr screening platform enables systematic dissection of the unfolded protein response, Cell, 167(7):167-18822 2016.
[15] Thomas M Norman, Max A Horlbeck, Joseph M Replogle, Alex Y Ge. Albert Xu, Marco Jost, Luke A Gilbert, and Jonathan Weissman. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science, 365(6455):786 2019.
[16] Eduardo Torre, Hannah Dueck, Sydney Shaffer, Janko Gospocic, Rohit Gupte, Roberto Bonasio, Junhyong Kim, John Murray, and Arjun Raj. Rare cell detection by single-cell mna sequencing as guided by minle-molecule ma fish. Cell systems, 6(2):17 2018.
11

---

[17] Joseph M Replogle, Reuben A Saunders, Angela N Pogson, Jeffrey A Hussmann, Alexander Lenail, Alina Guna, Lauren Mascibroda, Eric J Wagner, Karen Adelman, Gila Lithwick- Yanai, et al. Mapping information-rich genotype-phenotype landscapes with genome-scale perturb-seq. Cell, 185(14):2559-255 2575, 2022.
[18] Yusuf Roohani, Kexin Huang, and Jure Leskovec. Predicting transcriptional outcomes of novel multigene perturbations with gears. Nature Biotechnology, 42(6):927-9335 2024.
[19] Mohammad Lotfollahi, Anna Klimovskaia Susmelj, Carlo De Donno, Leon Hetzel, Yuge Ji, Ignacio L Ibarra, Sanjay R Srivatsan, Mohsen Naghipourfar, Riza M Daza, Beth Martin, et al. Predicting cellular responses to complex perturbations high-throughput screens. Molecular Systems Biology, page e 11177, 2023.
[20] Hoa Thi Nhu Tran, Kok Siong Ang, Marion Chevrier, Xiaomeng Zhang, Nicole Yee Shin Lee, Michelle Goh, and Jinmiao Chen. A benchmark of batch-effect correction methods for single-cell rna sequencing data. Genome biology, 21:1-32, 2020.
[21] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[22] Jacob Devlin, Ming-Wo Weig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume I (long and short papers), pages 4171 4171-4186, 2019.
[23] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efine eefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[24] Tri Dao. Flashattention--2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08669/ 2023.
[25] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[26] Harm-Jan Westra and Lude Franke. From genome to function by studying equls. Biochimica et Biophysica Acta -molecuul basis of Disease, 1842(10):196-1192 2014.
[27] Cédric Arisdakessian, Olivier Poirion, Breck Yunits, Xun Zhu, and Lana X Garmire. Deepim- pute: accurate, fast, and scalable deep neural network method to impute single-cell ma-seq data. Genome biology. 20: 1-14, 2019.
[28] Malte D Luecken, Maren Büttner, Kridsadakom Chaichoompu, Anna Danese, Marta Interlandi, Michaela F Müller, Daniel C Strobl. Luke Zappia, Martin Dugas, Maria Jolomé-Tatche et al. Benchmarking atlas-level data integration in single-cell genomics. Nature methods, 19(1):41-50, 2022.
[29] Maayan Baron, Adrian Veres, Samuel Wolock, Aubrey Faust, Renaud Gaujoux, Amedeo Vetere, Jennifer Hyoje Ryu, Bridget K Wagner, Shai S Shen-Orr, Allon M Klein, al. A single-cell transcriptomic map of the human and mouse pancreas reveals inter-and intra-cell population structure. Cell systems, 3(4):346-360 2016.
[30] Sonya A MacParland, Jeff C Liu, Xue-Zhong Ma, Brendan T Innes, Agata Bartezak, Blair K Gage, Justin Manuel, Nicholas Khuu, Juan Echeverri, Ivan I Linares, et al. Single cell ma sequencing of human liver reveals distinct intrahepatic macrophage populations. Nature communications, 9():3333, 2018.
[31] Nadim Aizarani, Antonio Saviano, null Sagar, Laurent Mailly, Sarah Durand, Josip S Herman, Patrick Pessaux, Thomas F Baumert, and Dominic Grün. A human liver cell atlas reveals heterogeneity and epithelial progenitors. Nature, 19977768):1992201 2019.
[32] Zhi-Jie Cao and Ge Gao. Multi-omies single-cell data integration and regulatory inference with graph-linked embedding, Nature Biotechnology,
2022.
12

---

[33] Malte D Luecken. Daniel Bernard Burkhardt, Robrecht Cannoodt, Christopher Lance, Aditi Agrawal, Hananeh Alice, Ann Chen, Louise Deconinck, Angela M Detweiler, Alejandro Granados, et al. A sandbox for prediction and integration of dna, ma, and proteins in single cells. In Thirty-fitth conference on neural information processing systems datasets and benchmarks track (Round 2), 2021.
[34] Kimberly Siletti, Rebecca Hodge, Alejandro Mossi Albiach, Ka Wai Lee, Song-Lin Ding, Lijuan Hu, Peter Lönnerberg, Trygve Bakken, Tamara Casper, Michael Clark, et al. Transcriptomic diversity of cell types across the adult human brain. Science, 382(6667))exdd77046, 2023.
[35] Alexandra-Chloé Villani, Rahul Satija, Gary Reynolds, Siranush Sarkizova, Karthik Shekhar, James Fletcher, Morgane Griesbeck, Andrew Butler, Shiwei Zheng. Suzan Lazo, et al. Single- cell ma-seq reveals new types of human blood dendritic cells, monocytes, and progenitors. Science, 356(6335)ccabb457. 2017.
[36] Xiaojie Qiu, Yan Zhang, Jorge D Martin-Rufino Chen Weng, Shayan Hosseinzadeh, Dian Yang, Angela Pogson, Marco Y Hein, Kyung Hoi Joseph Min, Wang, al. Mapping transcriptomic vector fields of single cells. Cell, 185(4))690-711, 2022.
[37] Jan Krumsiek, Carsten Marr, Timm Schroeder, and Fabian J Theis. Hierarchical differentiation of myeloid progenitors is encoded in the transcription factor network. PloS one, 6(8): 2011.
[38] T Yvanaa de Soysa, Sanjeev S Ranade, Satoshi Okawa, Srikanth Ravichandran, Yu Huang, Hazel T Salunga, Amelia Schricker, Antonio Del Sol, Casey A Gifford, and Deepak Srivastava. Single-cell analysis of cardiogenesis reveals basis for organ-level developmental defects. Nature, 572(7767):120-124 2019.
[39] Gioele La Manno, Kimberly Siletti, Alessandro Furlan, Daniel Gyllborg, Elin Vinsland, Alejan- dro Mossi Albiach, Christoffer Mattsson Langseth, Irina Khven, Alex R Lederer, Lisa M Dratva, et al. Molecular architecture of the developing mouse brain. Nature, 596(7870):92-8) 2-09999999699 2021.
[40] Junqiang Wang, Ruoshi Yuan, Xiaomei Zhu, and Ping Ao. Adaptive landscape shaped by core endogenous network coordinates complex early progenitor fate commitments in embryonic pancreas, Scientific reports, 10(1):1112, 2020.
[41] Sophia K Longo, Margaret G Guo, Andrew I. Ji, and Paul A Khavari. Integrating single-cell and spatial transcriptomics to elucidate intercellular tissue dynamics. Nature Reviews Genetics, 22 22(10):627-644, 2021.
[42] Jun Ding, Nadav Sharon, and Ziv Bar-Joseph. Temporal modelling using single-cell transcrip- tomics. Nature Reviews Genetics, 23(6):355-368 2022.
[43] Shilin Qiu, Qihe Liu, Shijie Zhou, and Chunjiang Wu. Review of artificial intelligence adversar- ial attack and defense technologies. Applied Sciences, 9(5)9909, 2019.
[44] Chaoning Zhang, Philipp Benz, Chenguo Lin, Adil Karjauv, Jing Wu, and In So Kweon. A survey universal adversarial attack. arXiv preprint arXiv:2103.01498, 2021.
[45] Yansong Gao, Bao Gia Doan, Zhi Zhang, Siqi Ma, Jiliang Zhang, Anmin Fu, Surya Nepal, and Hyoungshick Kim. Backdoor attacks and countermeasures on deep learning: A comprehensive review. arXiv preprint arXiv:2007.10766 2020.
[46] Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pages 1-12. Springer, 2006.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions of the paper, introducing TABULA, a federated foundation model for single-cell data that incorporates tabular modeling and achieves strong performance across benchmark tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are acknowledged in the Appendix I, Limitations TABULA.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification: This paper does not contain formal theoretical results or proofs. It is an empirical paper focused model development and biological evaluation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed descriptions of data, model architecture, and training setup are provided in Sections 3 and 4, with additional implementation details and evaluation protocols in Appendix A, C, and E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
The paper will provide open- code along with an Appendix. More data and implementation details can be found in Appendix A, C, and E.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https::/niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wwtpcc/pcccccpcccccooooooop httai//gggudeee/eeeSuubbbiiii wwweiisssonPoPiiiio
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are provided in Appendix A, C, and E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical significance is reported in Figure 4b with Wilcoxon P-values < 0.05 marked by for gene imputation. This demonstrates proper statistical testing for core performance claims.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C includes details on computer resources for TABULA pretraining. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification:
The work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are addressed throughout the introduction and discussion. TABULA reduces privacy risks in genomics, enabling collaborative model training across institutions without data sharing. Separate section talking about broader impact of TABULA in Appendix L.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Although the model is not released yet, the design of TABULA includes safeguards such as federated learning and local embedders to mitigate misuse and protect sensitive genomic data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external datasets and models (e.g., scGPT, CELLxGENE) are properly credited in the references and their terms of use respected. The paper relies on public datasets available under research licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thoode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new foundation model TABULA for single-cell has been trained and released along with its public training dataset after our in house processing.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The study does not involve human subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM just used for writing grammar checking.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20


Reviews:
[{"review_id": "Yd17HcXH9t", "reviewer": "Reviewer_bEnN", "summary": "The paper presents TABULA, a novel foundation model specifically tailored for single-cell transcriptomics. Unlike existing models like scGPT or Geneformer that adapt NLP paradigms by treating gene expressions as token sequences, TABULA directly models the tabular structure of cell-by-gene expression matrices. It introduces a dual-axis self-supervised learning framework: gene-wise reconstruction (columns) and cell-wise contrastive learning (rows), enabling robust representation learning without imposing artificial gene orders. To ensure privacy-preserving training, TABULA adopts a federated learning paradigm where client-specific embedders capture local (e.g., tissue-specific) patterns, and a global transformer aggregates shared biological features across clients. Despite using only half the data of competing models, TABULA demonstrates state-of-the-art performance on multiple downstream tasks such as gene imputation, perturbation prediction, cell type annotation, and multi-omics integration.", "questions": "1. How sensitive is the model performance to the α parameter balancing reconstruction and contrastive losses? Would different downstream tasks benefit from tuning this?\n2. Could you elaborate on the computational cost of federated training and whether it scales well with additional clients or larger datasets?\n3. Since gene positions are treated as unordered in TABULA, could there be any implicit biases introduced through embedding indices? Have you considered permutation-invariant alternatives?\n4. Can the tabular modeling strategy of TABULA be extended to integrate or pretrain on multi-omics data jointly (e.g., methylation, ATAC-seq)?\n5. While federated training mitigates data sharing, does the model include any formal privacy guarantees (e.g., differential privacy)? Or plans to support that?", "limitations": "Yes. The authors adequately address the limitations in Appendix I, including privacy, data distribution heterogeneity, and the model’s current restriction to single-cell transcriptomics. However, a brief explicit discussion in the main text would improve transparency.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The proposal to model scRNA-seq data explicitly as a table, respecting its unordered and continuous nature, is both novel and well-grounded.\n2. The integration of federated learning directly addresses a growing concern in genomics, offering a scalable and practical solution.\n3. The authors provide thorough evaluations across multiple datasets and tasks (including unseen perturbations and reverse prediction), showing strong performance.\n4. The rationale behind the corruption strategy, tabular modeling, and federated training is clearly articulated and justified.\n\nWeaknesses\n1. While TABULA excels in single-cell applications, its applicability to other tabular biological data (e.g., clinical, bulk RNA-seq) is not explored or discussed.\n2. The dual loss setup and federated training infrastructure introduce engineering and computational overhead; practical deployment in non-institutional settings (e.g., labs without compute clusters) could be challenging.\n3. Although mentioned in the appendix, the main paper could benefit from more visibility on the necessity of both reconstruction and contrastive losses.\n4. While the appendix addresses limitations, more discussion in the main text would strengthen the paper’s transparency.", "was_revised": true, "final_justification": "Thank you for your follow-up and for the detailed clarifications provided during the discussion period. I appreciate the additional information and the effort you invested in addressing my comments. While the responses helped clarify certain points, they do not fully resolve my main concerns. Therefore, I will maintain my original evaluation.", "rebuttal": "Dear reviewer, thank you so much for your valuable comments and recognition of the novelty, effectiveness, and presentation of our work. We are happy to address your concerns and questions with the following results and illustrations.\n\n**Question 1**:  How sensitive is the model performance to the α parameter balancing reconstruction and contrastive losses? Would different downstream tasks benefit from tuning this?\n\n**Answer**: We really appreciate your thoughtful question. We fix α=0.03 so that reconstruction and contrastive losses have similar magnitudes - this stabilizes optimization without manual, task‑specific tuning. Conceptually, α trades off gene‑level (reconstruction) vs. cell‑level (contrast) learning: higher α favors gene embeddings, lower α favors cell embeddings. Although downstream tasks might reap gains from fine‑tuning α, we aim to learn both high-quality cell embeddings and gene embeddings jointly. To that end, we chose α such that both objectives contribute comparably during training. We will clarify this motivation in the revised manuscript.\n\n**Question 2**: Could you elaborate on the computational cost of federated training and whether it scales well with additional clients or larger datasets?\n\n**Answer**: We thank the reviewer for this valuable question. Below we outline both client‐side computation and communication efficiency, backed by our experiments and real‐world runs. \n**Client‐Side Computation**:\nEach client trains its local model in parallel, so the wall‐clock time per round is governed solely by each client’s dataset size and remains effectively constant as you add more clients. Our architecture is lightweight—requiring only ~20 GB of GPU memory per client—so it runs comfortably on standard institutional hardware;\n**Communication Efficiency**:\nTo keep synchronization costs manageable at scale, we (1) decrease the global aggregation frequency as the client count grows—trading off a small number of extra local steps for far fewer global updates—and (2) exchange only the transformer‐encoder weights (partial weight updates) at each sync. This two‐pronged strategy, validated in prior federated‐learning work [1], dramatically cuts communication volume without harming convergence.\nIn Table 1, we report estimated training times (10 epochs, 20 GB GPUs per client) across various dataset sizes and client counts, demonstrating that our framework scales gracefully with both.\n\nIt is important to note that not every lab has computing resources to train a foundation model on its own. Our Tabula framework distributes training workloads across participating clients.\n| Total Dataset Size | # Clients | # Dataset Per Client | Estimated Training Time |\n|:------------------:|:---------:|:------------------:|:------------------------:|\n| 1M                 | 4         | 0.25M              | ~16 hours                |\n| 50M                | 20        | 2.5M               | ~7.5 days                  |\n| 100M               | 40        | 2.5M               | ~8 days                  |\n| 200M               | 80        | 2.5M               | ~9 days                 |\n\n**Table 1**: Estimated training time under different dataset sizes and client configurations.\n\n\n**Question 3**: Since gene positions are treated as unordered in TABULA, could there be any implicit biases introduced through embedding indices? Have you considered permutation-invariant alternatives?\n\n**Answer**: We thank the reviewer for this thoughtful question. We think there may be a slight misunderstanding here. The input gene token ids to the transformer are not fixed — they are dynamically assigned based on the gene order in the specific dataset, which can be randomly permuted during training. More clarification below: \n\n**1. Gene Token ID as Identity Tokens**: Each gene token has a unique index drawn from a shared vocabulary of 23,156 genes, but this index functions solely as an identity lookup key, not as a position marker. It does not encode any ordering, and thus does not introduce order-related bias.\n\n**2. Implicit Permutation via HVG Selection Per Dataset**: For each dataset, we select 1,200 highly variable genes (HVGs) as input. Since HVGs are dataset-specific, the gene columns are effectively shuffled across datasets, meaning that the same input position (e.g., position 10) refers to different genes across datasets. This further reduces the risk of order-dependent bias and reinforces TABULA's order-invariant behavior.\n\n\n**Question 4**: Can the tabular modeling strategy of TABULA be extended to integrate or pretrain on multi-omics data jointly (e.g., methylation, ATAC-seq)?\n\n**Answer**: Thank you for raising this insightful question. It is reasonable and straightforward to extend Tabula on multi-omics data. Take Multiome as an example: for the same single cell, the columns can consist of both peak features (from scATAC-seq) and gene features (from scRNA-seq). Column-wise reconstruction learning can be used to learn peak and gene embeddings separately, while row-level contrastive learning can integrate both modalities to jointly learn the cell embedding. This is a great direction to explore. Large‐scale, paired multi‑omics datasets remain scarce, so we did not incorporate multi‑omics data in our current training. Nonetheless, we view extending Tabula to multi-omics and spatial–temporal modalities as a promising direction for future work.\n\nWe did include multi‑omics integration as a downstream task to evaluate our Tabula’s efficiency (see **Appendix E.4 Task 4&5: Multi-Omics & Multi-Batch Integration**). In that evaluation, we extended the gene vocabulary to include peaks as tokens and fine-tuned jointly on scATAC-seq and scRNA-seq data. From our tabular perspective, this amounts to adding peak columns to the feature table while retaining the same reconstruction and contrastive objectives. As shown in **Appendix Figure 11**, Tabula performs on par with or surpasses scGPT, Geneformer and scBERT in both multi‑omics and multi‑batch integration.\n\n**Question 5**: While federated training mitigates data sharing, does the model include any formal privacy guarantees (e.g., differential privacy)? Or plans to support that?\n\n**Answer**:  We thank the reviewer for this important question. While our current implementation does not yet include formal privacy guarantees such as differential privacy (DP), we plan to incorporate such mechanisms in the next version of our framework. Tabula is fully compatible with client-side DP techniques (e.g., DP-SGD or noise-injected updates), and we view this as a promising direction—particularly in clinical or regulatory settings where stronger formal guarantees are essential.\nNotably, Tabula is, to our knowledge, the first framework to propose a federated foundation model for single-cell omics. We have observed increasing privacy concerns even in publicly available single-cell datasets [2]—a challenge likely to intensify as data scale and sensitivity grow. Existing foundation models largely overlook this issue. Tabula directly addresses this urgent need by enabling collaborative model training without data sharing. We hope this work highlights the importance of federated foundation models in single-cell research and encourages broader community participation.\n\n**Weakness 1**: While TABULA excels in single-cell applications, its applicability to other tabular biological data (e.g., clinical, bulk RNA-seq) is not explored or discussed.\n\n**Answer**: We appreciate the reviewer’s thoughtful comment. Yes, Tabula can be directly applied to other tabular biological data, such as bulk RNA-seq and multi-omics datasets. Please refer to Question 4’s answer for multi-omics data discussion. We will include additional discussion on the applicability of Tabula to other types of biological data in the revised manuscript.\n\n**Weakness 2**: The dual loss setup and federated training infrastructure introduce engineering and computational overhead; practical deployment in non-institutional settings (e.g., labs without compute clusters) could be challenging.\n\n**Answer**: We thank the reviewer for this excellent question and fully agree with the concern. Our goal is not only to propose a method but also to build a practical platform that benefits the broader research community. To support wider adoption, we will release Chiron, a distributed foundation model training platform, by the end of October. On Chiron, any user can launch a training job, and clients with private data can participate without sharing raw data. Importantly, no complex engineering setup is needed—clients simply upload their data to a local server (e.g., HPC) and run a single command to connect. Once connected, they can join any training job on the platform. Chiron enables scalable, privacy-preserving collaboration with minimal setup, and we’re extremely excited about Chiron release this year.\n\n**Weakness 3**:  Although mentioned in the appendix, the main paper could benefit from more visibility on the necessity of both reconstruction and contrastive losses.\n\n**Answer**: We thank the reviewer for this helpful suggestion. We totally agree with this. Accordingly, we will move and expand the relevant discussion from the appendix into the main text in the revised manuscript.\n\n**Weakness 4**:  While the appendix addresses limitations, more discussion in the main text would strengthen the paper’s transparency.\n\n**Answer**: We thank the reviewer for this valuable feedback. We agree with this. In the revised version, we will move and expand the relevant content from the appendix to the main manuscript.\n\n[1] Wang, Haolin, et al. \"Why Go Full? Elevating Federated Learning Through Partial Network Updates.\" NeurIPS. 2024.\n\n[2] Walker, C. R., Li, X., Chakravarthy, M., Lounsbery-Scaife, W., Choi, Y. A., Singh, R., & Gürsoy, G. (2024). Private information leakage from single-cell count matrices. Cell, 187(23), 6537-6549."}, {"review_id": "HmEDVh4ZhO", "reviewer": "Reviewer_xZMw", "summary": "The paper presents a privacy preserving foundation model for single cell transcriptomics that explicitly models the cell*gene matrix using a dual axis self supervised objective and federated learning. The model is empirically tested in a realistic federated (decentralized) setting, with data partitioned across multiple clients (tissues or institutions), each training locally and sharing only model weights for aggregation. The authors benchmark federated versus centralized pretraining, showing that federated pretraining matches or exceeds centralized performance despite data heterogeneity. TABULA achieves state of the art or competitive results on gene imputation, perturbation prediction, cell type annotation, and multiomics integration, often using only half the pretraining data. Scaling law experiments further demonstrate the benefits of large scale, decentralized pretraining. However, the work would benefit from deeper biological validation, more explicit benchmarking of computational and privacy trade-offs, and further analysis of interpretability and failure modes.", "questions": "In general, I do not have many comments, just a couple of questions. In particular, I am interested: \n- can the authors think of some issues related to practical challenges and privacy risks with federated learning (e.g. model inversion/membership inference, client drift, etc.) in particular for this domain? Furthermore, what are some of the failure cases that might arise from federated learning in this domain?\n- are there particular types of datasets or biological contexts that are not considered in this paper where the authors expect this method to perform poorly?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- The Tabula model is among the first foundation models in single cell omics to implement and empirically test decentralized pretraining, with a clear protocol and direct benchmarking against centralized training. The authors present results that show that federated pretraining matches or exceeds centralized performance, even under data heterogeneity.\n- The 'dual axis' self supervised objective (column wise gene reconstruction and row wise cell contrastive learning) directly addresses the tabular structure of single cell data.\n- The presented model is strong. The evaluation is comprehensive, and shows that Tabula achieves either state of the art or at least competitive results across several tasks (gene imputation, perturbation prediction, cell type annotation and multi omics integration), often with less pretraining data than prior models. Ablation studies confirm the importance of the proposed modeling objectives.\n- The authors perform an analysis of scaling laws, which supports the benefit of large scale decentralized pretraining and shows that such benefits hold under federated learning. \n- The discussion candidly acknowledges limitations and open challenges, including the need for further biological validation and practical benchmarking of federated learning. The manuscript is generally well organized, with clear figures and tables that support the main claims.\n\n**Weaknesses**\n- The results focus on average performance but do not analyze where the model fails (e.g., rare cell types, strong batch effects, missing data), nor do they provide qualitative examples of incorrect predictions. Furthermore, while the model is benchmarked on several datasets, all are within the same general domains. I would have liked to see more tests of generalization to e.g. new cell types, tissues, species, or even data modalities (e.g., ATAC-seq, proteomics). It seems the model's generalization is limited to perturbation space within a fixed biological context, rather than full OOD generalization.\n- I would have liked to see a bit more discussion on, and analysis of important issues of biological single cell models, such as robustness to batch effects, technical noise, or integration across different sequencing platforms, which are major sources of variation in real-world single-cell data.\n- Regarding intepretability, would have liked more analysis of e.g. learned embeddings or attention maps for biological plausibility. The results presented in this paper are quite interesting as is the method, so a bit more focus on this (even in the appendix) would be valuable.\n- Reporting of error bars and statistical significance is inconsistent - there are experiments for which these are not included and metrics are reported as single values with no indication for robustness or reproducibility.\n- Code/data release is not specified. There is no mention of random seed control, environment specification, or pipeline reproducibility.", "was_revised": true, "final_justification": "I am satisfied with the response of the authors and the discussion and other comments on this paper and will be maintaining my score.", "rebuttal": "Dear reviewer, thank you so much for your valuable comments and recognition of the novelty, effectiveness, and presentation of our work. We are happy to address your concerns and questions with the following results and illustrations.\n\n**Question 1:** Can the authors think of some issues related to practical challenges and privacy risks with federated learning (e.g. model inversion/membership inference, client drift, etc.) in particular for this domain? Furthermore, what are some of the failure cases that might arise from federated learning in this domain?\n\n**Answer:** We thank the reviewer for raising this important point. Beyond known risks like model inversion and membership inference, federated learning in single-cell genomics presents several practical challenges that are critical for real-world deployment.\n\nFirst, data quality control is essential, as the global model cannot access local data. Noisy or mislabeled samples can degrade model performance, especially for rare cell types. We plan to incorporate client-side quality checks and adaptive weighting in future work.\n\nSecond, there is a technical expertise gap among many biologists unfamiliar with distributed systems. To address this, we are developing Chiron, a platform that allows users to join training jobs by simply uploading data to a local server and running a single command—no complex setup required.\n\nThird, security threats are evolving, and current defenses for scRNA-seq privacy are still emerging. As federated adoption grows, more sophisticated attacks may compromise sensitive genetic data, especially in rare disease contexts. This underscores the need for ongoing development of domain-specific privacy strategies.\n\nWe will include this discussion and our deployment vision in the revised manuscript.\n\n**Question 2:** are there particular types of datasets or biological contexts that are not considered in this paper where the authors expect this method to perform poorly?\n\n**Answer:**  Thank you for the question. While Tabula is currently pre-trained only on scRNA-seq data, it is designed to be readily extensible to other biological contexts and data types.\n\nFor example, multi-omics integration is straightforward within Tabula’s tabular framework. In the case of Multiome data, where each cell is profiled with both scRNA-seq and scATAC-seq, the input table can include both gene expression features and chromatin accessibility peak features as separate columns. The column-wise reconstruction objective can then be applied independently to each modality to learn robust gene and peak embeddings, while the row-wise contrastive objective integrates both modalities to learn a joint cell representation.\n\nSimilarly, spatial transcriptomics data can be incorporated by adding the 2D or 3D spatial coordinates of each cell as additional numeric columns. We can include such spatial information along with a dedicated spatial loss function to better capture local tissue context and cell–cell interactions.\n\n**Weakness 1:** The results emphasize average performance but lack analysis of failure modes (e.g., rare cell types, batch effects) or qualitative errors. Can the authors evaluate generalization to new cell types, tissues, species, or modalities beyond the current domain?\n\n**Answer:** We thank the reviewer for this valuable feedback. To assess Tabula’s generalizability, we conducted a challenging zero-shot task focused on recovering pairwise and higher-order combinatorial gene regulation across four distinct developmental systems: hematopoiesis, cardiogenesis, neurogenesis, and pancreatic endogenesis—each supported by well-characterized regulatory networks.\n\nAs noted in our response to Reviewer Xu6M, Question 1, Tabula achieved strong zero-shot performance, with average accuracies of 0.878 for pairwise and 0.888 for combinatorial inference—without fine-tuning. These results highlight Tabula’s ability to generalize across diverse systems and demonstrate the effectiveness of its pretraining strategy. We will clarify these findings in the revised manuscript.\n\n**Weakness 2:**  I would have liked to see a bit more discussion on, and analysis of important issues of biological single cell models, such as robustness to batch effects, technical noise, or integration across different sequencing platforms, which are major sources of variation in real-world single-cell data.\n\n**Answer:** Thank you for this insightful comment. In our downstream evaluations (**Appendix E.4**), Tabula is explicitly benchmarked on multi-omics integration and batch correction tasks. The results demonstrate Tabula’s strong performance across diverse datasets, highlighting its ability to generalize across batches and sequencing protocols.\n\nAs the reviewer noted, addressing batch effects and technical variability is critical in single-cell analysis. Factors like sequencing platform, preservation method, and sample handling can introduce substantial noise. To mitigate this, Tabula incorporates several modeling choices during pretraining. Specifically, our gene expression binning strategy—adapted from scGPT—focuses on relative rather than absolute expression levels, helping reduce platform-specific noise while preserving biological signals.\n\nMoreover, Tabula’s dual-axis learning framework captures both inter-gene and inter-cell relationships, enabling the model to learn distributional patterns that may implicitly reflect batch effects.\n\nThat said, we acknowledge that robust integration often benefits from task-specific fine-tuning. While Tabula offers a strong, privacy-preserving initialization, fine-tuning remains essential for optimizing performance on heterogeneous downstream tasks.\n\n**Weakness 3:**  Regarding interpretability, would have liked more analysis of e.g. learned embeddings or attention maps for biological plausibility. The results presented in this paper are quite interesting as is the method, so a bit more focus on this (even in the appendix) would be valuable.\n\n**Answer:** In **Appendix F**, we further investigated the tissue-specific embedding learned by our federated learning framework to illustrate how the model captures distinctive tissue features. As shown in **Appendix Figure 12 a,b**, the tissue-specific embedder encodes discernible differences in both gene and cell embeddings. Moreover, we conducted a zero-shot multi-omics integration experiment on a brain dataset using random, lung, and brain embedders (**Appendix Figure 12c**). The results demonstrate that tissue-specific learning more effectively captures tissue-relevant representations, thereby facilitating improved performance on tissue-specific downstream tasks. In short, all those experiments explore the learned embedding of Tabula to show its practical biological plausibility. \n\nFurthermore, as we shown the response in **reviewer Xu6M question 1**, we showcased a zero-shot setting to explore activation and repression relationships between key transcription factors (genes). Under the two scenarios (single-gene pairwise inference and multi-gene combinatorial inference) presented above, Tabula achieves an average accuracy of 0.878 on pairwise inference and 0.888 on combinatorial inference. This innovative experiment provides solid evidence of biological interpretability for Tabula.\n\n\n**Weakness 4:**  Reporting of error bars and statistical significance is inconsistent - there are experiments for which these are not included and metrics are reported as single values with no indication for robustness or reproducibility.\n\n**Answer:** We repeated the experiments five times using five consistent random seeds across Tabula, scGPT, and Geneformer. Leveraging the GEARS package for data splitting, each seed generates distinct partitions of perturbed genes into train, validation, and test. We report the mean Pearson across 1/1 & 2/2 unseen splits for the Norman dataset and 1/1 unseen splits for Adamson and Replogle datasets. Tabula consistently outperforms the other models across nearly all evaluation metrics, except for the DE Genes in Adamson. These results highlight not only the robustness of Tabula but also its strong generalizability, as all test sets comprise perturbed genes entirely excluded from both training and validation. \n\n**Table 1: Gene Perturbation Prediction Performance**  \n*Performance (mean ± s.d.) across three benchmark datasets using all genes and differentially expressed (DE) genes.*\n\n**Note**: Geneformer Replogle are not finished due to limited resource, we will record and fill it out if accepted. \n\n| Model       | Norman All Genes | Norman DE Genes | Adamson All Genes | Adamson DE Genes | Replogle All Genes | Replogle DE Genes |\n|-------------|------------------|------------------|--------------------|-------------------|---------------------|--------------------|\n| **Tabula**      | **0.543±0.0043**  | **0.666±0.034**    | **0.719±0.044**     | 0.849±0.045       | **0.314±0.007**      | **0.464±0.017**     |\n| scGPT       | 0.467±0.0058     | 0.603±0.061      | 0.659±0.032        | 0.864±0.052       | 0.247±0.022         | 0.451±0.014        |\n| geneFormer  | 0.331±0.0028     | 0.527±0.087      | 0.248±0.018        | **0.865±0.050**    | 0.082                  | 0.362                  |\n\nWe also conducted five repeated experiments with different random seeds for cell type annotation on Tabula, using benchmark methods including TOSICA  and CellPLM , all run with their default settings. Please refer to the response of **Reviewer wTtD Weakness 1**.\n\n**Weakness 5:**  Code/data release is not specified. There is no mention of random seed control, environment specification, or pipeline reproducibility.\n\n**Answer:** Thank you for pointing this out. The code was provided along with the Tabula appendix document. We will include additional details in the revised version regarding random seed control, environment specification, and pipeline reproducibility to ensure greater transparency and ease of replication."}, {"review_id": "UwHCZRlDLS", "reviewer": "Reviewer_wTtD", "summary": "The paper proposes a foundation model for single-cell RNA-seq data, building on the Xtab architecture, which treats gene expression profiles as tabular data. Unlike prior foundation models such as scGPT that adopt transformer-based sequence modeling, Tabula employs row-wise contrastive learning and column-wise masked reconstruction objectives to learn cell and feature representations, respectively. To address data privacy constraints inherent to biomedical datasets, the model is trained using a federated learning framework across decentralized tissue-level datasets. The authors evaluate the pretrained model on five downstream tasks, demonstrating performance that matches or exceeds existing baselines, despite using fewer pretraining samples. The approach is well-motivated for the data modality, and the use of federated pretraining is a particularly notable design choice in the context of regulated transcriptomic data.", "questions": "See weaknesses above for major questions. I am noting some minor questions below:\n\n1. What's the effect of batch-size/batch diversity in pretraining given the use of the contrastive loss? \n2. Given that the embedders are all trained tissue-wise, are there any performance degradtions for normal v/s diseased cells?", "limitations": "The authors describe the limitations clearly in the supplementary material.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-motivated and easy to follow. The approach itself, while borrowing from Xtab, is well-suited towards scRNA-seq data.\n2. The experiments are well-designed, and the results support the efficacy of the architecture.\n3. While inspired by Xtab, the use of federated learning is especially compelling in this context, given the privacy constraints commonly associated with transcriptomics data.\n\nWeaknesses:\n\n1. While the results are interesting and support the conclusions presented by the authors, the cell annotation results do not appear to be conclusive. Could the authors provide error bars for the cell annotation table? I also suggest adding non foundation model approaches like TOSICA to ground the results. \n\n2. In terms of originality, several architectural and training contributions seem to be adapted from Xtab. This should be clearly highlighted in the paper, with differences specific to Tabula listed.  \n\n3. I also suggest benchmarking against CellPLM for the benchmarks. CellPLM reportedly outperforms scGPT on several downstream tasks, and also leverages multiple cells during pretraining, making it a fairer model to compare against.\n\n4. The last weakness is perhaps a bit more foundational, but Kedzierska et al. (https://www.biorxiv.org/content/10.1101/2023.10.16.561085v1) show significant results where scGPT and Geneformer underperform simpler non-foundation models in the zero-shot setting. Can the authors present/comment on Tabula's zero-shot performance, or the effectiveness of pre-training?", "was_revised": true, "final_justification": "The authors have presented clear responses to the weaknesses pointed out in the review. I believe the paper would be a valuable contribution to the field.", "rebuttal": "Dear Reviewer, Thank you for your valuable feedback and for recognizing the novelty, effectiveness, and clarity of our work. We are pleased to address your questions and concerns with the following results and illustrations.\n\n\n**Question 1:** What's the effect of batch-size/batch diversity in pretraining given the use of the contrastive loss?\n\n**Answer:** We appreciate the reviewer's question regarding batch size effects. We employed a batch size of 32 based on two considerations: First, our contrastive loss creates positive pairs between cells and their corrupted views, with other cells as negatives. Since data within each client comes from the same tissue type, cells exhibit greater similarity compared to the broader population, which may influence contrastive dynamics within batches. Second, larger batch sizes, while potentially beneficial for contrastive learning, require substantially greater computational resources. We acknowledge that exploring batch diversity effects represents a valuable direction for future work and appreciate the reviewer's insightful suggestion.\n\n**Question 2:**  Given that the embedders are all trained tissue-wise, are there any performance degradtions for normal v/s diseased cells?\n\n**Answer:** We appreciate the reviewer's important question about normal versus diseased cell performance. Our pretraining was conducted exclusively on normal human cells without including diseased samples. We acknowledge that for applications involving diseased cells, fine-tuning would likely be necessary to achieve optimal performance. This represents an important consideration for future work, and we thank the reviewer for highlighting this aspect.\n\n**Weakness 1:** While the results are interesting and support the conclusions presented by the authors, the cell annotation results do not appear to be conclusive. Could the authors provide error bars for the cell annotation table? I also suggest adding non foundation model approaches like TOSICA to ground the results.\n\n**Answer:** Thank you for the suggestion. Due to time constraints, we conducted five independent runs only for the hPancreas and Cell-Lines datasets. Each method was evaluated using five different random seeds, and we now report the mean ± standard deviation for all metrics. In addition to foundation model baselines, we have included CellPLM (a foundation model) and TOSICA (a non-foundation method) to further ground the comparison.\n\nTabula attains the highest accuracy and precision on both datasets and the best overall F1 on the Cell-Lines set, while scGPT achieves a slightly higher F1 on hPancreas, and CellPLM delivers the top recall. Importantly, Tabula reaches these results after being pre-trained on only ~15 M cells—roughly half the data used by leading foundation models such as scGPT and geneFormer—highlighting its tabular modeling efficiency.\n\nThese error bars and additional baselines will be incorporated into the revised manuscript.\n\n**Table 1: hPancreas Dataset Performance**  \n| Model         | Test Accuracy ↑     | Test Precision ↑     | Test Recall ↑        | Test F1 ↑            |\n|---------------|----------------------|------------------------|------------------------|------------------------|\n| **Tabula**     | **0.9716±0.0059**     | **0.7586±0.0164**       | 0.7295±0.0286          | 0.7277±0.0257          |\n| scGPT         | 0.9694±0.0062         | 0.7523±0.0359           | 0.738±0.0259           | **0.7343±0.022**       |\n| scFoundation  | 0.9548±0.0054         | 0.6277±0.0408           | 0.6247±0.0394          | 0.6191±0.0388          |\n| geneFormer    | 0.932±0.0204          | 0.6814±0.0998           | 0.6446±0.0572          | 0.6399±0.0654          |\n| cellPLM       | 0.7504±0.023          | 0.7409±0.0245           | **0.7504±0.023**       | 0.7319±0.0202          |\n| TOSICA        | 0.9648±0.0037         | 0.6301±0.0227           | 0.6633±0.0173          | 0.6363±0.0208          |\n\n\n**Table 2: Cell Lines Dataset Performance**  \n| Model        | Test Accuracy ↑     | Test Precision ↑     | Test Recall ↑        | Test F1 ↑            |\n|--------------|----------------------|------------------------|------------------------|------------------------|\n| **Tabula**     | **0.9933±0.0004**     | **0.9937±0.0005**       | **0.9928±0.0004**       | **0.9932±0.0004**       |\n| scGPT        | 0.9931±0.0005         | 0.9934±0.0003           | 0.9926±0.0007           | 0.9931±0.0004           |\n| scFoundation | 0.9931±0.0002         | 0.9935±0.0004           | 0.9927±0.0002           | 0.9931±0.0002           |\n| geneFormer   | 0.9914±0.0005         | 0.9918±0.0004           | 0.991±0.0006            | 0.9914±0.0005           |\n\n\n**Weakness 2:** In terms of originality, several architectural and training contributions seem to be adapted from Xtab. This should be clearly highlighted in the paper, with differences specific to Tabula listed.\n\n**Answer:** Thank you for raising this point. We will explicitly acknowledge XTab as our architectural foundation in the revision. Our contribution lies in adapting XTab’s tabular framework to the unique challenges of single-cell genomics—extreme sparsity, high dimensionality, and tissue heterogeneity. Specifically, we introduce tissue-specific embedders (see **Appendix F, Tissue-Specific Embedders Encode Distinctive Tissue Features**), where each client learns its own embedder to capture tissue-level variation while only the shared Transformer is aggregated with FedAvg, preserving privacy across sites. We also replace XTab’s supervised loss with a dual-axis self-supervision scheme that combines column-wise gene reconstruction and row-wise contrastive learning, both tailored to sparse expression matrices. These distinctions will be emphasized in the main text.\n\n**Weakness 3:** I also suggest benchmarking against CellPLM for the benchmarks. CellPLM reportedly outperforms scGPT on several downstream tasks, and also leverages multiple cells during pretraining, making it a fairer model to compare against.\n\n**Answer:** Please refer to our response to Weakness 1 for the detailed results of CellPLM, which has been included alongside other baselines in the cell type annotation comparison.\n\n**Weakness 4:** The last weakness is perhaps a bit more foundational, but Kedzierska et al. (https://www.biorxiv.org/content/10.1101/2023.10.16.561085v1) show significant results where scGPT and Geneformer underperform simpler non-foundation models in the zero-shot setting. Can the authors present/comment on Tabula's zero-shot performance, or the effectiveness of pre-training?\n\n**Answer:** We thank the reviewer for raising this important concern regarding zero-shot performance. While foundation models have shown promise across diverse tasks, a truly valuable single-cell foundation model must demonstrate accuracy, consistency, and generalizability in zero-shot settings—that is, the ability to perform well in entirely new situations without task-specific training—and derive mechanistic biological insights across a wide range of systems. To demonstrate this, we take on a novel and challenging task by validating Tabula’s performance in recovering **pairwise** and **higher-order combinatorial gene regulation** across four distinct developmental systems, ranging from hematopoiesis, cardiogenesis, neurogenesis, and pancreatic endogenesis. These systems have well-characterized regulatory networks validated through decades of biochemical research.\n\nAs detailed in our response to **Reviewer Xu6M, Question 1**, Tabula achieves strong performance in this setting: an average accuracy of 0.878 in pairwise gene inference and 0.888 in combinatorial inference—without any additional fine-tuning. These results provide compelling evidence that Tabula can generalize across diverse developmental systems, uncover meaningful biological mechanisms in a zero-shot manner, and demonstrate the efficacy of its pretraining strategy. We will clarify these findings and their implications in the revised manuscript."}, {"review_id": "jjwBbFlrW4", "reviewer": "Reviewer_ZGxU", "summary": "This paper introduces TABULA, a foundation model for single-cell transcriptomics that proposes two primary innovations over existing models: a novel self-supervised \"tabular learning\" objective and a federated learning framework for privacy-preserving pre-training. The tabular learning objective combines a gene reconstruction task, using a corruption strategy based on marginal distribution sampling, with a cell contrastive learning task. The model is evaluated on a range of downstream tasks, where it reportedly achieves state-of-the-art performance, particularly in genetic perturbation prediction.", "questions": "**Questions for the Authors**\nThe preliminary study in Figure 3 suggests that federated learning not only works but actually improves performance over centralized training. This is a surprising and counter-intuitive result, as federated learning typically incurs a performance penalty due to communication overhead and non-IID data distributions. Could you provide some intuition or a hypothesis for why this performance gain occurs in your framework?\n\nIn the gene imputation task, you compare TABULA to baselines after fine-tuning a decoder. Could you clarify if the baseline models (scGPT, Geneformer, etc.) were also fine-tuned for this task, or if they were used in a zero-shot setting as is common? Ensuring the experimental setup is identical is critical for a fair comparison.\n\nCould you provide a more detailed ablation study that isolates the individual contributions of the two components of your tabular loss (L_rec and L_contrast)? It would be valuable to understand how much each component contributes to the final performance.\n\nGiven the significant claims about your novel corruption strategy, could you provide an ablation that directly compares it to standard masking while keeping the corruption ratio the same (e.g., 60%)? This would help clarify whether the benefit comes from the strategy itself or simply the higher degree of input augmentation.\n\n**Conclusion and Recommendation**\n\nThis paper presents some interesting and promising ideas, particularly the novel pre-training objectives and the strong results in gene perturbation prediction. The presentation is also of high quality. However, these strengths are undermined by a flawed and overstated motivation, a lack of clarity regarding the true technical novelty, and an experimental evaluation that lacks the necessary rigor and depth, especially concerning the ablation and scaling studies. The federated learning component feels somewhat disconnected, and its surprising performance benefit is not adequately explained.\n\nWhile the work has potential, it requires a major revision to address these fundamental issues. The authors should reframe their contributions to focus on the specific self-supervised objectives, provide a more rigorous and insightful experimental analysis, and clarify the points of confusion raised in this review. In its current state, the paper does not meet the bar for publication. I would be open to reconsidering a substantially revised version.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well-presented, with a clear structure and high-quality figures that make it relatively easy to read and follow the authors' high-level arguments. The results on the genetic perturbation prediction task are particularly strong and represent a notable achievement. Furthermore, the ideas of using a more naturalistic corruption strategy (sampling from the marginal distribution) instead of artificial masking and employing a cell-level contrastive loss are conceptually interesting and promising directions for the field.\n\n**Major Weaknesses**\n\nFraming and Motivation: The paper's core motivation and positioning feel weak and, at times, misleading.\n\nThe central argument that existing models \"overlook the tabular nature of single-cell data\" by converting it to sequences is a mischaracterization. The permutation-invariant nature of gene expression is a well-understood problem that has been explicitly addressed by nearly all major single-cell foundation models (e.g., through the omission of positional encodings or the use of gene ranking).\n\nThe privacy argument for federated learning, while valid in a general sense, feels overstated as a primary motivator for this specific application. Secure handling of anonymized public scRNA-seq data is standard practice, and it is not currently perceived as a major bottleneck for the field in the same way as, for example, genomic data linked to individuals. The disjointed way the paper combines the \"tabular nature\" argument with the privacy argument suggests a post-hoc framing rather than a cohesive research goal.\n\nOverstated Novelty of \"Tabular Learning\": The paper claims to introduce a new \"tabular learning\" paradigm, but the underlying model architecture is largely identical to prior work (e.g., scGPT, CellPLM). The model still embeds gene IDs and expression values, sums them, and processes them with a standard Transformer. The true novelty lies in the specific choice of self-supervised objectives: (1) a reconstruction loss on values corrupted via marginal sampling, and (2) a cell-level contrastive loss. While these are interesting contributions, they are specific loss functions, not a fundamentally new way of modeling tabular data. The authors should tone down the framing and focus on clearly justifying and analyzing these specific contributions.\n\nInsufficient Experimental Rigor: The experimental evaluation, particularly the ablation and scaling studies, lacks the depth needed to convincingly support the paper's claims.\n\nMissing Details: The number of parameters for the TABULA model is not stated in the main paper, making it difficult to assess its efficiency and make fair comparisons.\n\n\"Scaling Law\" Analysis: The study presented in Figure 4b is not a proper scaling law analysis. Simply showing that performance on one task improves across three data points (300k, 3M, 15M) is expected behavior. A rigorous scaling law study should analyze performance as a function of compute, model size, and data size to identify optimal training regimes (e.g., isoFLOP curves), which is not done here. This section is overclaimed and should be reframed or significantly expanded.\n\nAblation Study: The ablation comparing the full \"tabular\" objective against a standard MLM objective with 15% masking is not an apples-to-apples comparison. The corruption rate is drastically different (60% vs. 15%), and the \"tabular\" objective combines two distinct losses. A more informative set of ablations would be required to disentangle the effects of:\n\n- The reconstruction loss vs. the contrastive loss.\n\n- The corruption strategy (marginal sampling vs. masking).\n\n- The corruption rate (e.g., comparing both strategies at 15% and 60%).", "was_revised": true, "final_justification": "I have raised my score from 3 (Borderline reject) to 5 (Weak accept) based on the authors' thorough and responsive rebuttal. They have successfully addressed the major concern raised in my initial review.\n\nMost importantly, they provided the ablation studies I requested, which now convincingly disentangle the contributions of their proposed loss components (reconstruction vs. contrastive) and their novel corruption strategy (marginal sampling vs. masking). The new data strongly supports their claims and resolves my primary concerns about experimental rigor.\n\nFurthermore, they have addressed all other key points: they provided model parameters and new efficiency benchmarks, offered a plausible explanation for the federated learning results, grounded their privacy motivation with a relevant high-impact citation, and agreed to tone down the framing of their contributions to avoid overclaiming.\n\nAssuming these significant changes and new results are incorporated into the final manuscript as promised, the paper's core weaknesses will be resolved. The work's novel pre-training objectives and strong empirical results now represent a solid and valuable contribution to the field. I am confident in changing my recommendation to acceptance.", "rebuttal": "Thank you for your valuable feedback and for recognizing the novelty, effectiveness, and clarity of our work. Please find our detailed responses to your comments below.\n\n**Question 1:** Figure 3 shows federated learning outperforming centralized training, which is surprising given typical drawbacks like communication overhead and non-IID data. Can the authors explain why this performance gain occurs?\n\n**Answer:** Thank you for this great question. You're absolutely right—the benefits largely come from using tissue-specific embedders. In Tabula’s pretraining, each client is trained on a specific tissue dataset, which aligns with the tissue-specific nature of most biological data and improves downstream performance. More demonstration in **Appendix F**. \n\n**Question 2:**  In the gene imputation task, you compare TABULA to baselines after fine-tuning a decoder. Could you clarify if the baseline models (scGPT, Geneformer, etc.) were also fine-tuned for this task, or if they were used in a zero-shot setting as is common? Ensuring the experimental setup is identical is critical for a fair comparison.\n\n*Answer:* Thank you for this important clarification. All baseline models were fine-tuned for the gene imputation task under identical experimental conditions to ensure fair comparison. Detailed experimental specifications can be found in Appendix E.1.\n\n**Question 3:**  Could you provide a more detailed ablation study that isolates the individual contributions of the two components of your tabular loss (L_rec and L_contrast)? It would be valuable to understand how much each component contributes to the final performance.\n\n*Answer:* Thank you for this valuable suggestion. We have conducted ablation studies isolating the individual contributions of L_rec and L_contrast components of our tabular loss. The detailed results and analysis of how each component contributes to the final performance can be found in **Appendix G**.\n\n**Question 4:** Given the significant claims about your novel corruption strategy, could you provide an ablation that directly compares it to standard masking while keeping the corruption ratio the same (e.g., 60%)? This would help clarify whether the benefit comes from the strategy itself or simply the higher degree of input augmentation.\n\n**Answer:** Thank you for the insightful suggestion. Following prior work (e.g., scBERT, Geneformer), we used a 15% masking ratio and conducted ablations on (1) loss type, (2) corruption strategy (marginal sampling vs. masking), and (3) corruption rate (15% vs. 60%). Results show that marginal sampling consistently outperforms masking, demonstrating the effectiveness of our dual-axis pretraining and the value of a more biologically realistic corruption strategy.\n\n**Table: Ablation Study on Gene Imputation Task (Lower RMSE/MAE, Higher Pearson Correlation)**\n\n(**Note**: \"corrupted\" indicates marginal sampling used in Tabula.)\n\n|  | **PBMC5K** |  |  | **hPancreas** |  |  | **Jurkat** |  |  |\n|-------------|------------|--|--|---------------|--|--|------------|--|--|\n| **Ablation** | **RMSE** | **MAE** | **Pearson** | **RMSE** | **MAE** | **Pearson** | **RMSE** | **MAE** | **Pearson** |\n| **Loss Ablation** |  |  |  |  |  |  |  |  |  |\n| TABULA (only contrastive) | 1.1431 | 0.9715 | 0.4872 | 0.787 | 0.5328 | 0.0108 | 0.9542 | 0.8314 | 0.6043 |\n| TABULA (only reconstruction) | 0.9856 | 0.8457 | 0.5647 | 0.5748 | 0.3355 | **0.3386** | 0.7403 | 0.6096 | 0.5848 |\n| TABULA (contrastive+reconstruction) | **0.9298** | **0.8016** | **0.5934** | **0.573** | **0.3226** | 0.3255 | **0.6295** | **0.5138** | **0.6511** |\n| **Corruption Strategy Ablation** |  |  |  |  |  |  |  |  |  |\n| TABULA (corrupted 0.6) | **0.9856** | **0.8457** | **0.5647** | **0.5748** | **0.3355** | **0.3386** | **0.7403** | **0.6096** | 0.5848 |\n| TABULA (mask 0.6) | 1.1978 | 1.0469 | 0.4244 | 0.6778 | 0.3985 | 0.2154 | 1.0268 | 0.9274 | **0.697** |\n| **Corruption Rate Ablation** |  |  |  |  |  |  |  |  |  |\n| TABULA (corrupted 0.6+contrastive) | **0.9298** | **0.8016** | **0.5934** | **0.573** | **0.3226** | **0.3255** | **0.6295** | **0.5138** | 0.6511 |\n| TABULA (mask 0.6) | 1.1978 | 1.0469 | 0.4244 | 0.6778 | 0.3985 | 0.2154 | 1.0268 | 0.9274 | **0.697** |\n| TABULA (corrupted 0.15 + contrastive) | **0.9913** | **0.8587** | **0.5728** | **0.5881** | **0.3264** | **0.2791** | **0.7593** | **0.6088** | 0.5313 |\n| TABULA (mask 0.15) | 1.2498 | 1.093 | 0.3569 | 0.6893 | 0.3965 | 0.1821 | 1.053 | 0.9571 | **0.696** |\n\n**Weakness 1:** Framing and Motivation: The paper's core motivation and positioning feel weak and, at times, misleading.\n\n**Answer:** Thank you for the thoughtful feedback. We clarify our motivation in the Design Principles section, highlighting three core ideas: (1) realistic corruption via marginal sampling, (2) tabular modeling through dual objectives, and (3) federated learning for privacy-preserving collaboration. These address key gaps in biological fidelity, structure, and data privacy. We will emphasize this framing more clearly in the revised Introduction.\n\n**Weakness 2:** The claim that existing models \"overlook the tabular nature\" of single-cell data is misleading. Major models already address permutation invariance via techniques like omitting positional encodings or using gene ranking.\n\n**Answer:** The key point we intend to emphasize is that existing foundation models do not explicitly model the tabular structure of single-cell data through their pretraining objectives, whereas Tabula is specifically designed to do so. We will revise the manuscript to better convey this distinction.\n\n**Weakness 3:** The privacy argument feels overstated for this setting, as public scRNA-seq data is generally anonymized. Combining tabular modeling and privacy appears somewhat post-hoc rather than a cohesive objective.\n\n**Answer:** We appreciate the reviewer’s perspective and would like to clarify that privacy risks in public scRNA-seq data are an emerging concern, particularly in light of recent findings. Specifically, the Cell paper \"Private information leakage from single-cell count matrices\" has demonstrated that cell-by-gene expression matrices—despite being anonymized—can still leak sensitive donor-level information. We discuss this in more detail in **Appendix D: Privacy Leakage in Single-Cell Expression Matrices**. \n\n**Weakness 4:**  The \"tabular learning\" claim seems overstated, as the architecture is similar to scGPT. The main novelty lies in the self-supervised objectives and should be framed as such, not as a new paradigm.\n\n**Answer:** We agree with the reviewer that the embedder architecture is similar to that of scGPT. However, the core value of a foundation model lies in the design of its pretraining objectives, which fundamentally determine what the model learns. In Tabula, our notion of tabular modeling is reflected in the objective design—through column-wise gene reconstruction and row-wise cell contrastive learning. This dual-axis formulation enables the model to explicitly capture the structure of the single-cell expression table.\n\nIn addition, we introduce a more realistic corruption strategy based on marginal resampling, as opposed to standard masking. As demonstrated in our ablation study, this strategy significantly improves downstream task performance, demonstrating its efficacy.\n\nWe will revise the manuscript to better reflect these contributions and clarify our use of the term \"tabular modeling.\n\n**Weakness 5:**  Insufficient Experimental Rigor: The experimental evaluation, particularly the ablation and scaling studies, lacks the depth needed to convincingly support the paper's claims. \n\n**Answer:** Please refer to more ablation studies in the response to Question 4.\n\n**Weakness 6:** Missing Details: The number of parameters for the TABULA model is not stated in the main paper, making it difficult to assess its efficiency and make fair comparisons.\n\n**Answer:**  Thank you for this helpful feedback. We will include the number of parameters for Tabula (approximately 13 million) in the revised manuscript to support fair comparisons.\n\nWe also benchmarked inference efficiency by measuring the time each model takes to process one million cells from the Norman perturbation dataset. Each model was run three times on a single NVIDIA A6000 GPU, and we report the average runtime and standard deviation (in hours).\n\nAs shown in the table (to be included in the revised version), Tabula is over 2× faster than other models and consistently achieves higher accuracy in perturbation prediction—demonstrating both its effectiveness and computational efficiency for downstream tasks.\n\n| Model       | Efficiency (Hr)     |\n|-------------|---------------------|\n| **TABULA**      | **4.65 ± 0.79**        |\n| scGPT       | 10.10 ± 0.72        |\n| Geneformer  | 10.37 ± 1.76        |\n\n**Weakness 7:** Scaling Law Overclaim: The analysis in Figure 4b is not a rigorous scaling law study. Showing performance improvement over three data points is expected and insufficient.\n\n**Answer:** Thank you for the valuable feedback. We completely agree with your point—our intention was to illustrate a data scaling rather than to present a rigorous scaling law analysis. We acknowledge the confusion and will revise the text to clarify this distinction and avoid overclaiming in the revised manuscript.\n\n**Weakness 8:** Ablation Study: A more informative set of ablations would be required to disentangle the effects of:\nThe reconstruction loss vs. the contrastive loss. \nThe corruption strategy (marginal sampling vs. masking). \nThe corruption rate (e.g., comparing both strategies at 15% and 60%).\n\n**Answer:** We sincerely appreciate the reviewer’s suggestion for more comprehensive ablation studies. We have conducted all the requested experiments as outlined, and kindly refer the reviewer to our response to Question 4 for detailed results and analysis."}, {"review_id": "gFEhnv4mZW", "reviewer": "Reviewer_Xu6M", "summary": "The authors address two important challenges in current single-cell foundation models: (1) the neglect of the inherent tabular nature of transcriptomic data, and (2) serious privacy concerns related to centralized model training. To this end, they propose TABULA, a large-scale foundation model for single-cell transcriptomics that integrates table-aware modeling with federated learning. TABULA directly models the cell-by-gene expression matrix, capturing both inter-gene and inter-cell relationships. Overall, the work avoids simply imposing NLP paradigms (e.g., token ranking) on transcriptomics and instead leverages the structural characteristics of the data. The incorporation of federated learning is timely and relevant for safeguarding sensitive biomedical datasets.", "questions": "1. The paper claims that the model captures gene–gene relationships, but does not provide downstream analysis to support this. Prior work, such as Geneformer, evaluates the learned embeddings by analyzing similarities between transcription factor targets (both direct and indirect) and housekeeping genes. Similar evaluations could help validate gene-level representation learning [1].\n\n2. The use of raw count inputs raises concerns about batch effects caused by tissue heterogeneity and differences in sequencing platforms. How were these issues mitigated during pretraining? Was any normalization or correction applied?\n\n3. How does TABULA defend against backdoor or poisoning attacks in the federated setting? Has any robustness analysis or adversarial simulation (e.g., following [2]) been conducted?\n\n4. The scaling factor is fixed at 0.03 throughout the paper. Was this value tuned empirically? How does model performance vary with different scaling factors?\n\n5. How does TABULA differ from existing tabular foundation models such as TabPFN[3] in terms of architecture and design? Was any empirical comparison conducted to validate the design choice over directly adapting those architectures?\n\n6. During pretraining and fine-tuning, the maximum input sequence length M varies (e.g., 1200 vs. 2000). Was this choice guided by any empirical observation? Is there any recommendation for selecting M based on dataset characteristics?\n\n[1] Cen et al., “Transfer learning enables predictions in network biology.” Nature, 2023.\n\n[2] Feng et al., “Unveiling potential threats: backdoor attacks in single-cell pre‑trained models.” Cell Discovery, 2024. doi:10.1038/s41421-024-00753-1\n\n[3] Hollmann et al., “Accurate predictions on small data with a tabular foundation model.” Nature, 2025 (vol. 637, pp.319–326). doi:10.1038/s41586-024-08328-6", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The paper is generally well-written and well-organized. The authors provide comprehensive benchmarks covering both gene-level and cell-level downstream tasks, demonstrating the versatility and broad applicability of the TABULA.\n2. The model appropriately respects the tabular format of single-cell data and employs a Transformer to capture inter-gene relationships.\n3. Despite being trained on only half of the available training data, the model appears to achieve good performance across multiple tasks.\n\nWeaknesses\n1. In the federated pretraining setup, model weights are aggregated via FedAvg using client-wise weights $p_k$. The paper states that equal weighting is used during pretraining. However, in downstream applications where users contribute their own models for weight updates, it is unclear how $p_k$ is determined or whether adaptive weighting is supported. This should be explicitly clarified.\n2. The current model architecture does not incorporate attention across cells—attention is computed only over genes. This design limits the model's ability to capture inter-cellular relationships, which are critical in tasks such as trajectory inference.\n3. The strategy for selecting highly variable genes (HVGs) in the federated learning setting is under-specified. Are HVGs selected independently per client? Was any study conducted to understand how the number of HVGs (M) affects model performance?\n4. Although the paper highlights the use of federated learning to enhance privacy, it does not address potential security vulnerabilities, such as backdoors or data poisoning attacks, which are well-documented threats in federated systems.", "was_revised": true, "final_justification": "Thank you to the authors for the detailed and thoughtful response. The clarifications provided regarding the downstream task on gene-gene relationships convincingly addressed my initial concerns, significantly improving the clarity and strengthening the validity of the results. As a result of the comprehensive responses and the corresponding manuscript revisions, I find the contribution of the paper to be more substantial and well-supported. Therefore, I am updating my score to 4.", "rebuttal": "**Question 1:** The paper claims to capture gene–gene relationships but lacks downstream analysis to support this.\n\n\"Answer:\"  To demonstrate that Tabula captures gene–gene relationships, we evaluate its zero-shot performance on two tasks: (1) Pairwise regulation – identifying regulatory effects between two genes; and (2) Combinatorial regulation – assessing joint effects of multiple genes on a target.\n\nFor pairwise validation, we evaluated four developmental systems—hematopoiesis (29 known pairs), cardiogenesis (17), neurogenesis (9), and pancreatic endogenesis (22)—and compared predictions to known regulatory relationships. Results are presented in Table 1.\n\n**Table 1:** Pairwise regulation prediction accuracy from Tabula across four developmental systems.\n\n| System                 | Accuracy (%) |\n|------------------------|--------------|\n| Hematopoiesis          | 0.83         |\n| Cardiogenesis          | 0.88         |\n| Neurogenesis           | 0.89         |\n| Pancreatic Endogenesis | 0.91         |\n\nWe tested combinatorial regulation prediction on hematopoiesis and cardiogenesis, which contain 4 and 9 known combinatorial relationships, respectively. Taking CR1(8) from the cardiogenesis system as an example, its regulatory rule is defined as: FGF8 = MESP1 ∩ (FOXC1 ∪ TBX1), indicating a three-gene combinatorial regulation with 8 possible regulatory combinations. The accuracy results are shown in Table 2.\n\nAcross both pairwise and combinatorial regulation tasks, Tabula demonstrates strong **zero-shot performance** spanning four developmental biological systems. These results highlight Tabula’s ability to capture gene–gene relationships during pretraining.\n\n**Table 2:** Combinatorial regulation prediction accuracy from Tabula across two developmental systems.\n\n| System        | Combinatorial Relationship | Accuracy (%) |\n|---------------|---------------|---------------|\n| Cardiogenesis | CR1 (8)       | 100           |\n|               | CR2 (8)       | 100           |\n|               | CR3 (8)       | 87.5          |\n|               | CR4 (8)       | 12.5          |\n| Hematopoiesis | CR1 (16)      | 100           |\n|               | CR2 (16)      | 100           |\n|               | CR3 (16)      | 81.25         |\n|               | CR4 (16)      | 75            |\n|               | CR5 (8)       | 87.5          |\n|               | CR6–CR9 (4)   | 100 (each)    |\n\n\n**Question 2:** The use of raw count inputs raises concerns about batch effects caused by tissue heterogeneity and differences in sequencing platforms. How were these issues mitigated during pretraining? Was any normalization or correction applied?\n\n**Answer:** This is an excellent question. This is a fundamental challenge for single-cell foundation models.\n\nWe clarify that Tabula does not use raw count data directly. Instead, following scGPT’s binning strategy (see **Column feature value embedding**), we discretize expression values into 50 bins, computed independently per cell to reduce batch effects and platform variability.  Additionally, rather than scGPT’s random selection of 1,200 non-zero genes per cell, we use 1,200 dataset-specific HVGs, enabling more targeted and efficient learning.\n\n**Question 3:** How does TABULA defend against backdoor or poisoning attacks in the federated setting? Has any robustness analysis or adversarial simulation (e.g., following [1]) been conducted?\n\n**Answer:** Thank you for this important question on Tabula’s robustness in federated settings.\n\n**1. Backdoor Defense in Pretraining**: Tabula uses self-supervised learning on unlabeled data, which inherently reduces vulnerability to traditional backdoor attacks, as adversaries cannot manipulate labels to inject specific triggers.\n\n**2. Robustness Analysis**: Following [1], we simulated label poisoning attacks during downstream fine-tuning (note: no federated learning is used here) by introducing abnormal cell type labels. To detect anomalies, we extended the get_abnormal_cells method from [1], adding K-means clustering on abnormality scores for automatic identification. Evaluated on the human pancreas dataset [2], our method achieved 96.53 ± 0.96% recall and 99.38 ± 1.47% precision, demonstrating strong first-pass defense. We will clarify these robustness efforts in the revision.\n\nWe are committed to proactively integrating privacy and security safeguards as the ecosystem evolves and plan to conduct systematic adversarial simulations in future work to comprehensively assess Tabula’s robustness against emerging attack vectors.\n\n**Question 4:** The scaling factor is fixed at 0.03 throughout the paper. Was this value tuned empirically? How does model performance vary with different scaling factors?\n\n**Answer**: We really appreciate your thoughtful question. We fix α=0.03 so that reconstruction and contrastive losses have similar magnitudes - this stabilizes optimization. Please see our response to **Reviewer bEnN’s Question 1** for more details. \n\n**Question 5:** How does TABULA differ from existing tabular foundation models such as TabPFN[3] in terms of architecture and design? Was any empirical comparison conducted to validate the design choice over directly adapting those architectures?\n\n**Answer:** Tabula fundamentally differs from TabPFN in pretraining objective, learning setting, and applicability to biological data.\n\n**Training Objective**:\nTabula uses a dual self-supervised approach—column-wise reconstruction and row-wise contrastive learning—tailored for sparse, high-dimensional single-cell data. In contrast, TabPFN relies on supervised in-context learning with synthetic tasks, assuming fixed feature order and semantics, which do not hold in single-cell settings.\n\n**Federated Training**:\nTabula supports federated learning while TabPFN lacks any federated or privacy-preserving mechanisms.\n\nDue to these limitations, TabPFN is not adaptable for single-cell data, while Tabula's design enables biologically meaningful representation learning across both gene and cell axes.\n\n**Question 6:** During pretraining and fine-tuning, the maximum input sequence length M varies (e.g., 1200 vs. 2000). Was this choice guided by any empirical observation? Is there any recommendation for selecting M based on dataset characteristics?\n\n**Answer:** We thank the reviewer for the thoughtful question. We kindly refer the reviewer to our response to Weakness 3 for further clarification.\n\n**Weakness 1**: The paper uses equal weighting in FedAvg during pretraining, but it's unclear how model updates are handled in downstream tasks. Is adaptive weighting supported?\n\n**Answer:** Thank you for the thoughtful question. During federated pretraining, we use FedAvg with equal client-wise weighting (see **Appendix B**). In downstream tasks, federated learning is not applied—users fine-tune the pretrained tissue embedder and transformer locally using standard supervised training without any aggregation (see **Appendix E**). We will clarify this distinction in the revised manuscript.\n\n**Weakness 2**: The model computes attention only over genes, not across cells, limiting its ability to capture inter-cellular relationships important for tasks like trajectory inference.\n\n**Answer:** Thank you for the insightful comment. Like most foundation models (e.g., Geneformer, scGPT), Tabula treats genes as tokens to learn robust gene-level embeddings. While effective, we agree that modeling cell-cell relationships is important. Tabula 2.0 will incorporate spatial coordinates and potentially a spatial loss. For multi-omics pretraining, see our response to Reviewer bEnN’s Question 4. We will clarify this in the revision.\n\n**Weakness 3**:  Is HVG selection done independently per client in the federated setting? Was the impact of the number of HVGs (M) on model performance studied?\n\n**Answer:**  Thank you for raising this important question.\n\n**1. HVG Selection Strategy**: In our framework, each client handles multiple tissue-specific datasets, and we perform independent HVG selection per dataset, choosing the top 1,200 HVGs locally (see Appendix A). Unlike methods like scGPT, which randomly sample from non-zero genes, Tabula selects dataset-specific HVGs to ensure that biologically informative genes are used. This strategy improves training efficiency and preserves the unique biological signals of each dataset.\n\n**2. Choice of HVG Count (M)**: We chose 1,200 HVGs based on precedent in single-cell foundation models: scGPT uses 1,200 randomly selected genes [3], and Geneformer uses 2,048 based on expression ranking [4]. Standard scRNA-seq workflows typically use 1k–5k HVGs [5]. While larger M values could enhance model capacity, they introduce significant computational overhead and would require redesigning parts of the architecture. Our choice reflects a practical balance between biological richness and efficiency. \n\n**Weakness 4**: The paper emphasizes privacy via federated learning but does not address security risks like backdoors or data poisoning—how are these threats handled?\n\n**Answer:** Thank you for highlighting this important point. As the first work on federated pretraining for single-cell foundation models, our focus was on demonstrating feasibility. We agree that backdoor and poisoning attacks are important. The structured nature of single-cell data may pose unique vulnerabilities but also enable biologically informed defenses. We will note this in the revision and leave it for future work.\n\n[1] Feng et al., “Unveiling potential threats: backdoor attacks in single-cell pre‑trained models.” Cell Discovery, 2024. \n\n[2] Chen, Jiawei, et al. TOSICA, Nature Communications.\n\n[3]Cui, Haotian, et al. scGPT, Nature Methods \n\n[4]Theodoris, Christina V., et al. Geneformer, Nature \n\n[5]Luecken, Malte D., and Fabian J. Theis. \"Current best practices in single‐cell RNA‐seq analysis: a tutorial.\" Molecular systems biology"}]

Author rebuttal:

Meta-review: This paper presents a foundation model designed for single-cell transcriptomics with two main contributions: a self-supervised tabular learning objective and privacy-preserving pre-training for federated learning.

All the reviewers agree that the paper presents novel and practically important contributions, and gave excellent suggestions regarding improving the manuscript and additional experimental results. They raised several important concerns regarding (a) applicability to other data/tasks particularly in OOD settings, (b) specific analyses related to biological single cell data and downstream applications and (c) additional benchmarking with other methods. These concerns have been satisfactorily addressed by the authors in their rebuttal, and 3 reviewers have accordingly increased their scores to Accept (5). The final scores (confidence in parentheses) are 3 (4), 5 (3), 5 (4), 5 (4), 4 (3). 

Overall, I believe this work makes important contributions to the field of single-cell foundation models. Experimental results are convincing in supporting the claims made, especially those that were done in the rebuttal. An important point raised was regarding the writing and overstating of contributions - authors should carefully incorporate this feedback in future revisions.
Decision: Accept (poster)