Title: CellPLM: Pre-training of Cell Language Model Beyond Single Cells
Venue: NeurIPS 2023
Accepted: no
Keywords: single-cell analysis

pretrained models

AI for science
Abstract: The current state-of-the-art single-cell pre-trained models are greatly inspired by the success of large language models. They trained transformers by treating genes as tokens and cells as sentences. However, three fundamental differences between single-cell data and natural language data are overlooked: (1) scRNA-seq data are presented as bag-of-genes instead of sequences of RNAs; (2) Cell-cell relations are more intricate and important than inter-sentence relations; and (3) The quantity of single-cell data is considerably inferior to text data, and they are very noisy. In light of these characteristics, we propose a new pre-trained model \method, which takes cells as tokens and tissues as sentences. In addition, we leverage spatially-resolved transcriptomic data in pre-training to facilitate learning cell-cell relationships and introduce a Gaussian mixture prior distribution as an additional inductive bias to overcome data limitation. \method is the first single-cell pre-trained transformer that encodes cell-cell relations and it achieves state-of-the-art performance in various downstream tasks.


Abstract

The current state-of-the-art single-cell pre-trained models are greatly inspired by the success of large language models. They trained transformers by treating genes as tokens and cells as sentences. However, three fundamental differences between single-cell data and natural language data are overlooked: (1) scRNA-seq data are presented as bag-of-genes instead of sequences of RNAs; (2) Cell-cell relations are more intricate and important than inter-sentence relations; and (3) The quantity of single-cell data is considerably inferior to text data, and they are very noisy. In light of these characteristics, we propose a new pre-trained model CellPLM , which takes cells as tokens and tissues as sentences. In addition, we leverage spatially-resolved transcriptomic data in pre-training to facilitate learning cell-cell relationships and introduce a Gaussian mixture prior distribution as an additional inductive bias to overcome data limitation. CellPLM is the first single-cell pretrained transformer that encodes cell-cell relations and it achieves state-of-the-art performance in various downstream tasks.

1 Introduction

Next-generation sequencing technologies such as single-cell RNA sequencing (scRNA-seq [1]) have produced vast amounts of data, sparking a surge of interest in developing large-scale pre-trained models for single-cell analysis [2, 3, 4, 5]. These models seek to capture underlying structures and patterns from unlabeled scRNA-seq data, and can be fine-tuned on specific downstream datasets to deliver accurate predictions and nuanced insights into cellular mechanisms. Particularly, these pre-trained models have been inspired by the success of large language models, such as BERT and GPT [6, 7], and treat genes as words (tokens) and cells as sentences to train transformers [8]. However, we argue that these approaches may have limitations due to the fundamental differences between single-cell data and natural language data, which have been largely overlooked in existing literature:

First , unlike sentences, the scRNA-seq data utilized by existing pre-trained models are not sequential. Before the training stage, RNA sequences have been identified as functional units, i.e., genes. Instead of original sequences, data is denoted as a cell-by-gene count matrix that measures the abundance of individual genes within each cell. This is analogous to bag-of-words model in natural languages, where the set of genes is fixed, and there is no sequential relationship among them.

Second , the relationship between cells is remarkably more intricate and important than that of 30 sentences, since cell-cell interactions play an essential role in determining cell states and cell 31 development [9]. Additionally, within tissues, there are numerous cells from the same or similar cell 32 lineage, which grants them similar gene expression profile and hence provides valuable supplementary 33 information for denoising and identifying cell states [10, 11, 12]. As a result, many recent methods [13, 34 14, 15, 16] have constructed cell-cell graphs to advance representation learning for single-cell data. 35

Cell

gene1

gene2

gene3

gene4

gene5

ExistingSingle-cellPretrainedModels "Gene Language Model"

gene1gene2gene3

3gene1gene2gene3gene1gene2gene3gene1gene2gene3

Cell 1

Cell 2

Cell3

Cell 4

CellPLM

"Cell Language Model"

Such evidence demonstrates the importance of cell-cell relationship, which is usually neglected by 36 existing pre-trained models. 37

Third , the quantity and quality of single-cell datasets are significantly lower than those of natural 38 language data. For comparison, the high-quality filtered English dataset extracted from Common 39 Crawl corpora [17] consists of 32 billion sentences, whereas the largest collection of single-cell 40 datasets, namely the Human Cell Atlas [18], includes less than 50 million cells. To make things 41 worse, single-cell data often suffer from technical artifacts and dropout events [19, 20], as well as 42 significant batch effects between sequencing platforms and experiments [21, 22]. 43

The aforementioned differences introduce distinct challenges which call for new pre-training strategies 44 tailored for single-cell data. To bridge this gap, we propose a novel singleCell P re-trained L anguage 45 M odel ( CellPLM ), which addresses these challenges from following perspective: First , As shown 46 in Figure 1, CellPLM proposes a cell language model to account for cell-cell relations. The cell 47 embeddings are initialized by aggregating gene embeddings since gene expressions are bag-of-word 48 features. Second , CellPLM leverages a new type of data, spatially-resolved transcriptomic (SRT) 49 data, to gain an additional reference for uncovering cell-cell interactions. Compared to scRNA-seq 50 data, SRT data provide additional positional information for cells. Both types of data are jointly 51 modeled by transformers. Third , CellPLM introduces inductive bias to overcome the limitation of 52 data quantity and quality by utilizing a Gaussian mixture model as the prior distribution in the latent 53 space. This design can lead to smoother and better cell latent representations [23, 15, 24]. To the best 54 of our knowledge, the proposed CellPLM is the first pre-trained transformer framework that encodes 55 inter-cell relations, leverages spatially-resolved transcriptomic data, and adopts a reasonable prior 56 distribution. It is evident from our experiments that CellPLM demonstrates superior performance in 57 various downstream tasks. 58

59

2 Single-cell Pre-trained Models

Deep learning methods for single-cell data have garnered significant research interest in recent 60 years [11]. However, due to the distinct model architectures, the knowledge learned by models is 61 not transferable across tasks. To address this issue, there is an emerging effort [2, 3, 4, 5] from the 62 research community to explore the potential of a foundation model that first extracts latent knowledge 63 from unlabeled scRNA-seq data and subsequently generalizes this knowledge to a variety of tasks. 64

65

The first such pre-trained model for single-cell data, scBERT [2], takes genes as tokens and leverages

66

67

68

69

70

71

72

73

74

75

76

an efficient transformer [25] to encode over 16,000 gene tokens for each cell. By randomly masking a fraction of non-zero gene expression values and predicting them based on the remaining data, scBERT

effectively learns intricate relationships between genes, leading to improved cellular representation.

Later, xTrimoGene [3] made two key enhancements to scBERT: pruning zero-expressed genes and improving expression binning strategies by an auto-discretization strategy.

These modifications notably enhance scalability and feature resolutions. Another latest preprint, scGPT [5], introduces a

variant of masked language modeling that mimics the auto-regressive generation in natural language processing, where the masked genes are iteratively predicted according to model's confidence. Unlike

the aforementioned models, tGPT [4] completely abandons masked language modeling. It constructs sequences of genes based on the ranking of gene expressions within each cell, and the model is trained

to autoregressively predict the name of the next gene. Despite discarding the precise expressions,

this approach demonstrates enhanced robustness against batch effects and can be generalized to bulk 77 RNA data. 78

The aforementioned models all regard genes as tokens and focus solely on modeling gene relationships 79 within individual cells, neglecting the intercellular information in an organism. In contrast, CellPLM 80 overcomes this limitation by introducing a cell language model that extends beyond single cells. 81 Furthermore, by leveraging the spatial information of cells acquired from SRT data, along with a 82 prior Gaussian mixture distribution, the model achieves unparalleled performance on a range of 83 downstream tasks. 84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

3 Cell Language Model Beyond Single Cells

In this section, we introduce the concept of the cell language models and detailed implementation of the proposed CellPLM . As illustrated in Figure 2, CellPLM consists of four modules: a gene expression embedder, an encoder, latent space, and a decoder, which we will demonstrate in Section 3.2. At a higher level, there are two stages in our framework: pre-training and fine-tuning. During pre-training, the model is trained on unlabeled data with a masked language modeling objective. For fine-tuning, the model is first initialized with the pre-trained parameters, and then all of the parameters are fine-tuned using data and labels (if available) from the downstream datasets. We demonstrate the pre-training and fine-tuning framework in Section 3.3 and 3.3, respectively.

3.1 Cell Language Model

Due to the recent achievements of large language models [7], several studies [2, 3, 4, 5] have drawn inspiration from natural language processing in an attempt to establish a foundational model for single-cell analysis. These studies consider genes as tokens and train transformers on them, aiming to model the conditional probability between gene expressions. Concretely, previous pre-trained models are trained on scRNA-seq data, which are stored in the format of a cell-by-gene matrix X ∈ R N × k , where N is the number of cells, and k is the number of distinct gene types. The value of X i,j denotes the count of gene j observed in cell i , also known as gene expression. The pre-training goal of these models is to estimate a conditional probability distribution, which can be formulated as:

where i refers to the i -th cell and O ( i ) is the set of observed genes in cell i whose expressions are known; U ( i ) denotes the set of unobserved genes in cell i whose expression will be predicted by the model, typically referring as masked genes. If we consider genes as words, this objective is analogous to the language model in computational linguistics [26], and thus can be named a 'gene language model'. In this way, the model is trained to capture the intrinsic relations between genes, which can provide prior knowledge for downstream analysis.

However, in Eq. (1), the distribution of unobserved gene expressions only depends on genes within the same cell, while disregarding the information of other cells within the same tissue, which does not align with the inherent nature of biology. Therefore, in CellPLM , we provide a different perspective to model scRNA-seq data by treating cells as tokens:

where we denote M as the set of masked gene expressions in X , and M C is the complement, i.e., the set of unmasked expressions. The distribution of a masked entry X i,j depends on both the observed genes in cell i and genes from other cells that are not masked. We hereby name it as 'cell language model', which models the distribution of cellular features beyond single cells. By estimating the conditional probability distribution in Eq. (2), CellPLM is trained to capture the intricate relationships that exist between not only genes but also cells.

From a biology perspective, there are particularly two types of inter-cell relations that can be beneficial 119 to CellPLM . First, within tissues, there are numerous cells from the same or similar cell lineage, which 120 mutually provide valuable supplementary information for denoising and identifying cell states [10, 121 11, 12]. The other type of relations, cell-cell interactions (a.k.a, cell-cell communications), plays 122 an essential role in determining cell development and cell states [9]. Existing analysis methods [27, 123 28, 29] have already explored the cell-cell communications on the cell type or cluster levels, while 124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

Self-Supervised Pretraining

Sample A

SampleC

Sample 1

Sample 2

Sample3

Input

Gene

Sample D

Sample E

Sample F

Sample 4

Sample 5

Sample6

Expression

CellPLM

Masked

Model

Pretraining

Embedder

Sample G

Sample H

Sample1

Sample7

Sample8

Sample 9

scRNA-seqAtlas

SpatialTranscriptomicAtlas

GeneExpressionEmbedder

CellPLM Model

Gene1

Encoder

Decoder

Latent Variable Batch Label

Unmeasured

Gene2

Cell1 Cell 2 Cell3 Cell 4

Mixtureas

ExpressionEmbedding

Masked

Masked

Prior

Z

Gene3

+

Unexpressed

Gene 4

Aggregate

Embedding

Expression

Positional Embedding

draw

Fully Connected

Layer

Masked

Gene5

N x

TransformerBlock

XN

Gene6

Estimate T,μ,

p(z) =

π(N(x1μ，o1)

FeatureReconstruction

Gene7

司

CellPLM aims to capture the intricate 'language' of cell-cell communications between single cells. Overall, CellPLM presents a novel cell language model that aligns well with biological principles and holds great potentials to enhance downstream tasks by extracting valuable cellular knowledge from unlabeled single-cell data.

3.2 Model Architecture

Gene Expression Embedder . The first module in CellPLM model is a gene expression embedder, which projects input gene expressions into a low-dimensional cellular feature space. In light of the nature that scRNA-seq is profiled as bag-of-genes features, CellPLM learns an embedding vector for each type of gene, and then aggregates these gene embeddings according to their expression levels in each cell. Formally speaking, for gene j ∈ { 1 , ..., k } , a learnable embedding vector h j ∈ R d is assigned, where d is the hidden dimension of the encoder layers. h j can be either randomly initialized or initialized by prior knowledge, e.g., gene2vec [30]. The gene expression embedding matrix E ∈ R N × d is then generated by aggregating gene embeddings according to their expressions:

where E i is the i -th row vector of E , corresponding to the gene expression embedding for cell i . Note that the gene expression matrix X is a sparse matrix since the zero-rate of scRNA-seq can be up to 90% [31]. In addition, unmeasured genes (per sequencing platforms) also lead to zero entries in X . Therefore, when implementing Eq. (3), CellPLM leverages a sparse linear layer instead of a regular fully connected layer. This significantly improves memory and computational efficiency.

Transformer Encoder . The proposed CellPLM follows an encoder-decoder structure, where the encoder is based on transformers [8]. The transformer model was originally developed for processing textual data. It leverages multi-head self-attention mechanisms to capture relationships between input tokens and incorporates positional encoding to represent the token positions. In CellPLM , by considering cells as tokens, we can readily apply the transformer model to capture intercellular relationships. When applying the transformer, we consider the embedding at l -th layer H ( l ) ∈ R N × d as a set of N tokens, where N is the total number of cells in a tissue sample, and d is the hidden dimension. By stacking L transformer layers, CellPLM gradually encodes cellular and inter-cellular

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

information into cell embeddings, formulated as:

In practice, N can scale up to ten thousands, which is out of the capacity of an ordinary transformer. Therefore, we adopt an efficient variant of transformers with linear complexity (i.e., Performer [25]) for the implementation of transformer layers.

To further inform inter-cellular relations, we incorporate spatial positional information of individual cells from a novel type of data, spatially-resolved transcriptomic (SRT) data. Specifically, SRT data consist of two parts. One is a gene expression matrix X ∈ R N × k same as scRNA-seq data, and the other part is a 2D coordinate matrix C ∈ R N × 2 . The coordinates denote the center position of each cell within a field-of-view (FOV) where the cells are located (an illustration can be found in Appendix A). This feature helps locate the microenvironment surrounding each cell, providing an additional reference for identifying cell lineage and cell communications, which were introduced in Section 3.1. To encode this extra positional information, we leverage the idea of positional encodings (PE) in transformers. Since sinusoidal PE achieves competitive performance and has lower complexity on SRT data [16], we generate a 2D sinusoid PE for cells in SRT data, denoted as P ∈ R N × d , where P i is the d dimensional PE vector for cell i (see details in Appendix B). For scRNA-seq data, a randomly initialized d -dimensional vector p ′ is shared among all cells, which also results in a placeholder PE matrix P . The initial cell embeddings are now formulated as H (0) = E + P , where E is the expression embeddings from Eq. (3) and P is the positional embeddings.

Gaussian Mixture Latent Space . One of the highlights of CellPLM is the design of probabilistic latent space. Prior studies have employed variational autoencoders for single-cell analysis, which typically assumes an isotropic Gaussian distribution as the prior distribution of the latent space [32, 33]. While this approach can effectively remove batch effects, it may also result in a loss of information regarding the underlying biological structure of cell groups. To address this limitation, CellPLM incorporates the concept of Gaussian mixture variational encoder [34, 35, 15], which utilizes a mixture of Gaussians to capture the information of distinct functional groups of cells. Formally, for i ∈ { 1 , . . . , N } , the generative model of cell i can be formulated as:

where y i ∈ R L represents the one-hot latent cluster variable and π is its prior; y i,l denotes the l -th entry of y i ; µ y l ∈ R d z and σ 2 y l ∈ R d z × d z denote the mean and variance of the l -th Gaussian component, respectively; and µ z i ∈ R k and σ 2 I ∈ R k × k denote the posterior mean and variance of expression x i , respectively. In this work, we assume that σ 2 is a constant and the posterior mean is parameterized by µ z i = f dec ( z i ; θ dec ) .

To estimate the posterior of z i and y i , we parameterize the inference process with neural networks. Specifically, we assume that the cluster variables y are independent of the expression x i condition on latent variables z i . The inference model can be formulated as:

where the estimations are given by 185

Here f enc ( · ; η enc ) represents the transformer encoder, f µ ( · ; η µ ) , f σ ( · ; η σ ) and f π ( · ; η π ) are neural 186 networks. A log-evidence lower bound (ELBO) can be derived from this generative model for 187 the optimization purpose [34]. However, as mentioned in Section 3.1, our pre-training framework 188

incorporates a cell language model, where parts of the input gene expression matrix X are masked. 189 This will result in a modified objective. To formalize the problem, recall that previously we defined 190 the masked set as M . On top of that, we denote M ∈ R N × k as a mask indicator matrix such that 191

Let ˜ X ∈ R N × k be the masked gene expression matrix given by the element-wise multiplication ˜ X = M ⊙ X . The objective of cell language model with Gaussian mixture prior, i.e., a denoising variational lower bound [36], can be formulated as:

Similar to previous works [34], we refer to the three terms in Eq. (8) as reconstruction term L recon , conditional prior term L cond and Y prior term L Y . The approximation and estimation of the denoising variational lower bound are specified in Section 3.3.

Batch-aware Decoder . The decoder in CellPLM operates by decoding each cell individually, given that the tissue context has already been encoded into the latent space by the encoder. The decoder's purpose is twofold: to reconstruct masked features and to help remove batch effects from the latent space. In order to accomplish this goal, the decoder stacks several feed-forward layers (FFLayers) atop the input of latent variables z , and a batch embedding, denoted as b ∈ R d z . Specifically, for each cell, the batch embedding is loaded from a learnable lookup table as b = LookUp( b ) , where b is the label indicating the specific tissue sample (or FOV for SRT data) from which the cell has been drawn. By feeding the batch label to the decoder, a batch-effect-free latent space can be achieved, as empirically evidenced in scVI [32]. The decoder can thus be formulated as:

where l indicates the number of the layer, h ( l ) is the hidden vector of layer l ∈ (1 ..L -1) , and L is the total number of fully connected layers. The dimension of the last layer is different from the previous layers because the last layer is considered as an output layer, with h L ∈ R k , where k is the size of gene sets in the gene expression matrix X ∈ R N × k .

3.3 Model Pre-training & Fine-tuning

Pre-training. The pre-training of CellPLM follows a cell language modeling objective, as demonstrated in Eq. (8). Specifically, given a batch of cell tokens as input, we first decide which cells should be masked. Instead of completely masking these cell tokens, we selectively mask a certain percentage of the gene expressions within them. This allows the model to recover underlying correlations between cells, as proposed in a recent preprint, SpaFormer [16]. A significant concern in CellPLM is the disparity in the number of genes measured by different sequencing platforms. Notably, the gap between scRNA-seq and SRT can be substantial, ranging from 1,000 to 30,000. Taking this into consideration, CellPLM only masks the expression of genes that are measured in each dataset, implying that the reconstruction loss is calculated exclusively on these measured genes. When optimizing the denoising variational lower bound in Eq. (8), we apply reparameterization trick and Monte Calo sampling, as proposed in VAE [37]. Furthermore, under the independent Gaussian assumption, we reformulate and estimate the reconstruction term L recon in Eq. (8) with a mean squared error (MSE). Therefore, the pre-training loss function of CellPLM can be formulated as:

where ⊙ signifies element-wise multiplication, H ( L ) ∈ R N × k is the output from the decoder, X and M are the ground-truth gene expression matrix and the mask indicator matrix respectively, as defined above. L cond and L Y are derived from Eq. (8).

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

Task-specific Fine-tuning . When fine-tuning CellPLM , the model is first initialized with the pretrained parameters. In downstream tasks that require gene expressions as output, the pre-trained decoder is fine-tuned on the downstream datasets. Otherwise, the decoder will be replaced with a task-specific head. The entire model is then fine-tuned with task-specific loss functions, which helps align the general knowledge of the model to the specific downstream task. For example, in the spatial transcriptomic imputation task, the model is fine-tuned on a query SRT dataset and a reference scRNA-seq dataset, where two datasets are sampled from the same type of tissue. In this case, the loss function remains the same as Eq.(9). After fine-tuned on these datasets, CellPLM fit the data distribution of the target tissue and can readily perform imputation. The design and implementation of heads and loss functions for some downstream tasks are elucidated in Appendix E.

4 Experiment

CellPLM is first pre-trained on more than 9 Million scRNA-seq cells and 2 Million SRT cells, with the masked language modeling objective demonstrated in Section 3.3. To explore an appropriate model size, we created three different sizes of pre-trained models, with 5M, 10M and 40M parameters, respectively. All experiments were finished within 24 hours on a GPU server with 8 Nvidia Tesla v100 16GB cards. The hyperparameters, datasets, and reproduciability information for pre-trained models are detailed in Appendix D. Our preliminary results (See Appendix D) show that the 10M model achieved the best parameter efficiency. Therefore, in the downstream evaluation, we take CellPLM 10M as the base model without special mentioning.

In the following sections, we evaluate the performance of CellPLM 10M on various downstream tasks, including scRNA-seq denoising, spatial transctiptomic imputation, and perturbation prediction. With the selected tasks, we aim to answer the following research questions:

RQ1: Does CellPLM present extraordinary denoising power compared to non-pretrained models?

RQ2: Does CellPLM succeed in jointly modeling scRNA-seq and SRT data, thus benefiting from both the spatial information of SRT and the abundant transcriptomic profiles of scRNA-seq?

RQ3: Although being trained on a cell language model beyond single cells, does CellPLM also perform well on gene-level task?

4.1 Task 1: scRNA-seq Denoising

Given that single-cell RNA-Seq protocols capture only a subset of the mRNA molecules within individual cells, the resulting measurements exhibit substantial technical noise [38]. Therefore, we consider denoising power as the most desired and essential power for a single-cell foundation model. The goal of the denoising task is to estimate the true expression level of each gene in each cell from a noisy observation. To assess the denoising efficacy of CellPLM , we conduct an evaluation on two single-cell RNA-Seq datasets, i.e., PBMC 5K and Jurkat from 10x Genomics [39]. Following the setting of scGNN [13] and scGNN2.0 [40], we apply a random flipping process to a subset of non-zero entries, transforming them into zeros in order to simulate the effects of dropout. In order to establish a performance benchmark for CellPLM , we conduct a comparative analysis with contemporary approaches, including DeepImpute [41], scGNN2.0 [40], SAVER [42], DCA [43], MAGIC [44] and scImpute [45], which are considered state-of-the-art methods in the field. We evaluate scRNA-seq denoising performance based on two popular regression metrics, i.e., Root Mean Square Error (RMSE) and Mean Absolute Error (MAE), to measure the degree of similarity between predicted gene expression and the actual ones. More details pertaining to these methods, the fine-tuning of CellPLM , and the evaluation metrics under the task of scRNA-seq denoising can be found in Appendix E.1.

It is evident that the fine-tuned CellPLM consistently exhibits superior performance compared to all baseline models on both datasets. Note that even under the zero-shot setting, CellPLM shows satisfactory results that surpass five baselines on the Jurkat dataset. These observations support that our proposed CellPLM outperforms the state-of-the-art denoising techniques, which answers the question of RQ1 . This superiority can be attributed to the knowledge it acquires from unsupervised pre-training.

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

4.2 Task 2: Spatial Transcriptomic Imputation

Spatially resolved transcriptomics has revolutionized single-cell analysis by incorporating physical locations along with gene expression, leading to exciting breakthroughs. However, due to the highly detailed spatial resolution, spatial transcriptomic data at the cellular level often encounter substantial missing values, which pose challenges in data analysis. To assess the potential benefits of the pre-trained model in the given task, we evaluate CellPLM on two spatial transcriptomic datasets at single-cell resolution, i.e., Lung2 and Liver2 [46]. Following the setting of baselines including SpaGE [47], stPlus [48], gimVI [49] and Tangram [50], we impute the unseen genes of the SRT dataset utilizing a scRNA-seq dataset as reference. We identify the testing gene set in SRT data by stratified sampling according to gene sparsity [51] and holdout those genes in fine-tuning stage. To evaluate the accuracy of spatial transcriptomic imputation, we employ Root Mean Square Error (RMSE), Pearson correlation coefficient (Corr), and cosine similarity (Cosine) to measure the degree of similarity between the predicted spatial gene expressions and the corresponding ground-truth expression values.

Remarkably, the fine-tuned CellPLM takes the lead in all three metrics on both datasets. In addition, the impressive zero-shot performance indicates that CellPLM can leverage pre-training information to impute the SRT data, effectively addressing the research question RQ2 . For additional information regarding baselines, the fine-tuning of the CellPLM , and the evaluation metrics under this task, please refer to Appendix E.2.

4.3 Task 3: Perturbation Prediction

The perturb-seq technology has been established to examine the gene expression response at the singlecell level when subjected to pooled perturbations [52]. By comparing the gene expression before and after perturbation, downstream analysis of differential expression (DE) enables the identification of genes that play a crucial role in disease progression. To assess the potential benefits of CellPLM in the given task, we conduct experiments to predict the expression value of genes after perturbation. Following the setting of GEARS [53], we partition the perturbations into training, validation, and test sets, ensuring that none of the test perturbations are encountered during the optimization process. Two perturbation datasets are employed for evaluation: (1) the Adamson Perturb-Seq dataset [54], consisting of 87 one-gene perturbations; and (2) the Norman Perturb-Seq dataset [55], containing 131 two-gene perturbations and 105 one-gene perturbations. To evaluate the performance of perturbation prediction, we employ Root Mean Square Error (RMSE) to measure the degree of similarity between the predicted gene expressions and the corresponding ground-truth expression values. In addition,

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

Adam. DE

Norman.0 All

Norman.0 DE

Norman.1 All

Norman.1 DE

0.00

0.05

0.10

0.15

0.20

0.25

0.30

RMSE

CellPLM

GEARS

scGEN

0.620

CellPLM10M

0.920

0.626

w/oSRTdata

10.973

0.624

w/oTransformer

0.947

0.637

w/oGMM

0.921

0.4

0.5

0.6

0.7

0.8

0.9

1.0

SRTImputation

scRNA-seqDenoising

following previous settings in GEARS [53], we also present the RMSE calculated on the top 20 deferentially-expressed genes.

We compare the performance between CellPLM and two baselines, i.e., a recent preprint GEARS method [53], and scGen [56]. The results in Figure 3 imply that CellPLM achieves the lowest RMSE values across all settings, which successfully tackles research question RQ3 . For additional information regarding baselines, the fine-tuning of the CellPLM , and the evaluation metrics, please refer to Appendix E.3.

4.4 Ablation study

To verify the contribution of our model design, we conduct an ablation study on SRT data, Gaussian mixture prior and transformer encoder. Specifically, we remove SRT data from the pre-training dataset, replace transformer encoder with an MLP encoder and remove the Gaussian mixture prior, to examine its impact on the zero-shot performance in downstream tasks. All three models are modified based on CellPLM 10M. Our results demonstrate that, on the whole, the full 10M model exhibits the best performance, and its individual components display notable significance. Specifically, in the SRT imputation task, the GMM latent model contributes the most, while the removal of SRT data or the transformer component leads to the most substantial decrease in scRNA-seq denoising performance. The ablation study provides additional support, indicating that all elements within CellPLM offer valuable assistance in specific tasks.

5 Discussion

In this work, we propose cell language model, a novel paradigm of single-cell pre-trained model, which aligns well with the fundamental characteristics of single-cell data. This has leaded to CellPLM , the first pre-trained transformer framework that encodes inter-cell relations, leverages spatially-resolved transcriptomic data, and adopts a reasonable prior distribution. Our experiments on three downstream tasks demonstrate the power of CellPLM , which has a great potential to facilitate future research in single-cell biology.

Limitations and future directions : Despite the superior performance and results from the ablation study suggesting that our model has learned complex cell-cell relationships, extracting explicit knowledge and insights from the model remains a challenging task. Therefore, enhancing model interpretability is one foremost future objective. Moreover, due to the unavailability of implementations, we could not compare our model with existing pre-trained models. However, we intend to conduct a more comprehensive comparison in future studies.

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

References

Fuchou Tang, Catalin Barbacioru, Yangzhou Wang, Ellen Nordman, Clarence Lee, Nanlan Xu, Xiaohui Wang, John Bodeau, Brian B Tuch, Asim Siddiqui, Kaiqin Lao, and M Azim Surani. mrna-seq whole-transcriptome analysis of a single cell. Nature Methods , 6(5):377-382, 2009.

Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nature Machine Intelligence , 4(10):852-866, 2022.

Jing Gong, Minsheng Hao, Xin Zeng, Chiming Liu, Jianzhu Ma, Xingyi Cheng, Taifeng Wang, Xuegong Zhang, and Le Song. xtrimogene: An efficient and scalable representation learner for single-cell rna-seq data. bioRxiv , pages 2023-03, 2023.

Hongru Shen, Jilei Liu, Jiani Hu, Xilin Shen, Chao Zhang, Dan Wu, Mengyao Feng, Meng Yang, Yang Li, Yichen Yang, et al. Generative pretraining from large-scale transcriptomes for single-cell deciphering. iScience , 2023.

Haotian Cui, Chloe Wang, Hassaan Maan, and Bo Wang. scgpt: Towards building a foundation model for single-cell multi-omics using generative ai. bioRxiv , pages 2023-04, 2023.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Erick Armingol, Adam Officer, Olivier Harismendy, and Nathan E Lewis. Deciphering cell-cell interactions and communication from gene expression. Nature Reviews Genetics , 22(2):71-88, 2021.

Robrecht Cannoodt, Wouter Saelens, and Yvan Saeys. Computational methods for trajectory inference from single-cell transcriptomics. European journal of immunology , 46(11):2496-2506, 2016.

Dylan Molho, Jiayuan Ding, Zhaoheng Li, Hongzhi Wen, Wenzhuo Tang, Yixin Wang, Julian Venegas, Wei Jin, Renming Liu, Runze Su, et al. Deep learning in single-cell analysis. arXiv preprint arXiv:2210.12385 , 2022.

Kelly Street, Davide Risso, Russell B Fletcher, Diya Das, John Ngai, Nir Yosef, Elizabeth Purdom, and Sandrine Dudoit. Slingshot: cell lineage and pseudotime inference for single-cell transcriptomics. BMC genomics , 19:1-16, 2018.

Juexin Wang, Anjun Ma, Yuzhou Chang, Jianting Gong, Yuexu Jiang, Ren Qi, Cankun Wang, Hongjun Fu, Qin Ma, and Dong Xu. scgnn is a novel graph neural network framework for single-cell rna-seq analyses. Nature communications , 12(1):1882, 2021.

Xin Shao, Chengyu Li, Haihong Yang, Xiaoyan Lu, Jie Liao, Jingyang Qian, Kai Wang, Junyun Cheng, Penghui Yang, Huajun Chen, et al. Knowledge-graph-based cell-cell communication inference for spatially resolved transcriptomic data with spatalk. Nature Communications , 13(1):4429, 2022.

Junlin Xu, Jielin Xu, Yajie Meng, Changcheng Lu, Lijun Cai, Xiangxiang Zeng, Ruth Nussinov, and Feixiong Cheng. Graph embedding and gaussian mixture variational autoencoder network for end-to-end analysis of single-cell rna sequencing data. Cell Reports Methods , page 100382, 2023.

Hongzhi Wen, Wenzhuo Tang, Wei Jin, Jiayuan Ding, Renming Liu, Feng Shi, Yuying Xie, 388 and Jiliang Tang. Single cells are spatial tokens: Transformers for spatial transcriptomic data 389 imputation. arXiv preprint arXiv:2302.03038 , 2023. 390

391

392

393

394

395

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 4003-4012, Marseille, France, May 2020. European Language Resources Association.

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

Aviv Regev, Sarah A Teichmann, Eric S Lander, Ido Amit, Christophe Benoist, Ewan Birney, Bernd Bodenmiller, Peter Campbell, Piero Carninci, Menna Clatworthy, et al. The human cell atlas. elife , 6:e27041, 2017.

Valentine Svensson, Kedar Nath Natarajan, Lam-Ha Ly, Ricardo J Miragaia, Charlotte Labalette, Iain C Macaulay, Ana Cvejic, and Sarah A Teichmann. Power analysis of single-cell rnasequencing experiments. Nature methods , 14(4):381-387, 2017.

Peng Qiu. Embracing the dropouts in single-cell rna-seq analysis. Nature communications , 11(1):1169, 2020.

Hoa Thi Nhu Tran, Kok Siong Ang, Marion Chevrier, Xiaomeng Zhang, Nicole Yee Shin Lee, Michelle Goh, and Jinmiao Chen. A benchmark of batch-effect correction methods for single-cell rna sequencing data. Genome biology , 21:1-32, 2020.

Ricard Argelaguet, Anna SE Cuomo, Oliver Stegle, and John C Marioni. Computational principles and challenges in single-cell data integration. Nature biotechnology , 39(10):12021215, 2021.

Christopher Heje Grønbech, Maximillian Fornitz Vording, Pascal N Timshel, Casper Kaae Sønderby, Tune H Pers, and Ole Winther. scvae: variational auto-encoders for single-cell gene expression data. Bioinformatics , 36(16):4415-4422, 2020.

Jing Jiang, Junlin Xu, Yuansheng Liu, Bosheng Song, Xiulan Guo, Xiangxiang Zeng, and Quan Zou. Dimensionality reduction and visualization of single-cell rna-seq data with an improved deep variational autoencoder. Briefings in Bioinformatics , page bbad152, 2023.

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 , 2020.

Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems , 13, 2000.

Rui Hou, Elena Denisenko, Huan Ting Ong, Jordan A Ramilowski, and Alistair RR Forrest. Predicting cell-to-cell communication networks using natmi. Nature communications , 11(1):5011, 2020.

S Jin, CF Guerrero-Juarez, L Zhang, I Chang, R Ramos, CH Kuan, P Myung, MV Plikus, and Q Nie. Inference and analysis of cell-cell communication using cellchat. nat. commun. 12, 1088, 2021.

Micha Sam Brickman Raredon, Taylor Sterling Adams, Yasir Suhail, Jonas Christian Schupp, Sergio Poli, Nir Neumark, Katherine L Leiby, Allison Marie Greaney, Yifan Yuan, Corey Horien, et al. Single-cell connectomic analysis of adult mammalian lungs. Science advances , 5(12):eaaw3851, 2019.

Jingcheng Du, Peilin Jia, Yulin Dai, Cui Tao, Zhongming Zhao, and Degui Zhi. Gene2vec: distributed representation of genes based on co-expression. BMC genomics , 20:7-15, 2019.

Ruochen Jiang, Tianyi Sun, Dongyuan Song, and Jingyi Jessica Li. Statistics or biology: the zero-inflation controversy about scrna-seq data. Genome biology , 23(1):1-24, 2022.

Romain Lopez, Jeffrey Regier, Michael B Cole, Michael I Jordan, and Nir Yosef. Deep 435 generative modeling for single-cell transcriptomics. Nature methods , 15(12):1053-1058, 2018. 436

437

438

439

Chenling Xu, Romain Lopez, Edouard Mehlman, Jeffrey Regier, Michael I Jordan, and Nir Yosef. Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models. Molecular systems biology , 17(1):e9620, 2021.

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648 , 2016.

Linxiao Yang, Ngai-Man Cheung, Jiaying Li, and Jun Fang. Deep clustering by gaussian mixture variational autoencoders with graph embedding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6440-6449, 2019.

Daniel Im Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising criterion for variational auto-encoding framework. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

Dominic Grün, Lennart Kester, and Alexander Van Oudenaarden. Validation of noise models for single-cell transcriptomics. Nature methods , 11(6):637-640, 2014.

10x genomics datasets. https://support.10xgenomics.com/ single-cellgene-expression/datasets .

Haocheng Gu, Hao Cheng, Anjun Ma, Yang Li, Juexin Wang, Dong Xu, and Qin Ma. scgnn 2.0: a graph neural network tool for imputation and clustering of single-cell rna-seq data. Bioinformatics , 38(23):5322-5325, 2022.

Cédric Arisdakessian, Olivier Poirion, Breck Yunits, Xun Zhu, and Lana X Garmire. Deepimpute: an accurate, fast, and scalable deep neural network method to impute single-cell rna-seq data. Genome biology , 20(1):1-14, 2019.

Mo Huang, Jingshu Wang, Eduardo Torre, Hannah Dueck, Sydney Shaffer, Roberto Bonasio, 462 John I Murray, Arjun Raj, Mingyao Li, and Nancy R Zhang. Saver: gene expression recovery 463 for single-cell rna sequencing. Nature methods , 15(7):539-542, 2018. 464

465

466

467

Gökcen Eraslan, Lukas M Simon, Maria Mircea, Nikola S Mueller, and Fabian J Theis. Singlecell rna-seq denoising using a deep count autoencoder. Nature communications , 10(1):390, 2019.

468

469

470

471

David Van Dijk, Roshan Sharma, Juozas Nainys, Kristina Yim, Pooja Kathail, Ambrose J Carr, Cassandra Burdziak, Kevin R Moon, Christine L Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion. Cell , 174(3):716-729, 2018.

Wei Vivian Li and Jingyi Jessica Li. An accurate and robust imputation method scimpute for 472 single-cell rna-seq data. Nature communications , 9(1):997, 2018. 473

Merscope ffpe human immuno-oncology datasets. https://info.vizgen.com/ 474 ffpe-showcase?submissionGuid=88ba0a44-26e2-47a2-8ee4-9118b9811fbf . 475

Tamim Abdelaal, Soufiane Mourragui, Ahmed Mahfouz, and Marcel JT Reinders. Spage: 476 spatial gene enhancement using scrna-seq. Nucleic acids research , 48(18):e107-e107, 2020. 477

Chen Shengquan, Zhang Boheng, Chen Xiaoyang, Zhang Xuegong, and Jiang Rui. stplus: a 478 reference-based method for the accurate enhancement of spatial transcriptomics. Bioinformatics , 479 37(Supplement_1):i299-i307, 2021. 480

Romain Lopez, Achille Nazaret, Maxime Langevin, Jules Samaran, Jeffrey Regier, Michael I 481 Jordan, and Nir Yosef. A joint model of unpaired data from scrna-seq and spatial transcriptomics 482 for imputing missing gene expression measurements. arXiv preprint arXiv:1905.02269 , 2019. 483

484

485

486

487

Tommaso Biancalani, Gabriele Scalia, Lorenzo Buffoni, Raghav Avasthi, Ziqing Lu, Aman Sanger, Neriman Tokcan, Charles R Vanderburg, Åsa Segerstolpe, Meng Zhang, et al. Deep learning and alignment of spatially resolved single-cell transcriptomes with tangram. Nature methods , 18(11):1352-1362, 2021.

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

Gülben Av¸ sar and Pınar Pir. A comparative performance evaluation of imputation methods in spatially resolved transcriptomics data. Molecular Omics , 2023.

Atray Dixit, Oren Parnas, Biyu Li, Jenny Chen, Charles P Fulco, Livnat Jerby-Arnon, Nemanja D Marjanovic, Danielle Dionne, Tyler Burks, Raktima Raychowdhury, et al. Perturb-seq: dissecting molecular circuits with scalable single-cell rna profiling of pooled genetic screens. cell , 167(7):1853-1866, 2016.

Yusuf Roohani, Kexin Huang, and Jure Leskovec. Gears: Predicting transcriptional outcomes of novel multi-gene perturbations. BioRxiv , pages 2022-07, 2022.

Britt Adamson, Thomas M Norman, Marco Jost, Min Y Cho, James K Nuñez, Yuwen Chen, Jacqueline E Villalta, Luke A Gilbert, Max A Horlbeck, Marco Y Hein, et al. A multiplexed single-cell crispr screening platform enables systematic dissection of the unfolded protein response. Cell , 167(7):1867-1882, 2016.

Thomas M Norman, Max A Horlbeck, Joseph M Replogle, Alex Y Ge, Albert Xu, Marco Jost, Luke A Gilbert, and Jonathan S Weissman. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science , 365(6455):786-793, 2019.

Mohammad Lotfollahi, F Alexander Wolf, and Fabian J Theis. scgen predicts single-cell perturbation responses. Nature methods , 16(8):715-721, 2019.

Grace XY Zheng, Jessica M Terry, Phillip Belgrader, Paul Ryvkin, Zachary W Bent, Ryan Wilson, Solongo B Ziraldo, Tobias D Wheeler, Geoff P McDermott, Junjie Zhu, et al. Massively parallel digital transcriptional profiling of single cells. Nature communications , 8(1):14049, 2017.

Feiyang Ma and Matteo Pellegrini. Actinn: automated identification of cell types in single cell rna sequencing. Bioinformatics , 36(2):533-538, 2020.

C Domínguez Conde, C Xu, LB Jarvis, DB Rainbow, SB Wells, T Gomes, SK Howlett, O Suchanek, K Polanski, HW King, et al. Cross-tissue immune cell analysis reveals tissuespecific features in humans. Science , 376(6594):eabl5197, 2022.

Yuqi Tan and Patrick Cahan. Singlecellnet: a computational tool to classify single cell rna-seq data across platforms and across species. Cell systems , 9(2):207-213, 2019.

Jiawei Chen, Hao Xu, Wanyu Tao, Zhaoxiong Chen, Yuxuan Zhao, and Jing-Dong J Han. Transformer for one stop interpretable cell type annotation. Nature Communications , 14(1):223, 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new pre-training strategy that takes cells as tokens and tissues as sentences. It also encodes cell-cell relations by leveraging the spatial information of cells acquired from spatially-resolved transcriptomic data. The proposed method achieves state-of-the-art performance in various downstream tasks.\n\nSTRENGTHS:\n- It's a good insight and more biologically meaningful to take cells as tokens and tissues as sentences instead of using genes as tokens and cells as sentences, as there's no sequential relationship among the order of genes.\n- It's also an interesting idea to utilize SRT data to encode spatial information as the input to the transformer.\n- Good reproducibility: the authors have provided code. \n- The paper is overall well-written and conveys the idea clearly.\n\nWEAKNESSES:\n- Though the model is pretrained, full finetuning is required when deploying to the downstream tasks. This is somehow in contrast to the idea of training a pre-trained model that can be easily adapted to downstream tasks by only tuning the task-specific layer. The drawback is 1) it requires a lot of computing resources and data, and 2) we don't know if the performance (i.e. representation learning ability) is obtained from the pretraining or just fine-tuning. \n- Following the above problem, the author should compare the performance w/ and w/o pretraining, and also the performance of fine-tuning the task-specific layer, to ablate whether the pretraining helps.\n- The authors evaluated tasks like denoising, imputation, and perturbation prediction, but why not the most common and important task like cell-type classification? I think it's the most straightforward way to benchmark the performance of the proposed method.\n\nQUESTIONS:\n- The authors used a gene expression embedder to encode the gene expression. I'm curious will the aggregation at the end make the gene expression indistinguishable. E.g., is CellPLM able to perform downstream tasks such as highly variable gene detection that require specific gene information?\n- The embedding layer of the transformer encoder has a shape of $N \\times d$, where N is the total number of cells in a tissue sample. If the pretraining datasets contain several different datasets that have different numbers of cells, how can the embedding layer adapt to them? The similar problem also applies to the gene embedder, how to adapt the embedder to different numbers of genes with different sequencing techniques of different datasets?\n- Does the downstream fine-tuning also require paired SRT data? What if the target single-cell dataset does have corresponding SRT data - is there any way to solve this?\n\nLIMITATIONS:\nOverall I find the idea interesting, but the descriptions of the method are not very clear and the experimental evaluations are not strong enough to support the claim.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a pre-trained model based on single-cell data motivated by the special characteristics (i.e., bag of genes structure, cell-cell relation, noisy) in cell-data, which viewed cells as tokens and tissues as sentences. This is different from the most existing pre-trained models treat genes as tokens and cells as sentences. The paper designed a new training paradigm based on transformer while introducing Gaussian mixture prior to handle data limitation.\n\nExperiments over some representative down-stream tasks are conducted by fine-tuning the pretrained models to demonstrate the effectiveness of the proposed training paradigm.\n\nSTRENGTHS:\n1.pre-trained model over cell-data which considers the cell-to-cell relation and bag-of-genes structure and introduces Gaussian mixture prior distributions to handle noisy and limited data.\n\n2.Extensive experiments were conducted over different down-stream tasks (including both cell-level and gene-level tasks) to demonstrate the effectiveness of the proposed pre-trained models.\n\nWEAKNESSES:\nThe training procedure is inspired from masked language model, however, as stated in this paper, cell/tissues are different from text sequence, the technical details of pre-trained data process are not clear enough. This is very important due to that it is helpful to a) understand the characteristics of cell-data and b) repeat the experiments and utilize more related data to further enhance the performance of pretrained models.\n\nPlease refer to questions for details.\n\nQUESTIONS:\n1.How to perform pre-train over two different data sources? Such as mixture of two data (how to batch) or training over these two data in sequential manner? And how many epochs and what is the batch size during pre-training? Or how to determine the pretraining procedure is completed?\n2.In mask language model, how to perform random mask and the mask ratio are quite important. There are two mask rates, cell mask rate and gene mask rate. How to mask the cell-data with these two rates. There are two cell-data sources with different characteristics. Is the mask schema sensitive to different data sources?\n3.The mask is performed offline (that is, generating the random corrupted cells offline) or online (performing random corrupt during training)?\n4.How to determine the number of Gaussian distributions or number of clusters in GMM? Is this parameter related to the number of genes in each cell? Or sensitive analysis of such hyper-parameter?\n5.In gene expression embedder, if we make use of gene2vec to initialize the gene embedding, do we suffer from OOV problem? That is, whether we encounter some genes not appear in training phase? Is gene2vec is better than random initialization? If we use random initialization, which kind of random initialization we used? This is important for transformer based pretraining.\n6.For scRNA-seq, a random position embedding is shared for all data. Is this assumption reasonable? What is the motivation comparing to SRT with coordinate matrix?\n7.How to perform zero-shot learning over different down-stream tasks?\n8.In task 3, the variance of CellPLM seems higher than the other baselines, is this due to the fact the limitation of data?\n\nTypos: \n\n1.Line 207: where l indicates the number of the layer, --> l indicates the index of the layer\n2.Line 93: fine-tuning framework in Section 3.3 and 3.3\n\nLIMITATIONS:\nThe setting of pre-training should be clear stated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a scRNA-seq model that takes into account correlations between gene expression values in nearby cells in a tissue. The proposed model represents cells as tokens with a very simple embedding model. The tissue representation is modelled using self-attention between cells. The model is trained with a probabilistic objective where the transformer outputs represent the latents with a mixture distribution and then an MLP decoder is used to decode the expression values for genes in the cell from the latent and a batch encoding.\n\nThe model is pretrained using 9M cells and 2M spatially resolved transcriptomics cells. The model is tested on 3 tasks scRNA denoising, spatial transcriptomic imputation and perturbation prediction.\n\nSTRENGTHS:\n* Understanding cell-cell communication is a very interesting open problem.\n* Task 2 zero-shot performance is on par with finetuning which seems promising.\n\nWEAKNESSES:\n* I have doubts about the scRNA denoising results and how meaningful the comparison to single cell imputation methods. (see question 1 below).\n\nQUESTIONS:\n1) For the denoising experiments are you using the transformer attention with noisy cells ? Because that seems hardly fair to the other methods. How good is just averaging the noisy cells ?\n2) Is it the claim with tasks 1 and 3 that there is more information in cross attention to other cells than between the genes ? \n3) How did you make sure there is no leakage in task 2 ? What is the data you are pre-training on ?\n4) GEARS uses the most differentially expressed genes to measure the perturbation effect. That is more robust to noise in the other genes than a global average. Have you compared using their experimental protocol ?\n5) Is X in (3) really the count matrix ? I find that very surprising because of the dynamic range should make it hard both as an input feature but even more so in the L_mse.\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a study on the problem of enhancing cell representation learning through pretraining. Inspired by the recent success of pretrained language models, the researchers explore the use of Transformer architecture to achieve improved cell representations.\n\nIn contrast to previous approaches that only consider modeling a single cell, the authors propose a novel model called CellPLM that takes into account a sequence of cells, enabling the capturing of inter-cell relationships more effectively.\n\nTo incorporate spatial relationships into the model, 2D position embeddings are introduced to enhance the basic Transformer architecture. Additionally, a mixture of Gaussian distribution is employed to model the latent embedding space, enabling the capturing of cell group information and mitigating batch effects.\n\nThe proposed CellPLM model is first pretrained using a combination of scRNA-seq cells and SRT cells. Subsequently, three downstream tasks are conducted to evaluate the effectiveness of the pretrained model.\n\nSTRENGTHS:\nThe representations and foundation models of genes and cells proposed in this study have the potential to be highly interesting and useful in advancing AI applications in the field of science (AI for Science).\n\nThe model architecture and adaptations made from the vanilla Transformer model are sensible and allow for efficient modeling of cells. In particular, the joint modeling of cell groups convincingly addresses the issue of data sparsity at the gene level.\n\nThe empirical design chosen to validate the pretrained CellPLM using three different use cases is reasonable and provides strong evidence of its effectiveness. Whether in a zero-shot or fine-tuned setting, the proposed CellPLM consistently performs well. Additionally, the results of ablation experiments further demonstrate the significance of each component designed in the model.\n\nWEAKNESSES:\nOne weakness of this paper is that most of the datasets and tasks considered in the experiments are focused on single-cell analysis. Although the paper introduces a novel aspect by jointly modeling cell groups and sequences, it would be valuable to observe how the proposed model performs on tasks that involve multi-cell information.\n\nAdditionally, it is commendable that the author provides both the mean and standard deviation of the runs. However, to further support the significance of the improvements, it would be more informative to conduct statistical significance tests on the small downstream datasets, in order to validate whether the observed improvements are statistically significant.\n\nQUESTIONS:\nRelated to weakness:\n1) Are there other datasets requiring multi-cell modeling?\n2) Are all those results statistically significant?\n\nOthers:\n1) Position embeddings: \n    i) It seems that there is no 2D position info for scRNA-seq cells. What position embedding values are used for those cases? \n    ii) Is absolute position the most sensible choice for SRT data? In other words, are those cell positions globally meaningful? Do you think relative position encodings are better alternatives?\n2) Are there other evaluations that are targeted at rare genes? As mentioned in line 220, it is possible that those frequence genes/cells are represented better compared with those rare ones?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe CellPLM paper presents a novel pre-trained model that encodes cell-cell relationships in spatially-resolved transcriptomic data. The authors demonstrate that their model outperforms existing state-of-the-art methods in various downstream tasks, including cell type classification, cell-cell interaction prediction, and spatial gene expression imputation. \n\nThe paper is well-written and provides a clear motivation for the need to develop a pre-trained model that can capture cell-cell relationships in spatially-resolved transcriptomic data. The authors also provide a detailed description of the architecture and training procedure of their model, which is helpful for readers who are interested in implementing the model in their own research.\n\nSTRENGTHS:\n1. In terms of originality, the CellPLM paper presents a new approach for encoding cell-cell relationships in spatially-resolved transcriptomic data. The authors leverage a pre-trained transformer framework that encodes inter-cell relations and adopts a reasonable prior distribution. \n2. In terms of clarity, the CellPLM paper is well-organized and easy to follow. The authors provide a clear motivation for the need to develop a pre-trained model that can capture cell-cell relationships in spatially-resolved transcriptomic data and provide a detailed description of the model's architecture and training procedure. The authors also provide a clear evaluation of the model's performance on various downstream tasks, which is helpful for readers who are interested in implementing the model in their research.\n\nWEAKNESSES:\n1. It is unclear to me why to use a Gaussian Mixture Latent Space to capture the information of distinct functional groups of cells. First, I do not understand the distinct functional groups of cells, which have not been explained clearly. Moreover, it seems that you can directly decode the masked cells like BERT without complicated latent space estimation. From Figure 4, the ablation study show that removing this part almost does not hurt the performance. \n2. Not the weakness (just suggestion), please use a clearer figure or vectorgraph (e.g., Figure 2, 4).\n\nQUESTIONS:\n1. Why to a Gaussian Mixture Latent Space capture the information of distinct functional groups of cells?\n\nLIMITATIONS:\nAs mentioned in the discussion, the work does not compare with other pre-trained models, which may be very important.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}