Single-Cell Capsule Attention : an interpretable method of cell type classification for single-cell RNA-sequencing data
Authors: Tianxu Wang
Xiuli Ma
Venue: ICLR 2022 Submitted (2022)
Keywords: Single-Cell RNA-sequencing
cell type classification
capsule network
attention
interpretable model

Abstract: Single-cell RNA-sequencing technique can obtain genes’ expression level of every cell. Cell type classification (also known as cell type annotation) on single-cell RNA-seq data helps to explore cellular heterogeneity and diversity. Previous methods for cell type classification are either based on statistical hypotheses of gene expression or deep neural networks. However, the hypotheses may not reflect the true expression level. Deep neural networks lack interpretation for the result. Here we present an interpretable neural-network based method single-cell capsule attention(scCA) which assigns cells to different cell types based on their different feature patterns. In our model, we first generate capsules which extract different features of the cells. Then we obtain compound features which combine a set of features’ information through a LSTM model. In the end, we train attention weights and apply them to the compound features. scCA provides a strong interpretation for cell type classification result. Cells from the same cell type share a similar pattern of capsules’ relationship and similar distribution of attention weights for compound features. Compared with previous methods for cell type classification on nine datasets, scCA shows high accuracy on all datasets with robustness and reliable interpretation.

Under review as a conference paper at ICLR 2022
SINGLE-CELL CAPSULE ATTENTION AN INTER- PRETABLE METHOD OF CELL TYPE CLASSIFICATION FOR SINGLE-CELL RNA-SEQUENCING DATA
Anonymous authors Paper under doublc-blind review
ABSTRACT
Single-cell RNA-sequencing technique can obtain genes' expression level of ev- ery cell. Cell type classification (also known cell type annotation) single- cell RNA-seq data helps to explore cellular heterogeneity and diversity. Previous methods for cell type classification gene expression or deep neural networks. However, the hypotheses may not re- flect the true expression level. Deep neural networks lack interpretation for the result. Here we present an interpretable neural-network based method single-cell capsule attention(scCA) which assigns cells to different cell types based on their different feature patterns. In our model, we first generate capsules which extract different features of the cells. Then we obtain compound features which com- bine set of features' information through LSTM model. In the end, we train attention weights and apply them to the compound features. scCA provides a strong interpretation for cell type classification result. Cells from the same cell type share similar pattern of capsules' relationship and similar distribution of attention weights for compound features. Compared with previous methods for cell type classification on nine datasets, scCA shows high accuracy on all datasets with robustness and reliable interpretation.
either based statistical hypotheses of
INTRODUCTION
Genes' expression on celluar level provides lots of information for us to explore homogeneity and heterogeneity among different cells(Liang et al. 2014 Muraro et al., 2016 |aaron al., 2016). Traditional bulk sequencing technique can only measure the average gene expression level of all cells in a sample. Compared with bulk data, single-cell RNA Serusuuuunniingssngleeeeom technique gives a more accurate measurement of every cell. Though single-cell RNA-sequencing is a rising technique, is not mature enough and still has some limitations. For example, single-cell RNA-seq data has high 'dropout'. 'Dropout' means zero low read counts in the data because many genes' expression are hard to detect(Huang et al. 2018 Pierson & Yau 2015). Also, single- cell RNA-seq data of high dimension because thousands of genes are in the transcriptome. These all bring challenges for studies on single-cell RNA-seq data. So far, many data mining and analysis methods have been applied on single-cell RNA-seq data solve these challenges and explore more information on cells and
RN)-seq)
et al. 2018 Han et al. 2018: Lopez et al. 2018 Hwang
al. 2018 Guo et al. 2015).
Cell type classification is one of the most important tasks in single-cell RNA-seq data analysis. helps to identify different cell types and explore the cellular heterogeneity. Previous methods on cell type classification can be categorized into methods based on statistical distribution hypotheses of gene expression and deep learning methods based on neural AAbelaa et al. (209)). These methods lay solid foundation for cell type classification. However, they still have limitations and weaknesses. Methods based on statistical hypotheses rely on prior knowledge of marker genes or gene expression's 's distribution assumptions. Up to now, there is no widely recognized hypotheses of gene expression. Deep neural (DNN), a fast and effective learning method, is widely used in various fields. DNN does not rely distribution hypotheses, however, it's hard to provide an reasonable explanation for the result behind the deep "black-box" (Lin et al.) 2017). ACTINN & Pellegrini 2020), deep neural network based method for cell type classification, uses three hidden

---

Under review as a conference paper at ICLR 2022
layers in its architecture. Though its model can predict cell types, the model still lacks reliable interpretations for the result. The challenge is that DNN does not provide a recognizable pattern that is closely related to different cell types.
Here, we introduce a new neural network based method scCA for cell type classification. scCA provides an reasonable interpretation for its classification result. Our model learns features from cells and provides patterns behind these features that are related to cell types. The first part is feature extraction. Given the gene expression level in every cell, We use capsules to extract different features. A capsule is vector that represents a feature, which captures a certain type of information of the cell. Capsules also serve as dimension reduction for single cell RNA-seq data. We calculate the Pearson coefficients between every two capsules and generate the coefficient heatmap of every cell. We discover that cells from same cell type share similar pattern in the heatmap while cells from different cell types have different patterns. These capsules capture features that are important for cell type classification.
And then, we use sequence to sequence model, bidirectional-LSTM, to generate compound fea- tures. Each compound feature combines a set of features' information. Furthermore, we train multi- ple attention weights for these compound features. Then we multiply these weights to the compound features and generate the classification result. Attention weights help to lay more emphasis on im- portant compound features. The larger the attention weight is, the more crucial its corresponding compound feature is. we also draw heatmaps for every cells' attention weights. We discover that cells from same cell type have similar patterns in the heatmaps of attention weights. This leads to better classification performance and also provides an reliable interpretation behind the network.
We evaluate scCA on several datasets and compare with previous work. Our method shows high accuracy and robustness on all datasets.
The main contributions of our article are as follow: We propose an interpretable neural network based method scCA for cell type classification. Through heatmaps of capsules' Pearson coef- ficients and attention weights, we find that cells from same cell type share a similar pattern. scCA achieves high accuracy and stable performances on all datasets.
RELATED WORK
Statistical Hypotheses Based Model
Many machine learning methods with statistical hypotheses of gene expression level have been proposed for cell type classification. Some of these methods suggest distribution hypotheses of gene expression level and employ machine learning models based on these hypotheses. Some build a hierarchy with prior knowledge and statistical hypotheses. Moana takes advantage of KNN and SVM model (Wagner & Yanai, 2018). It first preprocesses data with KNN, then it uses dimension reduction method to extract useful features, last it uses Support Vector Machine classifier. In addition, Moana uses marker gene as a prior knowledge for cell type identification. Different from Moana, Garnett proposes model based tree hierarchy. It discovers the relation between cell types and subtypes with tree structure dtructurePPineer et al. 2019). With the help of marker gene, it builds linear model classify cell types. CellAssign builds a probabilistic model based on prior knowledge to classify cell types(Zhang et al., 2019b). SCINA makes use of EM algorithm and the information of marker gene to accomplish clustering as well as cell type identification task Zasnghang et al., 2019a). scPred extracts features using SVD, then with a nonlinear kernel SVM, it predicts different cell Aalluicira-Hernannie
et al.] (201)). However, these methods have their limitations. Their hypotheses such as ZiNorZZero-ntflated Negative Binomial) NB distribution of gene expression are not the exact reflections of gene 's expression levels in cells. Actually, there is not a consensus statistical distribution for single-cell RNA-seg data. Also they need prior knowledge like marker gene for cell type classification.
Neural Network Based Models
Deep neural netN) based methods have been applied to cell type classification task. They fast and robust. Neural networks don't heavily rely on prior knowledge and statistical hypothe- ses. ACTINN uses three fully connected layers and ReLU, softmax activation function to classify different cell Mypes(Ma & Pellegrini 2020). seCapsNet builds a capsule network extract features
2

---

Under review as a conference paper at ICLR 2022
from the data and use dynamic routings among capsules to get classification result Wang al, 2020). However, neural networks lack interpretation for their result. Through training the whole network by propagation, it's hard to explain every component's contribution for the result.
PRELIMINARIES
Capsule Network and Dynamic Routing
Hinton, Sabour and other outhos first proposed Capsule network and dynamic routing in com- puter (Saour et al., 2017). A capsule is a set of neurons or a vector that can represent an entity. Dynamic routing is the mechanism between capsule's layers. As they point out, the capsules in the layer above are parents and the capsules in the layer below are Salldren.(Hintoon et al., 2000) 2011). Dynamic routing helps to send the results from capsules in children layer to its appropriate parents in the above layer. By adjusting the coupling coefficients between capsules in two layers, the network learns which parent capsule is more important for the children capsule. Here in scCA, we use capsules to capture different features of the cell. We use a vector for implementation of capsule in model. However, different from dynamic routing, we use LSTM and attention the architecture for cell type classification.
Long Short-Term Memory and Attention
Long Short-Term Memory(LSTMM, is a model widely used in Natural Language Process- Hnochreee & Schmidhuber 1997). It is a sequence to sequence model. The inputs are usually the word rrerrddtnntrrressttttttooss of sentence and the output is sequence of repre- sentations. Each output representation not only contains information from its corresponding input, but also information from representations in the context, especially its neighbor inputs.
In our method, after extracting different features into capsules, we input these capsules into the LSTM model and obtain the output sequence. Every vector in the output sequence contains informa- tion of its corresponding input feature as well other features. We name these vectors compound features. Some of the compound features have closer relationships with certain cell types. Inspired by Attention Attention(Vaswani Vaswani et al. 2017) mechanism, we apply attention weights to compound features in order to differentiate and emphasize those important compound features for classification.
SINGLE-CELL CAPSULE ATTENTION
In this section, we provide a detailed description of single-cell Capsule Attention(scAA). We will also give vivid and transparent explanation for our classification result. scCA can be divided into three parts. The figure below shows its architecture.
Feature
Compound
Extraction
Feature
Synthesis
Cene Expression Matrix
bo.g
Classification resuit
D
+
00.
Multi
Attention
Weights
Figure scCA framework
3

---

Under review as a conference paper at ICLR 2022
4.1 FEATURE EXTRACTION
The first part of scCA feature extraction. Given thousands of genes' expression levels for a cell, we hope learn simple but efficacious features for classification. Capsules not only dimension reduction, but also extract type of information of the cell. Through fully connected layers, we generate capsules to extract different features.
(1) In this formula, represents the number of capsules. W and b are the weight matrices of fully connected layers. ReLU is the activation function we use.
capsule, = Re.LRWW'' x + b [1,2...m.]
Every capsule contains a feature of cell. We hope to discover different patterns of these features for different cell types. We calculate the Pearson coefficients between every two capsules. The Pearson coefficient is an indication of relationship between two features behind capsules. For every cell, we come to a matrix where every element a Pearson coefficient of two capsules. We then draw heatmap of the matrix. We discover that cells from same cell type share similar pattern while cells from different cell types differ from each other.
GSE72056 capsule relationship cell type
GSE72056 capsule relationship cell type
Figure GSE72056 capsule relationship heatmap
Taking six cells from dataset GSE72056 examples, have a similar pattern of capsules' relationship. The deeper the color is in the heatmap, the closer the relationship of two capsules is. The pattern is distinguishable among different cell types. From the figures above, we conclude that there exists some particular feature patterns for different cell types. Our feature extraction really extracts useful and discriminative features.
cells from same cell type
4.2 LSTM FOR COMPOUND FEATURES
Single feature is not enough for cell type classification because cells from different cell types may share same features. It's the specific combination of features that decides the cell type. Moreover, some features may have special relationships with other features. We need explore the special patterns behind the features that determine cell types. The motivation for compound features lays in here. A LSTM model captures other features' information in the context. The capsules generated in feature extraction serve as the input of LSTM. For every feature in the sequence, a LSTM model takes in hidden information from other features well as its information. Then L.TTM model
4

---

Under review as a conference paper at ICLR 2022
outputs sequence of compound features. For implementation, bi-directional LSTM model for training which more stable and efficient.
Comppoundfeatuure compound.feature22.. L [appMaaplllll,, capsule2,
(2)
4.3 ATTENTION MECHANISM
In the last part of scCA, scCA learns attention weights for these compound features. We have two intentions for these attention weights. The first intention is that we want to eliminate the effect of capsule's order in the input sequence. LSTM model is widely used in natural language processing where input data is a sequence. However, for cell type classification, capsules or features' order in the sequence shouldn's influence the result. So add attention to emphasize the compound fea- tures' information itself, rather than its position information in the sequence. The second intention that some compound features may be more significant in cell type classification. The attention weights for these compound features will be large, showing their importance. In ScCA, we imple- ment
and apply them to these compound features. We generate several combinations of compound features based on different attention weights. Then we use the norms of these combi- nations to serve as the final classification result.
oum-of-feature- i [1,2...m)
attention_weight; (W _aattention' fewn.of.feauuree--batttettte [1,2 cell.type.
(4)
classi result = L2 norm(attention weeetteenttoouuiuitttttt compound
(5)
For implementation, first add all capsules together. Then generate attention weights. In formula(3), m represents the number of capsules. In formula(4), cype tatpetyppmmmmm represents the number of cell types the dataset. We multiply the compound features which are generated by bi-directional LSTM by the attention weights. Then we use the L. norm get classification result. Adding attention weights in the model structure lays more emphasis on important compound features. To provide more illustration for this, We also draw heatmaps of attention weight matrix for every cell. We find that cells from same cell type share a similar pattern of heatmap, which means certain cell type has their particular attention combinations of compound features.
fully connected layers
GSE70580 attention weights cell type
GSE70580 attention weights call type
Figure 3: GSE70580 attention weights heatmap
5

---

Under review as a conference paper at ICLR 2022
The figure above shows 6 cells from dataset GSE70580. Three cells come from cell type while the other come from cell type The deeper the color is in the figure, the larger the attention weight is. We can find that attention weights' distribution vary from one cell type to another. Different cell types have different attention weight combinations of compound features. This provide reliable interpretation for model.
EXPERIMENTS
5.1 INTRODUCTION OF DATASETS
Every dataset for single-cell RN RNA-sequencing consists of a gene expression matrix and cell type labels. Gene expression matrix is the input of our model. Every line in the matrix shows different genes' expressions in a cell. The figure below demonstrates a sketch heatmap of gene expression matrix after data preprocessing mentioned in next section. The higher the expression level is, the deeper the blue color will be. There are many places in the matrix that are shallow, which are the reflection of dropouts.
28
12-
Figure 4: gene expression matrix heatmap
We then elaborate the nine datasets we use in experiments, listing out their numbers of cells and genes as well as the numbers of cell types.
Table 1: Single- Cell RNA-Seq data description
Dataset
cell number
gene number
cell type number
CRC
8496
12547
20
GSE70580
647
26087
GSE72056
4636
22280
GSE75688
515
27420
5
GSE96993
334
10827
4
NSCLC
9051
1241.5
16
PRMC
5356
14218
Spleen human
4406
14064
Spleen mouse
4432
12699
7
5.2 PREPROCESSING
There are tens of thousands of genes in the dataset. Before train model, we first implement data preprocessing. For every cell in the data, we first divide every gene's 's expression level by the sum of gene expression levels in the cell. This makes genes' expression levels among cells comparable. Then, we multiply the gene expression levels with a constant 100000 and employ a log transform to adjust the expression level in the data into reasonable range.

---

Under review as a conference paper at ICLR 2022
5.3 PARAMETER SETTING
For feature extraction part, we generate 8 capsules through fully connected layers. Each of the capsule has a dimension of 128. We add L2 normalization to these capsule vectors and we use a single layer bidirectional LSTM model to generate compound features. We set the output dimension of LSTM to 16 and dropout of LSTM to 0.3. A bidirectional LSTM has two output sequences. We add them together and get single sequence of compound features. To obtain attention weights, first add all capsules together. Then we input the sum of capsules into fully connected layers. The output dimension of these layers is also the number of capsules. The number of attention weights is set to the same as cell type number in the dataset. We then multiply the attention weights to the compound features. After multiplying the attention weights the compound features, we obtain matrix whose row number equals to the number of cell types and column number equals to the dimension of compound features. Then we calculate the 1.2noorm of every row and apply Softmax function. In this way, we obtain a vector whose dimension equals to the number of cell types in the dataset and the sum of the elements in the vector is 1. This vector the output of scCA. During the training process, cross entropy loss is used as the loss function. The activation function in model ReLU. We set the dropout to 0.3 and we also add Softmax to the classífication result. We split the dataset into training set and test set with the ratio of 4:1. The model is trained for 50 epochs with Pytorch implementation.
5.4 COMPARISON WITH OTHER METHODS
We first compare scCA with some machine learning methods, including Support Vector Ma- Chine(SVM), Linear Discriminant Analysis(LDA), Decision Tree(DT) and Random Forest(RF). From the table below, we can find that scCA has a robust performance with high accuracy on all datasets. It also has some outstanding performances some of the datasets.
Table 2: Accuracy comparison with various machine learning methods
dataset
ScCA
SVM
LDA
DT
RF
CRC
88.29%
90%
97.29%
64%
$1.47%
GSE70580
96.92%
97.69%
96.15%
90%
96.92%
GSE72056
94.50%
93.85%
91.91%
86.63%
91.59%
GSE75688
92.23%
92.23%
91.26%
88.34%
91.26%
GSE96993
82.83%
80.59%
77.61%
68.65%
82.08%
NSCLC
84.15%
85.20%
82.77%
62.39%
79.18%
PBMC
98.32%
98.13%
97.48%
95.52%
97. 97.94%
Spleen human
92.85%
92.85%
87.86%
82.53%
87.86%
Spleen mouse
96.84%
97.29%
94.58%
88.61%
92.33%
Table 3: Accuracy comparison with various neural network methods
Dataset
ScCA
scCapsNet
denoising connected network
fully
autoencoder
CRC
88.29%
84.47%
88.64%
83.71 13.71%
GSE70580
96.92%
94.62%
96.92%
96.15%
GSE72056
94.50%
92.20%
93.53%
91.33%
GSE75688
92.23%
91.74%
92.23%
91.98%
GSE96993
82.83%
77.61%
80.59%
79.59%
NSCLC
84.15%
79.18%
82.93%
79.80%
PBMC
98.32%
98.23%
98.5%
97.60%
Spleen human
92.85%
91.40%
92.63%
89.79%
Spleen mouse
96.84%
96.17%
97.64%
95.41%
We then compare scCA with neural network based methods. Here we implement fully- culne-coneected network and denoising autoencoder. We use the code provided in scCapsNet. From the table above,
7

---

Under review as a conference paper at ICLR 2022
that among models based neural networks, scCA still has good performance. It outperforms other methods on most the datasets.
In conclusion, through the comparison of our model with other models, scCA shows robustness and high accuracy. On some datasets, ScCA has the best performance of all models.
INFLUENCE OF DIFFERENT PARAMETERS
In our model, the number and dimension of capsules, the dimension of compound features can be adjusted. To test their influence scCA's performance, carry experiments under different combinations of parameters.
Table 4: scCA's performance under different parameters with 128-dimension capsule
Dataset
8capsule 16-dim compound features
16capsuue + 16-dim compound features
8capsule + 32-dim
16capsule + 32-dim
compound features
compound features
CRC
88.29%
88.41%
88.58%
88.94%
GSE70580
96.92%
96.92%
96.15%
96.92%
GSE72056
94.50%
93.96%
94.07%
94.18%
GSE75688
92.23%
92.23%
92.23%
92.23%
GSE96993
82.83%
82.08%
83.58%
82.78%
NSCLC
84.15%
84.15%
83.76%
84. 70%
PBMC
98.32%
98.32%
98.32%
98.60%
Spleen human
92.85%
92.40%
92.29%
92.85%
Spleen mouse
96.84%
96.73%
97.06%
96.73%
In table set the capsule dimension 128 and change the number of capsules and the dimen- sion of compound features. Through the result from the table, we discover that there is no visible improvements no matter we rise the number of capsule or the dimension of compound features.
Table 5: scCA's performance under different parameters with 64-dimension capsule
Dataset
8capsule 16-dim compound features
16capsule 16-dim compound features
8capsule 32-dim compound features
16capsule 32-dim compound features
CRC
87.70%
88.17%
88%
88.29%
GSE70580
96.53%
96.92%
96.53%
96.92%
GSE72056
93.64%
94.07%
93.75%
94.07%
GSE75688
91.26%
91.26%
92.23%
92.23%
GSE96993
82.08%
82.83%
81.34%
83.58%
NSCLC
82.21%
82.44%
82.93%
84.32%
PBMC
98.32%
98.41%
98.04%
98.41
Spleen human
92.06%
92.17%
91.95%
92.74%
Spleen mouse
96.73%
96.73%
96.61% %
96.92%
In table5, we set the capsule dimension to 64 and change the number of capsules and the dimension of compound features. Together with table 4, we discover that number of capsules, the dimension of capsules and dimension of compound features have little influence scCA's performance. Our model performs well under different settings of parameters.
CONCLUSION
In this article, we provide a neural network based method ScCA for single-cell RNA-seq data cell type classification. scCA uses capsules to extract features from cells. Then through LSTM network it generates compound features which contain information from a set single features. By apply- ing attention weights these compound features, lay emphasis compound features which are more closely related to cell types. Deep neural network based. scCA provides reliable interpreta- tions for its result. We calculate the Pearson coefficients among capsules and draw heatmaps of these
8

---

Under review as a conference paper at ICLR 2022
coefficients. We discover that cells from same cell types share similar pattern of capsules' relation- ship while cells from different cell type differ from each other. Moreover, we also draw heatmaps of the attention weights. We also find out that same cell types emphasize on same compound features. These provide vivid explanations and illustrations for our model. We test seCA on nine datasets. Compared with other machine learning and deep learning methods, scCA shows robustness and has a high classification accuracy. Furthermore, we adjust the parameters in our model and discover that our model sensitive them.
REFERENCES
T. R. M. Abdelaal, L. C. M. Michielsen, Davy Cats, Dylan Hoogduin, Hailiang Mei, M. J. T. Rein- ders, and A. M. E. T. A. Mahfouz. A comparison of automatic cell identification methods for single-cell ma sequencing data. Genome biology, 20(1):194-194, 2019.
Jose Alquicira-Hernande Anuja Sathe, Hanlee P. Ji, Quan Nguyen, and Joseph E. Powell. Scpred: Accurate supervised method for cell-type classification from single-cell ma-seq data. Genome Biology, 20(1) (111)::242266 2019.
Maayan Baron, Adrian Veres, Samuel L Wolock, Aubrey L Faust, Renaud Gaujoux, Amedeo Vetere, Jennifer Hyoje Ryu, Bridget K Wagner, Shai S Shen-Orr, Allon M Klein, Douglas A Melton, and Itai Yanai. A single-cell transcriptomic map of the human and mouse pancreas reveals inter- and intra-cell population structure. Cell systems, 3(4):346-360.ek 2016.
Minzhe Guo, Hui Wang. S. S. Potter, Jeffrey A. Whitsett, and Yan Xu. Sincera: A pipeline for single-cell ma-seq profiling analysis. PLoS computational biology, 11(1)ee0004533-1100055. 2015.
Xiaoping Han, Renying Wang, Yincong Zhou, Lijiang Fei, Huiyu Sun, Shujing Lai, Assieh Saadat- pour, Ziming Zhou, Haide Chen, Fang Ye, Daosheng Huang, Yang Xu, Wentao Huang, Meng- meng Jiang, Xinyi Jiang. Jie Mao, Yao Chen, Chenyu Lu, Jin Xie, Qun Fang, Yibin Wang, Rui Yue, Tiefeng Li, He Huang, Stuart H. Orkin, Guo-Cheng Yuan, Ming Chen, and Guoji Guo. Mapping the mouse cell atlas by microwell-seq, Cell, 37765): 2018.
Geoffrey E. Hinton, Zoubin Ghahramani, and Yee W. Teh. Learning to parse images. pp. 463-469, 2000. ISBN 1049-5258.
Geoffrey E. Hinton, Alex Krizhevsky, and Sida D. Wang. Transforming Auto-Encoders, volume 6791, chapter 1, pp. 44-51. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011. ISBN 0302- 9743.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9 15:1735-1780. 11 1997. ISSN 1899.7667, doi: 10. 22.116//nec..1977.0 7.89..9..... 1735.. URL https:
//dei.org//0... 1122////....1......... 1 1997. 1.97.9.64 8..8.1735
Mo Huang, Jingshu Wang, Eduardo Torre, Hannah Dueck, Sydney Shaffer, Roberto Bonasio, John I. Murray, Arjun Raj, Mingyao Li, and Nancy R. Zhang. Saver: Gene expression recovery for single-cell ma sequencing. Nature methods, 157))559-542. 2018.
Byungjin Hwang, H. Lee, and Duhee Bang. Single-cell ma sequencing technologies and bioin- formatics pipelines. Experimental & molecular medicine, 9068);96-14, 2018.
Jialong Liang, Wanshi Cai, and Zhongsheng Sun. Single-cell sequencing technologies: Current and future. Journal of genetics and genomics, 41(10):513-52) 2014.
Chich Lin, Siddhartha Jain, Hannah Kim, and Ziv Bar-Joseph. Using neural networks for reducing the dimensions of single-cell ma-seq data. Nucleic acids research, 45(17)xe156-e156 2017.
Romain Lopez, Jeffrey Regier, Michael B. Cole, Michael I. Jordan, and Nir Yosef. Deep generative modeling for single-cell transcriptomics. Nature methods, 1553)) 2018.
Feiyang Ma and Matteo Pellegrini. Actinn: automated identification cell types in single cell ma sequencing. BIOINFORMATICS, 36(2):533-53 2020.

---

Under review as a conference paper at ICLR 2022
Mauro J Muraro, Gitanjali Dharmadhikari, Dominic Grün, Nathalic Groen, Tim Dielen, Erik Jansen, Leon van Gurp, Marten A Engelse, Francoise Carlotti, Eelco Koning, and Alexander van Oudenaarden. A single-cell transcriptome atlas of the human pancreas. Cell systems, 3(4):
385-394.e3, 2016.
Emma Pierson and Christopher Yau. Zifa: Dimensionality reduction for zero-inflated single-cell gene expression analysis. Genome biology, 16:241, 2015.
Hannah A. Pliner, Jay Shendure, and Cole Trapnell. Supervised classification enables rapid annota- tion of cell atlases. Nature methods, 16(10):983986, 2019.
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. Dynamic routing between capsules. 2017.
Valentine Svensson, Roser Vento-Tormo, and Sarah A. Teichmann. Exponential scaling of single- cell ma-seq in the past decade. Nature protocols, 13(4)5599-644 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. volume 2017-, pp. 5999-6009, 2017. ISBN 1049-5258.
Florian Wagner and Itai Yanai. Moana: A robust and scalable cell type classification framework for single-cell maa data. bioRxiv, 2018. doi: 10.1101/456129, hRLtttts::/wwwt http:::/wwwwwbb.oxx....
Lifei Wang, Rui Nie, Zeyang Yu, Ruyue Xin, Caihong Zheng, Zhang Zhang, Jiang Zhang, and Jun Cai. An interpretable deep-learning architecture of capsule networks for identifying cell-type gene expression programs from single-cell ma-sequencing data. Nature machine intelligence, (11):693, 2020.
Zhang, Luo, Zhong, Choi, Ma, Wang, Mahrt, Guo, Stawiski, Modrusan, Seshagiri. Kapur, Hon, and Brugarolas. Scina: Semi-supervised analysis single cells in silico. Genes, 10(7):531, 2019a.
Allen W. Zhang, Ciara O' Flanagan, Elizabeth A. Chavez, Jamie L. P. Lim, Nicholas Ceglia, Andrew McPherson, Matt Wiens, Pascale Walters, Tim Chan, Brittany Hewitson, Daniel Lai, Anja Mot- tok, Clementine Sarkozy, Lauren Chong, Tomohiro Aoki, Xuehai Wang, Andrew P. Weng, Jes- sica N. McAlpine, Samuel Aparicio, Christian Steidl, Kieran Campbell, and Sohrab P. Shah. Probabilistic cell-type assignment of single-cell ma-seq for tumor microenvironment profiling. Nature methods, 16(10):0007-1055, 2019b.
10


Reviews:
[{"review_id": "lV9-f-z_rQU", "reviewer": "Reviewer_HMhu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper\n\n* Benchmarking is undertaken across a large number of datasets. These datasets span different number of cells and cell types, diverse biological systems\n\n* The robustness of scCA is well tested. The broad range of parameters tested give a degree of confidence in the results. \n\n\nWeaknesses of the paper\n\n* The presented approach is unfortunately of minimal practical utility. Single-cell datasets have severe batch effects with technical differences between technologies, tissue handling etc. Therefore, a classifier trained on one dataset is not guaranteed to generalize to other datasets even if the data is derived from the biological system. Therefore as highlighted by the authors, most of the cell-type classification methods make use of known markers to ensure generalization. Other methods (such as scVI) use a semi-supervised approach rather than a fully supervised approach. Finally, there have been recent developments which use xGBoost to ensure generalization to different datasets (https://www.biorxiv.org/content/10.1101/2021.05.20.445014v1.full). \n\n* The authors claim scCA provides greater interpretation compared to existing approaches. I do not believe the capsule features and compound features necessarily add to the biological interpretation of the cell-type gene expression states. The additional interpretation as claimed by the authors is not sufficiently elaborated. \n\n* One of the main claims of the paper is that capsules encode similar information for cells of the same celltype. This claim is demonstrated by examples of few cells in Fig. 2. Given the centrality of the claim, the authors should elaborate and provide additional metrics to prove. The authors should also specify how the model architecture ensures this consistency. \n\n* LSTM as noted by the authors is used for data with sequences. So, it is not clear why LSTM architecture is the choice here given there is no sequence data - infact, the authors adjust the weights to account for the fact that single-cell data and capsules are not sequential in nature. Therefore, it is not clear why LSTM is the model choice compared to alternatives. \n\n* A few other issues in terms of single-cell data processing:\n\n\t- Given that single-cell data follows negative binomial distribution, variance is a function of the mean and hence use of most variable genes would not correct for the mean. Dispersion estimates are typically used as a work-around (https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1874-1)\n\n\t- The gene expression map in Fig. 4 is rather unrepresentative since the dynamic range of scRNA-seq in linear scale spans 2-3 orders of magnitude\n\n\t- Is the single-cell data normalized for differing molecule counts in each cell?\n\t\t\n", "summary_of_the_paper": "This manuscript describes scCA (single-cell capsule attention) for cell-type classification using single-cell RNA-seq data. scCA usesa combination of capsule networks, LSTN and attentions to undertake cell-type classifiation. Capsules extract features from single-cell data, the combination of capsules called \"compound features\" through LSTM and attention weights are employed. scCA is compared against a host of other techniques and performs well across 9 diverse single-cell datasets. scCA results are also robust to various parameter choices across datasets.", "summary_of_the_review": "The main reasons for the recommendation are as follows:\n* The manuscript does not demonstrate generalization from one dataset to another which is the main practical application of cell-type classification using single-cell data. \n* The claims of additional biological interpretation are not sufficiently elaborated\n* Given the subject specific nature of the submission, it might not be of broad interest to the ICLR audience.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gB2RKIpoOE", "reviewer": "Reviewer_YcRA", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work is a straightforward application of well-known methods to the single-cell RNA seq data and do not provide technical contributions\n\nResults provide weak support to authors’ claims: \n* The classification results show minor improvement compared to standard methods like SVM and fully connected network. No other insights are provided to support the benefit of using scCA to classify scRNA data. \n* On the interpretability of scCA, only heat maps of feature correlations for two cell types are shown. First, It is not clear such visualization is unique to the capsule network in any way. I believe other methods can provide similar results, for example, by looking at the hidden layers of a fully connected network. Second, it is not clear such interpretable patterns can generalize to other cell types/datasets. Since the names of cell types are not specified, it is also difficult to judge the resolution of learned features.  \n\nThe presentation of the paper needs to be improved. Particularly\n\n* Missing important details. For example, it is not clear Fig 2 and 3 are generated. No train/test split/cross validation and parameter tuning are described. \n", "summary_of_the_paper": "In this work, the authors implemented a capsule network to classify cell types for single-cell RNA-seq data. The capsule network was compared with standard methods such as SVM, LDA, and a fully connected neural network. The authors claim the capsule network provides interpretable results on cell type classification.", "summary_of_the_review": "I think this work is a straightforward application of well-known methods to the single-cell RNA seq data. The results do not support the proposed method is superior to alternatives. I am therefore inclined to reject this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DhbRigmnzwQ", "reviewer": "Reviewer_mrct", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper suggests a neural network architecture (combining capsules, attention, and LSTM) to perform cell type classification for single-cell RNA-seq data. The authors claim that their method performs well and is interpretable -- more so than the competitors.\n\nI found the paper difficult to understand, classification results not convincing, and interpretability claims greatly overstated. Unfortunately I believe this is a clear reject.\n\nDisclaimer: I have heard about attention and capsules but do not know how they work. However, I think this paper can be assessed even if one treats the model as a black box. Also, I do work with single-cell RNA-seq data all the time.\n\n\nMAJOR COMMENTS\n\n* The authors claim that their method is \"interpretable\" (it's right in the title) but then does not demonstrate how it can be interpreted in practice. How is it more interpretable than a fully-connected network, a SVM, or a random forest (that have similar classification performance: Tables 1-2)? Unclear.\n\n* The authors claim that their method performs well, but in fact it does not perform better than a fully-connected network (Table 3), which does not use either attention, or capsules, or LSTM.\n\n* The paper is hard to follow, e.g. I could not fully understand what exactly is shown in Figures 2 and 3. Figures do not have meaningful captions, subplots and axes are not labeled, surrounding text is hard to read.", "summary_of_the_paper": "The paper suggests a neural network architecture (combining capsules, attention, and LSTM) to perform cell type classification for single-cell RNA-seq data. The authors claim that their method performs well and is interpretable -- more so than the competitors.", "summary_of_the_review": "I found the paper difficult to understand, classification results not convincing, and interpretability claims greatly overstated. Unfortunately I believe this is a clear reject.\n\nDisclaimer: I have heard about attention and capsules but do not know how they work. However, I think this paper can be assessed even if one treats the model as a black box. Also, I do work with single-cell RNA-seq data all the time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7ihRKHUM5gy", "reviewer": "Reviewer_khq7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well structured and relatively easy to follow and understand (at least on a high level).\nHowever, I am having major concerns with several of the tenets of this work.\n\nFirst of all I am not a specialist of capsule networks, and I may very well be missing something.\nHowever, given that there is only one layer of capsules (that are then combined using an LSTM), there is no dynamic routing, which I understand are (at least partly) the essence of Capsule Nets.\nAs such, I am not convinced the suggested architecture really falls into the Capsule framework.\nAt the very least, it can be seen as a multi-headed single layer MLP that extracts features, which can be interesting.\n\nI understand that the features extracted, or capsules need to be combined.\nHowever, I do not believe that an LSTM is the right tool for that, given that they do not constitute a sequence.\nThe authors mention that this is an issue and suggest that they used attention to make the model invariant to the order of the capsules but I don't understand why that would be the case.\nCan you elaborate?\nIn any case, along the lines of what the authors have in mind, I wonder if using Deep Sets (Zaheer et al. '18), which can be combined with attenion like in (Ilse et al. '18) to provide interpretability, is not a better candidate.\nIn all honestly, deep sets are mostly useful when the number of elements in the set is not constant across example.\nHere, I suspect even a concatenation of the capsules with a MLP should do the trick.\n\nAn important claim of the paper is that the suggested model offers interpretability.\nHowever, I am not convinced at all that is the case.\nThe capsules are obtained from a non-linear combination of all the genes.\nAs such, I am not sure what benefit there is from observing that a given cell type is associated to a specific capsule pattern as those will have no clear interpretability.\nI am maybe missing something, but in any case, despite the claims, the authors never comment on the interpretation of those features and whether they are biologically relevant or interesting.\n\nIt is interesting that some patterns are emerging from the capsule activations.\nBut in fact, the model is trained on a classification task.\nAs such, it is only natural that the features extracted will be made separable.\nI don't think there is anything that is emerging specifically with the given architecture.\nWith a vanilla MLP, the features extracted in the late layers of the model would also exhibit such patterns because they would eventually need to be separable to be classified.\n\nThe presented experimental results are reasonable but certainly not groundbreaking, especially as they are mostly compared to vanilla methods such as SVM, Decision Trees, Random Forests etc.\nThey seem to mostly be on-par with other methods.\nThis would be fine if the suggested approach was providing further benefits or a very novel approach.\nPerhpas that could be the case if there were more analysis on interpretability and scCA was highlighting some biolgically interpretable and relevant features but in the current manuscript, that is not sufficient.\n\n\nBeyond those major points, here are a few more remarks:\n - Given the similarities with scCaps, there definitely should be more emphasis on the comparison between the two approaches.\n - Through the manuscript, understand the \"mathematical\" details of the paper is made difficult because the authors never mention the dimensions of the different vectors and matrices they define. Can you correct that?\n - At the end of the Statistical Hypotheses Based Model subsection, the sentence \"Their hypotheses of gene expression may not match with the true expression level.\" is not clear to me. Can you elaborate?\n - It is also mentioned in the \"preprocessing\" that the data is \"amplified\". What does that mean?\n - In the \"parameter setting\" section, there is mention of a normalization. Can you elaborate on which?\n - Also, the writing in that section is really not clear to me, for instance :\"Then, we get the classification result\". What is actually meant here?\n - It is never mentioned which loss was used to train the model. Is it the cross-entropy loss?\n - In several instances, it is written that \"Hinton\" proposed Capsule networks, or \"pointed out\" something. It would be more approriate to refer to Sabour and co-authors (or the authors of author specific papers) rather than \"name dropping\" Hinton.\n - There are a couple typos that I suspect are due to reducing to fit into the conference format (e.g. at the top of page 2, \"scCA provides *a* reasonable\", \"in every cell, *we* use\" etc...)", "summary_of_the_paper": "The paper tackles the problem of classifying single-cell RNA-seq expression data according to their underlying cell types.\nTo do so, they suggest a deep neural network, inspired from Capsule Networks, and making use of attention to provide interpretable results.\nThe features extracted by the network are empirically shown to exhibit cell-type-specific patterns.\nIn terms of performance, the suggested method compares favourably or is on par with classic, standard ML approaches.\nFinally, the authors show that the results seem to be relatively stable with respect to the choice of some architecture hyper-parameters.", "summary_of_the_review": "In its current form, the proposed approach does not seem groundbreakingly novel, nor significantly improve over existing methods, nor offer extra benefits.\nI think some of the design choices are debatable and I would love to hear what the authors have to say to the different points I raised.\nWithout further justification, though, I think the contributions are too minor to justify an acceptance at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers believe that the paper is not ready for publication and clarity issue remain. All reviewers read the rebuttal responses, but they found that the paper wasn't revised during rebuttal, thus they retain their decisions.
Decision: Reject