Single-Cell Capsule Attention : an interpretable method of cell type classification for single-cell RNA-sequencing data

1 INTRODUCTION . Genes ’ expression on celluar level provides lots of information for us to explore homogeneity and heterogeneity among different cells ( Liang et al. , 2014 ; Muraro et al. , 2016 ; Baron et al. , 2016 ) . Traditional bulk sequencing technique can only measure the average gene expression level of all cells in a sample . Compared with bulk data , single-cell RNA-sequencing ( single-cell RNA-seq ) technique gives a more accurate measurement of every cell . Though single-cell RNA-sequencing is a rising technique , it is not mature enough and still has some limitations . For example , single-cell RNA-seq data has high ’ dropout ’ . ’ Dropout ’ means zero or low read counts in the data because many genes ’ expression are hard to detect ( Huang et al. , 2018 ; Pierson & Yau , 2015 ) . Also , singlecell RNA-seq data are of high dimension because thousands of genes are in the transcriptome . These all bring challenges for studies on single-cell RNA-seq data . So far , many data mining and analysis methods have been applied on single-cell RNA-seq data to solve these challenges and explore more information on cells and genes ( Svensson et al. , 2018 ; Han et al. , 2018 ; Lopez et al. , 2018 ; Hwang et al. , 2018 ; Guo et al. , 2015 ) . Cell type classification is one of the most important tasks in single-cell RNA-seq data analysis . It helps to identify different cell types and explore the cellular heterogeneity . Previous methods on cell type classification can be categorized into methods based on statistical distribution hypotheses of gene expression and deep learning methods based on neural networks ( Abdelaal et al. , 2019 ) . These methods lay a solid foundation for cell type classification . However , they still have limitations and weaknesses . Methods based on statistical hypotheses rely on prior knowledge of marker genes or gene expression ’ s distribution assumptions . Up to now , there is no widely recognized hypotheses of gene expression . Deep neural network ( DNN ) , a fast and effective learning method , is widely used in various fields . DNN does not rely on distribution hypotheses , however , it ’ s hard to provide an reasonable explanation for the result behind the deep ’ black-box ’ ( Lin et al. , 2017 ) . ACTINN ( Ma & Pellegrini , 2020 ) , a deep neural network based method for cell type classification , uses three hidden layers in its architecture . Though its model can predict cell types , the model still lacks reliable interpretations for the result . The challenge is that DNN does not provide a recognizable pattern that is closely related to different cell types . Here , we introduce a new neural network based method scCA for cell type classification . scCA provides an reasonable interpretation for its classification result . Our model learns features from cells and provides patterns behind these features that are related to cell types . The first part is feature extraction . Given the gene expression level in every cell , We use capsules to extract different features . A capsule is a vector that represents a feature , which captures a certain type of information of the cell . Capsules also serve as dimension reduction for single cell RNA-seq data . We calculate the Pearson coefficients between every two capsules and generate the coefficient heatmap of every cell . We discover that cells from same cell type share a similar pattern in the heatmap while cells from different cell types have different patterns . These capsules capture features that are important for cell type classification . And then , we use a sequence to sequence model , bidirectional-LSTM , to generate compound features . Each compound feature combines a set of features ’ information . Furthermore , we train multiple attention weights for these compound features . Then we multiply these weights to the compound features and generate the classification result . Attention weights help to lay more emphasis on important compound features . The larger the attention weight is , the more crucial its corresponding compound feature is . we also draw heatmaps for every cells ’ attention weights . We discover that cells from same cell type have similar patterns in the heatmaps of attention weights . This leads to a better classification performance and also provides an reliable interpretation behind the network . We evaluate scCA on several datasets and compare with previous work . Our method shows high accuracy and robustness on all datasets . The main contributions of our article are as follow : We propose an interpretable neural network based method scCA for cell type classification . Through heatmaps of capsules ’ Pearson coefficients and attention weights , we find that cells from same cell type share a similar pattern . scCA achieves high accuracy and stable performances on all datasets . 2 RELATED WORK . Statistical Hypotheses Based Model Many machine learning methods with statistical hypotheses of gene expression level have been proposed for cell type classification . Some of these methods suggest distribution hypotheses of gene expression level and employ machine learning models based on these hypotheses . Some build a hierarchy with prior knowledge and statistical hypotheses . Moana takes advantage of KNN and SVM model ( Wagner & Yanai , 2018 ) . It first preprocesses data with KNN , then it uses dimension reduction method to extract useful features , at last it uses a Support Vector Machine classifier . In addition , Moana uses marker gene as a prior knowledge for cell type identification . Different from Moana , Garnett proposes a model based on tree hierarchy . It discovers the relation between cell types and subtypes with a tree structure ( Pliner et al. , 2019 ) . With the help of marker gene , it builds a linear model to classify cell types . CellAssign builds a probabilistic model based on prior knowledge to classify cell types ( Zhang et al. , 2019b ) . SCINA makes use of EM algorithm and the information of marker gene to accomplish clustering as well as cell type identification task ( Zhang et al. , 2019a ) . scPred extracts features using SVD , then with a nonlinear kernel SVM , it predicts different cell types ( Alquicira-Hernandez et al. , 2019 ) . However , these methods have their limitations . Their hypotheses such as ZINB ( Zero-inflated Negative Binomial ) or NB distribution of gene expression are not the exact reflections of gene ’ s expression levels in cells . Actually , there is not a consensus statistical distribution for single-cell RNA-seq data . Also they need prior knowledge like marker gene for cell type classification . Neural Network Based Models Deep neural network ( DNN ) based methods have been applied to cell type classification task . They are fast and robust . Neural networks don ’ t heavily rely on prior knowledge and statistical hypotheses . ACTINN uses three fully connected layers and ReLU , softmax activation function to classify different cell types ( Ma & Pellegrini , 2020 ) . scCapsNet builds a capsule network to extract features from the data and use dynamic routings among capsules to get a classification result ( Wang et al. , 2020 ) . However , neural networks lack interpretation for their result . Through training the whole network by propagation , it ’ s hard to explain every component ’ s contribution for the result . 3 PRELIMINARIES . Capsule Network and Dynamic Routing Hinton , Sabour and other co-authors first proposed Capsule network and dynamic routing in computer vision ( Sabour et al. , 2017 ) . A capsule is a set of neurons or a vector that can represent an entity . Dynamic routing is the mechanism between capsule ’ s layers . As they point out , the capsules in the layer above are parents and the capsules in the layer below are children . ( Hinton et al. , 2000 ; 2011 ) . Dynamic routing helps to send the results from capsules in children layer to its appropriate parents in the above layer . By adjusting the coupling coefficients between capsules in two layers , the network learns which parent capsule is more important for the children capsule . Here in scCA , we use capsules to capture different features of the cell . We use a vector for implementation of capsule in our model . However , different from dynamic routing , we use LSTM and attention as the architecture for cell type classification . Long Short-Term Memory and Attention Long Short-Term Memory ( LSTM ) is a model widely used in Natural Language Processing ( NLP ) ( Hochreiter & Schmidhuber , 1997 ) . It is a sequence to sequence model . The inputs are usually the word embeddings ( representations ) of a sentence and the output is a sequence of representations . Each output representation not only contains information from its corresponding input , but also information from representations in the context , especially its neighbor inputs . In our method , after extracting different features into capsules , we input these capsules into the LSTM model and obtain the output sequence . Every vector in the output sequence contains information of its corresponding input feature as well as other features . We name these vectors compound features . Some of the compound features have closer relationships with certain cell types . Inspired by Attention ( Vaswani et al. , 2017 ) mechanism , we apply attention weights to compound features in order to differentiate and emphasize those important compound features for classification . 4 SINGLE-CELL CAPSULE ATTENTION . In this section , we provide a detailed description of single-cell Capsule Attention ( scCA ) . We will also give a vivid and transparent explanation for our classification result . scCA can be divided into three parts . The figure below shows its architecture . 4.1 FEATURE EXTRACTION . The first part of scCA is feature extraction . Given thousands of genes ’ expression levels for a cell , we hope to learn simple but efficacious features for classification . Capsules not only serve as dimension reduction , but also extract a type of information of the cell . Through fully connected layers , we generate capsules to extract different features . capsulei = ReLU ( W ix+ bi ) , i ∈ [ 1 , 2 · · ·m ] ( 1 ) In this formula , m represents the number of capsules . W and b are the weight matrices of fully connected layers . ReLU is the activation function we use . Every capsule contains a feature of a cell . We hope to discover different patterns of these features for different cell types . We calculate the Pearson coefficients between every two capsules . The Pearson coefficient is an indication of relationship between two features behind capsules . For every cell , we come to a matrix where every element is a Pearson coefficient of two capsules . We then draw a heatmap of the matrix . We discover that cells from same cell type share a similar pattern while cells from different cell types differ from each other . Taking six cells from dataset GSE72056 as examples , we can see cells from same cell type have a similar pattern of capsules ’ relationship . The deeper the color is in the heatmap , the closer the relationship of two capsules is . The pattern is distinguishable among different cell types . From the figures above , we conclude that there exists some particular feature patterns for different cell types . Our feature extraction really extracts useful and discriminative features .

Summary: In this work, the authors implemented a capsule network to classify cell types for single-cell RNA-seq data. The capsule network was compared with standard methods such as SVM, LDA, and a fully connected neural network. The authors claim the capsule network provides interpretable results on cell type classification.