Single-Cell Capsule Attention : an interpretable method of cell type classification for single-cell RNA-sequencing data

1 INTRODUCTION . Genes ’ expression on celluar level provides lots of information for us to explore homogeneity and heterogeneity among different cells ( Liang et al. , 2014 ; Muraro et al. , 2016 ; Baron et al. , 2016 ) . Traditional bulk sequencing technique can only measure the average gene expression level of all cells in a sample . Compared with bulk data , single-cell RNA-sequencing ( single-cell RNA-seq ) technique gives a more accurate measurement of every cell . Though single-cell RNA-sequencing is a rising technique , it is not mature enough and still has some limitations . For example , single-cell RNA-seq data has high ’ dropout ’ . ’ Dropout ’ means zero or low read counts in the data because many genes ’ expression are hard to detect ( Huang et al. , 2018 ; Pierson & Yau , 2015 ) . Also , singlecell RNA-seq data are of high dimension because thousands of genes are in the transcriptome . These all bring challenges for studies on single-cell RNA-seq data . So far , many data mining and analysis methods have been applied on single-cell RNA-seq data to solve these challenges and explore more information on cells and genes ( Svensson et al. , 2018 ; Han et al. , 2018 ; Lopez et al. , 2018 ; Hwang et al. , 2018 ; Guo et al. , 2015 ) . Cell type classification is one of the most important tasks in single-cell RNA-seq data analysis . It helps to identify different cell types and explore the cellular heterogeneity . Previous methods on cell type classification can be categorized into methods based on statistical distribution hypotheses of gene expression and deep learning methods based on neural networks ( Abdelaal et al. , 2019 ) . These methods lay a solid foundation for cell type classification . However , they still have limitations and weaknesses . Methods based on statistical hypotheses rely on prior knowledge of marker genes or gene expression ’ s distribution assumptions . Up to now , there is no widely recognized hypotheses of gene expression . Deep neural network ( DNN ) , a fast and effective learning method , is widely used in various fields . DNN does not rely on distribution hypotheses , however , it ’ s hard to provide an reasonable explanation for the result behind the deep ’ black-box ’ ( Lin et al. , 2017 ) . ACTINN ( Ma & Pellegrini , 2020 ) , a deep neural network based method for cell type classification , uses three hidden layers in its architecture . Though its model can predict cell types , the model still lacks reliable interpretations for the result . The challenge is that DNN does not provide a recognizable pattern that is closely related to different cell types . Here , we introduce a new neural network based method scCA for cell type classification . scCA provides an reasonable interpretation for its classification result . Our model learns features from cells and provides patterns behind these features that are related to cell types . The first part is feature extraction . Given the gene expression level in every cell , We use capsules to extract different features . A capsule is a vector that represents a feature , which captures a certain type of information of the cell . Capsules also serve as dimension reduction for single cell RNA-seq data . We calculate the Pearson coefficients between every two capsules and generate the coefficient heatmap of every cell . We discover that cells from same cell type share a similar pattern in the heatmap while cells from different cell types have different patterns . These capsules capture features that are important for cell type classification . And then , we use a sequence to sequence model , bidirectional-LSTM , to generate compound features . Each compound feature combines a set of features ’ information . Furthermore , we train multiple attention weights for these compound features . Then we multiply these weights to the compound features and generate the classification result . Attention weights help to lay more emphasis on important compound features . The larger the attention weight is , the more crucial its corresponding compound feature is . we also draw heatmaps for every cells ’ attention weights . We discover that cells from same cell type have similar patterns in the heatmaps of attention weights . This leads to a better classification performance and also provides an reliable interpretation behind the network . We evaluate scCA on several datasets and compare with previous work . Our method shows high accuracy and robustness on all datasets . The main contributions of our article are as follow : We propose an interpretable neural network based method scCA for cell type classification . Through heatmaps of capsules ’ Pearson coefficients and attention weights , we find that cells from same cell type share a similar pattern . scCA achieves high accuracy and stable performances on all datasets . 2 RELATED WORK . Statistical Hypotheses Based Model Many machine learning methods with statistical hypotheses of gene expression level have been proposed for cell type classification . Some of these methods suggest distribution hypotheses of gene expression level and employ machine learning models based on these hypotheses . Some build a hierarchy with prior knowledge and statistical hypotheses . Moana takes advantage of KNN and SVM model ( Wagner & Yanai , 2018 ) . It first preprocesses data with KNN , then it uses dimension reduction method to extract useful features , at last it uses a Support Vector Machine classifier . In addition , Moana uses marker gene as a prior knowledge for cell type identification . Different from Moana , Garnett proposes a model based on tree hierarchy . It discovers the relation between cell types and subtypes with a tree structure ( Pliner et al. , 2019 ) . With the help of marker gene , it builds a linear model to classify cell types . CellAssign builds a probabilistic model based on prior knowledge to classify cell types ( Zhang et al. , 2019b ) . SCINA makes use of EM algorithm and the information of marker gene to accomplish clustering as well as cell type identification task ( Zhang et al. , 2019a ) . scPred extracts features using SVD , then with a nonlinear kernel SVM , it predicts different cell types ( Alquicira-Hernandez et al. , 2019 ) . However , these methods have their limitations . Their hypotheses such as ZINB ( Zero-inflated Negative Binomial ) or NB distribution of gene expression are not the exact reflections of gene ’ s expression levels in cells . Actually , there is not a consensus statistical distribution for single-cell RNA-seq data . Also they need prior knowledge like marker gene for cell type classification . Neural Network Based Models Deep neural network ( DNN ) based methods have been applied to cell type classification task . They are fast and robust . Neural networks don ’ t heavily rely on prior knowledge and statistical hypotheses . ACTINN uses three fully connected layers and ReLU , softmax activation function to classify different cell types ( Ma & Pellegrini , 2020 ) . scCapsNet builds a capsule network to extract features from the data and use dynamic routings among capsules to get a classification result ( Wang et al. , 2020 ) . However , neural networks lack interpretation for their result . Through training the whole network by propagation , it ’ s hard to explain every component ’ s contribution for the result . 3 PRELIMINARIES . Capsule Network and Dynamic Routing Hinton , Sabour and other co-authors first proposed Capsule network and dynamic routing in computer vision ( Sabour et al. , 2017 ) . A capsule is a set of neurons or a vector that can represent an entity . Dynamic routing is the mechanism between capsule ’ s layers . As they point out , the capsules in the layer above are parents and the capsules in the layer below are children . ( Hinton et al. , 2000 ; 2011 ) . Dynamic routing helps to send the results from capsules in children layer to its appropriate parents in the above layer . By adjusting the coupling coefficients between capsules in two layers , the network learns which parent capsule is more important for the children capsule . Here in scCA , we use capsules to capture different features of the cell . We use a vector for implementation of capsule in our model . However , different from dynamic routing , we use LSTM and attention as the architecture for cell type classification . Long Short-Term Memory and Attention Long Short-Term Memory ( LSTM ) is a model widely used in Natural Language Processing ( NLP ) ( Hochreiter & Schmidhuber , 1997 ) . It is a sequence to sequence model . The inputs are usually the word embeddings ( representations ) of a sentence and the output is a sequence of representations . Each output representation not only contains information from its corresponding input , but also information from representations in the context , especially its neighbor inputs . In our method , after extracting different features into capsules , we input these capsules into the LSTM model and obtain the output sequence . Every vector in the output sequence contains information of its corresponding input feature as well as other features . We name these vectors compound features . Some of the compound features have closer relationships with certain cell types . Inspired by Attention ( Vaswani et al. , 2017 ) mechanism , we apply attention weights to compound features in order to differentiate and emphasize those important compound features for classification . 4 SINGLE-CELL CAPSULE ATTENTION . In this section , we provide a detailed description of single-cell Capsule Attention ( scCA ) . We will also give a vivid and transparent explanation for our classification result . scCA can be divided into three parts . The figure below shows its architecture . 4.1 FEATURE EXTRACTION . The first part of scCA is feature extraction . Given thousands of genes ’ expression levels for a cell , we hope to learn simple but efficacious features for classification . Capsules not only serve as dimension reduction , but also extract a type of information of the cell . Through fully connected layers , we generate capsules to extract different features . capsulei = ReLU ( W ix+ bi ) , i ∈ [ 1 , 2 · · ·m ] ( 1 ) In this formula , m represents the number of capsules . W and b are the weight matrices of fully connected layers . ReLU is the activation function we use . Every capsule contains a feature of a cell . We hope to discover different patterns of these features for different cell types . We calculate the Pearson coefficients between every two capsules . The Pearson coefficient is an indication of relationship between two features behind capsules . For every cell , we come to a matrix where every element is a Pearson coefficient of two capsules . We then draw a heatmap of the matrix . We discover that cells from same cell type share a similar pattern while cells from different cell types differ from each other . Taking six cells from dataset GSE72056 as examples , we can see cells from same cell type have a similar pattern of capsules ’ relationship . The deeper the color is in the heatmap , the closer the relationship of two capsules is . The pattern is distinguishable among different cell types . From the figures above , we conclude that there exists some particular feature patterns for different cell types . Our feature extraction really extracts useful and discriminative features .

Summary: The paper tackles the problem of classifying single-cell RNA-seq expression data according to their underlying cell types. To do so, they suggest a deep neural network, inspired from Capsule Networks, and making use of attention to provide interpretable results. The features extracted by the network are empirically shown to exhibit cell-type-specific patterns. In terms of performance, the suggested method compares favourably or is on par with classic, standard ML approaches. Finally, the authors show that the results seem to be relatively stable with respect to the choice of some architecture hyper-parameters.