Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This manuscript describes scCA (single-cell capsule attention) for cell-type classification using single-cell RNA-seq data. scCA usesa combination of capsule networks, LSTN and attentions to undertake cell-type classifiation. Capsules extract features from single-cell data, the combination of capsules called "compound features" through LSTM and attention weights are employed. scCA is compared against a host of other techniques and performs well across 9 diverse single-cell datasets. scCA results are also robust to various parameter choices across datasets.
main_review: Strengths of the paper

* Benchmarking is undertaken across a large number of datasets. These datasets span different number of cells and cell types, diverse biological systems

* The robustness of scCA is well tested. The broad range of parameters tested give a degree of confidence in the results. 


Weaknesses of the paper

* The presented approach is unfortunately of minimal practical utility. Single-cell datasets have severe batch effects with technical differences between technologies, tissue handling etc. Therefore, a classifier trained on one dataset is not guaranteed to generalize to other datasets even if the data is derived from the biological system. Therefore as highlighted by the authors, most of the cell-type classification methods make use of known markers to ensure generalization. Other methods (such as scVI) use a semi-supervised approach rather than a fully supervised approach. Finally, there have been recent developments which use xGBoost to ensure generalization to different datasets (https://www.biorxiv.org/content/10.1101/2021.05.20.445014v1.full). 

* The authors claim scCA provides greater interpretation compared to existing approaches. I do not believe the capsule features and compound features necessarily add to the biological interpretation of the cell-type gene expression states. The additional interpretation as claimed by the authors is not sufficiently elaborated. 

* One of the main claims of the paper is that capsules encode similar information for cells of the same celltype. This claim is demonstrated by examples of few cells in Fig. 2. Given the centrality of the claim, the authors should elaborate and provide additional metrics to prove. The authors should also specify how the model architecture ensures this consistency. 

* LSTM as noted by the authors is used for data with sequences. So, it is not clear why LSTM architecture is the choice here given there is no sequence data - infact, the authors adjust the weights to account for the fact that single-cell data and capsules are not sequential in nature. Therefore, it is not clear why LSTM is the model choice compared to alternatives. 

* A few other issues in terms of single-cell data processing:

	- Given that single-cell data follows negative binomial distribution, variance is a function of the mean and hence use of most variable genes would not correct for the mean. Dispersion estimates are typically used as a work-around (https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1874-1)

	- The gene expression map in Fig. 4 is rather unrepresentative since the dynamic range of scRNA-seq in linear scale spans 2-3 orders of magnitude

	- Is the single-cell data normalized for differing molecule counts in each cell?
		

summary_of_the_review: The main reasons for the recommendation are as follows:
* The manuscript does not demonstrate generalization from one dataset to another which is the main practical application of cell-type classification using single-cell data. 
* The claims of additional biological interpretation are not sufficiently elaborated
* Given the subject specific nature of the submission, it might not be of broad interest to the ICLR audience.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper tackles the problem of classifying single-cell RNA-seq expression data according to their underlying cell types.
To do so, they suggest a deep neural network, inspired from Capsule Networks, and making use of attention to provide interpretable results.
The features extracted by the network are empirically shown to exhibit cell-type-specific patterns.
In terms of performance, the suggested method compares favourably or is on par with classic, standard ML approaches.
Finally, the authors show that the results seem to be relatively stable with respect to the choice of some architecture hyper-parameters.
main_review: The paper is well structured and relatively easy to follow and understand (at least on a high level).
However, I am having major concerns with several of the tenets of this work.

First of all I am not a specialist of capsule networks, and I may very well be missing something.
However, given that there is only one layer of capsules (that are then combined using an LSTM), there is no dynamic routing, which I understand are (at least partly) the essence of Capsule Nets.
As such, I am not convinced the suggested architecture really falls into the Capsule framework.
At the very least, it can be seen as a multi-headed single layer MLP that extracts features, which can be interesting.

I understand that the features extracted, or capsules need to be combined.
However, I do not believe that an LSTM is the right tool for that, given that they do not constitute a sequence.
The authors mention that this is an issue and suggest that they used attention to make the model invariant to the order of the capsules but I don't understand why that would be the case.
Can you elaborate?
In any case, along the lines of what the authors have in mind, I wonder if using Deep Sets (Zaheer et al. '18), which can be combined with attenion like in (Ilse et al. '18) to provide interpretability, is not a better candidate.
In all honestly, deep sets are mostly useful when the number of elements in the set is not constant across example.
Here, I suspect even a concatenation of the capsules with a MLP should do the trick.

An important claim of the paper is that the suggested model offers interpretability.
However, I am not convinced at all that is the case.
The capsules are obtained from a non-linear combination of all the genes.
As such, I am not sure what benefit there is from observing that a given cell type is associated to a specific capsule pattern as those will have no clear interpretability.
I am maybe missing something, but in any case, despite the claims, the authors never comment on the interpretation of those features and whether they are biologically relevant or interesting.

It is interesting that some patterns are emerging from the capsule activations.
But in fact, the model is trained on a classification task.
As such, it is only natural that the features extracted will be made separable.
I don't think there is anything that is emerging specifically with the given architecture.
With a vanilla MLP, the features extracted in the late layers of the model would also exhibit such patterns because they would eventually need to be separable to be classified.

The presented experimental results are reasonable but certainly not groundbreaking, especially as they are mostly compared to vanilla methods such as SVM, Decision Trees, Random Forests etc.
They seem to mostly be on-par with other methods.
This would be fine if the suggested approach was providing further benefits or a very novel approach.
Perhpas that could be the case if there were more analysis on interpretability and scCA was highlighting some biolgically interpretable and relevant features but in the current manuscript, that is not sufficient.


Beyond those major points, here are a few more remarks:
 - Given the similarities with scCaps, there definitely should be more emphasis on the comparison between the two approaches.
 - Through the manuscript, understand the "mathematical" details of the paper is made difficult because the authors never mention the dimensions of the different vectors and matrices they define. Can you correct that?
 - At the end of the Statistical Hypotheses Based Model subsection, the sentence "Their hypotheses of gene expression may not match with the true expression level." is not clear to me. Can you elaborate?
 - It is also mentioned in the "preprocessing" that the data is "amplified". What does that mean?
 - In the "parameter setting" section, there is mention of a normalization. Can you elaborate on which?
 - Also, the writing in that section is really not clear to me, for instance :"Then, we get the classification result". What is actually meant here?
 - It is never mentioned which loss was used to train the model. Is it the cross-entropy loss?
 - In several instances, it is written that "Hinton" proposed Capsule networks, or "pointed out" something. It would be more approriate to refer to Sabour and co-authors (or the authors of author specific papers) rather than "name dropping" Hinton.
 - There are a couple typos that I suspect are due to reducing to fit into the conference format (e.g. at the top of page 2, "scCA provides *a* reasonable", "in every cell, *we* use" etc...)
summary_of_the_review: In its current form, the proposed approach does not seem groundbreakingly novel, nor significantly improve over existing methods, nor offer extra benefits.
I think some of the design choices are debatable and I would love to hear what the authors have to say to the different points I raised.
Without further justification, though, I think the contributions are too minor to justify an acceptance at ICLR.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: In this work, the authors implemented a capsule network to classify cell types for single-cell RNA-seq data. The capsule network was compared with standard methods such as SVM, LDA, and a fully connected neural network. The authors claim the capsule network provides interpretable results on cell type classification.
main_review: This work is a straightforward application of well-known methods to the single-cell RNA seq data and do not provide technical contributions

Results provide weak support to authors’ claims: 
* The classification results show minor improvement compared to standard methods like SVM and fully connected network. No other insights are provided to support the benefit of using scCA to classify scRNA data. 
* On the interpretability of scCA, only heat maps of feature correlations for two cell types are shown. First, It is not clear such visualization is unique to the capsule network in any way. I believe other methods can provide similar results, for example, by looking at the hidden layers of a fully connected network. Second, it is not clear such interpretable patterns can generalize to other cell types/datasets. Since the names of cell types are not specified, it is also difficult to judge the resolution of learned features.  

The presentation of the paper needs to be improved. Particularly

* Missing important details. For example, it is not clear Fig 2 and 3 are generated. No train/test split/cross validation and parameter tuning are described. 

summary_of_the_review: I think this work is a straightforward application of well-known methods to the single-cell RNA seq data. The results do not support the proposed method is superior to alternatives. I am therefore inclined to reject this paper.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper suggests a neural network architecture (combining capsules, attention, and LSTM) to perform cell type classification for single-cell RNA-seq data. The authors claim that their method performs well and is interpretable -- more so than the competitors.
main_review: The paper suggests a neural network architecture (combining capsules, attention, and LSTM) to perform cell type classification for single-cell RNA-seq data. The authors claim that their method performs well and is interpretable -- more so than the competitors.

I found the paper difficult to understand, classification results not convincing, and interpretability claims greatly overstated. Unfortunately I believe this is a clear reject.

Disclaimer: I have heard about attention and capsules but do not know how they work. However, I think this paper can be assessed even if one treats the model as a black box. Also, I do work with single-cell RNA-seq data all the time.


MAJOR COMMENTS

* The authors claim that their method is "interpretable" (it's right in the title) but then does not demonstrate how it can be interpreted in practice. How is it more interpretable than a fully-connected network, a SVM, or a random forest (that have similar classification performance: Tables 1-2)? Unclear.

* The authors claim that their method performs well, but in fact it does not perform better than a fully-connected network (Table 3), which does not use either attention, or capsules, or LSTM.

* The paper is hard to follow, e.g. I could not fully understand what exactly is shown in Figures 2 and 3. Figures do not have meaningful captions, subplots and axes are not labeled, surrounding text is hard to read.
summary_of_the_review: I found the paper difficult to understand, classification results not convincing, and interpretability claims greatly overstated. Unfortunately I believe this is a clear reject.

Disclaimer: I have heard about attention and capsules but do not know how they work. However, I think this paper can be assessed even if one treats the model as a black box. Also, I do work with single-cell RNA-seq data all the time.
review_title: null

Meta-review: All reviewers believe that the paper is not ready for publication and clarity issue remain. All reviewers read the rebuttal responses, but they found that the paper wasn't revised during rebuttal, thus they retain their decisions.
Decision: Reject