Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed DISSECT, a regularization-based deep learning approach for bulk RNA-seq deconvolution that decomposes the bulk RNA-seq matrix into a cell type fraction matrix $X$ and an expression profile $S$. The authors introduced consistency regularization which utilizes a mixture of the true bulk RNA-seq and simulated RNA-seq data. The proposed learning paradigm includes two deep networks, MLP and VAE, estimating X and S, respectively. The authors reported the model performance for multiple bulk RNA-seq datasets and showed DISSECT outperforms the competing approaches in both cell fraction and gene expression estimation tasks. 


**Update after rebuttal:** I have read all reviewers' comments and the authors' responses. I still think the paper lacks enough contribution as a machine learning paper and my score remains unchanged.
strength_and_weaknesses: Strength:

- The authors suggested a simple regularization loss that factorizes bulk RAN-seq and estimate cell type-specific expression with limited number of cells.

- The authors reported their model performance for several RNA-seq data and demonstrated the outperformance of their regulation-based deconvolution in compared with other deconvolution methods, 


Weaknesses / Concerns / Comments:

- The paper does not have sufficient contributions. There is no justification for the consistency term and it is not clear to me that why such a simple linear formulation can improve cell type fraction approximation.

- Is there sample-to-sample correspondence between $B_i$ and $B_i^{sim}$?

- While bulk measurements and simulated dataset (based on single-cell measurements) consist of different sources of noise and biological variabilities, can you justify why a linear mixture of these two cell measurements can enhance cell deconvolution? 

- Is there any equation missing after the third equation, Section 2 (page 2)?

- In Section 2, 4th equation, why does the signature matrix which is a $c$ x $n$ matrix is a function of three indices, including sample id? Does not $S$ represent the expression profiles across cell types? 

- Section 2.1.2, Eq. 4 should be written as $x_i s_i$, not $s_i x_i$.

- Is the number of cell types given to the model apriori? How does the deconvolution performance change in the absence of having the true number of cell types in the bulk RNA-seq data?

- In Figure 1.a, I think the first “+” operation should be “=”.

- Why do you use a variational approach to learn signature matrix? Why not just an MLP, similar to $X$ matrix estimation? Is there any plan to use the latent factors? 

- How many genes are profiled in datasets in Tables 1 & 2? Are you using the same subset of genes in Eq. 3?

- I think reporting MSE (which is reported in the supplement) is more informative rather than reporting correlation between estimated variables and true ones. My suggestion is to report MSE results in the main text.

clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is well organized. There are some typos (and after question, “+” in Figure 1) in the text that need to be fixed.

Novelty:

The paper does not have any methodological originality and lacks sufficient contribution.

Reproducibility:

The results seem to be reproducible. 

summary_of_the_review: I think the paper has limited contributions to the machine learning field and the manuscript is not sufficient for a ML focused venue like ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors proposed DISSECT as a novel deconvolution framework estimating both celltype fractions and celltype specific gene expression. DISSECT leveraged both the training data with ground-truth proportions and simulated data based on scRNA-seq reference. It can be seen as an augmentation strategy which mixes the training data with synthetic data or a regularization framework that solves the domain shift problem between the training domain and reference (scRNA-seq) domain. Authors completed extensive evaluations over a large collection of datasets and compared DISSECT with other competitive methods. Authors highlighted the superior performance of DISSECT and provided a possible extension on applying DISSECT on the spatial transcriptomic data which was a new application scenario.
strength_and_weaknesses: The manuscript was well written and included major players in the field of celltype deconvolution to compare in the computational experiments. The shared code repository was helpful to better understand the algorithm. There are some experiment details that need to be clarified as follows.

Major Concerns

1.	(section 4.3.2) The experiments estimating celltype-specific gene expression was only conducted on the simulated bulk RNA-seq data. This is insufficient to evaluate how DISSECT behaves in practice. One possible source of verification data can be imaging-based spatial transcriptomics data [1], as authors have tried applying DISSECT on the spatial transcriptomic data.

2.	(section 3) Authors constructed the Linear MLPs removing the consistency loss which may be not a fair comparison. I am wondering how Linear MLPs would behave if the consistency loss was incorporated.

3.	(section 2.2.1) The first training phase was purely based on the simulated examples. I am curious to see what the performance of DISSECT is right after this simulation-based training phase. This can be seen as an ablation test to check how mixing the training data with training data helps improve the deconvolution performance.

4.	I didn’t find descriptions on how the training/validation/test sets were split for each dataset. Could you elaborate on this?

Minor Concerns

1.	Scaden [2] included both SDY67 (called PBMC1) and Monaco (named PBMC2) for comparison in their manuscript. Their reported correlations were different compared to those reported in Table 3. I am wondering what could possibly lead to this difference.


[1] Cable, Dylan M., et al. "Cell type-specific inference of differential expression in spatial transcriptomics." Nature methods 19.9 (2022): 1076-1087.

[2] Menden, Kevin, et al. "Deep learning–based cell composition analysis from tissue expression profiles." Science advances 6.30 (2020): eaba2619.

clarity,_quality,_novelty_and_reproducibility: Authors need to clarify some details on the experiments (see comments above).
summary_of_the_review: The manuscript is not ready yet to be presented in its current form. Concerns raised above need to be addressed.


Response to authors

Thank you for the detailed response. Most of my concerns have been addressed properly. I still have two minor comments:
1. I am confused that the consistency loss will be zero for linear MLP as I follow the definition of consistency loss in equation 6.

2. Thanks for including appendix table 13 and 14. It will be great if they can be merged to table 3 and table 4 as one separate column accordingly. Also the column name of the first field in table 13 should be updated from "Dataset" to "Metrics" or something similar.

I will keep my scores for now.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a new model for cell deconvolution task by introducing consistency regularization. 

strength_and_weaknesses: Strength

- The proposed model was extensively tested against existing models using various datasets.
- The proposed model showed superior preformance among compared.

Weaknesses

- Whole contents of this paper is focusing on a particular task in cell biology. Very domain specific.
- It is unclear how the model was trained. Which dataset was used?

clarity,_quality,_novelty_and_reproducibility: The quality, clarity and originality of the work are OK.

summary_of_the_review: The authors propose a new model for cell deconvolution task by introducing consistency regularization. The paper really focusing on a particular task in cell biology thus more specific venue is suitable.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present DISSECT, a method for estimating cell fraction and gene expression from bulk RNA sequencing data. The primary innovation in DISSECT is the introduction of a regularization term during training that encourages similar expression profiles between simulated bulk data (drawn from scRNA-seq data) and real bulk RNA-seq data. A multilayer perceptron is used to estimate cell-type proportions and an autoencoder is used to estimate cell-type specific gene expression profiles. The authors evaluate their method on several real bulk RNA-seq datasets of PBMCs.
strength_and_weaknesses: I believe the authors are pursuing a good research direction within the field of RNA-seq deconvolution, with the incorporation of a consistency loss into training, and this problem is important for biological studies, but this field has been studied extensively and to make a contribution to it, a more thorough investigation is needed. 

Strengths:
- The problem of RNA-seq deconvolution is of great importance for many biological studies.
- Adding a consistency loss is a good insight for addressing the issue of differing characteristics between scRNA-seq and bulk RNA-seq.
- The authors are aware of other primary competing methods in the field of cell-type deconvolution.
- The authors have clearly laid out the logic of their reasoning about the consistency loss and the math details are explained clearly and are easy to follow.
- Overall, the paper is well-written.

Weaknesses:
- As mentioned, there has been extensive work toward solving the problem of deconvolution. The authors have only applied their method to datasets of PMBCs, and if I understand their training procedure correctly for cell-type estimation, only used the PBMC8k dataset as the single-cell reference (or did they use all 4 from Table 1?). Even so, these datasets are from one individual each (per the description in Table 1), thus the datasets are limited in their ability to capture possible person-to-person or technical variation. Thus, it could be that their advantage comes largely from this limitation. In comparison, MuSiC compared to several different tissue types and used scRNA-seq data of 10-20 individuals. To make a compelling case, the authors need to apply to more tissue types and use a more diverse set of reference single-cell datasets. (The authors include results for some other types of data in the Appendix, but need to provide quantitative comparisons and an explanation of the results.)
- In particular, the authors should consider doing an evaluation that exactly compares to some of those reported in MuSiC or (Avila Cobos et al., 2020); this would give greater confidence that their evaluation matches what is reported in those works.
- Furthermore, as was shown in the benchmarking study of (Avila Cobos et al., 2020), the normalization procedure can have dramatic effects on the results of deconvolution. The authors comment in the Appendix C.3 that their procedure is standard for Scaden, but given the potential dramatic impact, more reasoning should be provided for the other methods as well, especially in light of the analysis of (Avila Cobos et al., 2020).
- The incorporation of the consistency regularization, while sensible and a good consideration, is not in itself a significant algorithmic advance, at least for the scope of ICLR.
- There are other methods that are not referenced: Bisque (Jew et al., Nature Communications 2020) and DWLS (Tsoucas et al., Nature Communications 2019). The authors are correct that MuSiC and Cibersort are the main established methods, but still some reference should be made to these methods to distinguish theirs and a comparison against them would strengthen their work.
clarity,_quality,_novelty_and_reproducibility: - As mentioned above, there is some ambiguity about how the cell-type estimation was performed. It is stated that the PBMC8k dataset was used as the single-cell reference dataset for deconvolving the bulk RNA-seq data in Table 1. This produced the results in Table 4. Were the other scRNA-seq datasets not used then for DISSECT or any other method to produce Table 4? Was this then the only dataset used for the MuSiC? Again, if so, this would be a disadvantage for MuSiC, since it is designed for multi-sample data.
- The incorporation of the consistency regularization, while sensible and a good consideration, is not in itself a significant algorithmic advance. A more extensive evaluation would help overcome this weakness.
- I do commend the authors on the clarity of their work and believe it has potential if further developed.
summary_of_the_review: The current state of the evaluation and its limitations, as mentioned above, are the primary weaknesses of the work in its current state. A more extensive evaluation, following the analysis done in these other works in deconvolution, would greatly strengthen the paper. But in its current state, it is not ready for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a new model for cell deconvolution task by introducing consistency regularisation.
The main weaknesses raised by the reviewers is the lack of novelty. There is limited technical novelty. The application, while relevant, is also not a novel ML application, but many related ML methods exist.

justification_for_why_not_higher_score: All reviews recommend rejection of the paper.
Decision: Reject