Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors present a GAN for multiplexed imaging (MIBI-TOF) data called CCIGAN. They propose an interesting architecture design with protein-specific attention to find association between cell types and neighboring pattens and cell-cell interactions. They also propose new and biologically interpretable metrics including a reconstruction metric, projected EMD and regressing of expression on neighbors.

They present improved reconstruction of interactions compared to other models in the context of PD-1 and PD-L1 interactions. It would be great to extend the evaluation to other interactions and tissue types.

Overall the paper is well written, the application and especially the focus on cell-cell interactions is novel. The model is properly justified and evaluated, and there is a high demand for this framework in the multiplexed imaging field.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper applies the SPADE semantic image synthesis technique (with a custom attention mechanism) to MIBI-TOF data to examine hypotheses about cell-to-cell interactions in the context of an immune infiltrated tumor sample. I think that this is potentially an interesting application of GANs -- to generalize beyond specific gathered data and instead start engaging with counterfactual scenarios like "what effect does it have to add cell type X next to cell type Y". 

Unfortunately, I think this paper is not yet finished with regard to both (1) conveying sufficient motivation for the use of image synthesis for distilling biological insights and (2) evaluating the success or failure of the technique primarily in terms of generating useful biological insights. Simultaneously, there are enough red flags with regard to knowledge of the data generation and underlying biology that it's unclear if the authors could correctly sanity check any insights they extract from their model. 

More specific criticisms & suggestions

1) It would help to provide much clearer motivation for applying image synthesis to MIBI-TOF data. Section 1.2 skips straight from describing the MIBI-TOF instrument to “we made a new kind of GAN”.  Again in the beginning of the Related Work the paper states: "We are interested in the task of generating biologically consistent expression patterns of cellular proteins given a segmentation map of cell neighborhoods”. But, why are synthetic images interesting given that we can actually look at real MIBI-TOF data. I start getting a sense of why this model might be interesting or useful only in Section 5 — more rationale is required earlier in the paper. 

2) The evaluation section is hard to follow. I think it would be helpful to more clearly describe a larger set of biological phenomena that a practitioner would expect to see in the data, choose a single metric for each case, and show that that these phenomena are recapitulated. No one is going to trust a GAN to give them scientific insights unless they're very confident that all known / simple cell-to-cell interactions have a clear signal. 

3) "MIBI-TOF bombards a tissue sample with elemental metals tethered to respective antibodies for dozens of distinct cellular proteins and detects each to obtain image channels” — this is not an accurate description of MIBI-TOF, at least not the instrument I'm familiar with. Typically the tissue is first stained with antibodies tagged with heavy metals and the instrument then bombards the tissue sample with simple ions (like O2+), causing the release of metal ions. It seems very unlikely that bombarding anything with antibody/metal conjugates could be informative. 

4) "Tumor cells could be identified by markers such as pan-keratin and beta-catenin” — I guess in the context of a tumor sample beta-catenin could be over-expressed but it's present in pretty much every cell type, including lymphocytes (https://www.proteinatlas.org/ENSG00000168036-CTNNB1)

Small nits:

* Repeated use of “antibodies” in "Engineered antibodies for PD-1/PD-L1 antibodies” — maybe better to write “Antibodies which block the interaction of PD-1/PD-L1"

* "While MIBI-TOF is capable of 36 different markers, we discarded uninformative and irrelevant markers1 resulting in M = 24.” — I’m really surprised that someone put 12/36 uninformative markers in a MIBI-TOF panel. Aren’t these tagged antibodies expensive? Can we at least get a list of what got discarded?

* The model interpretability control seems weak in that lymphocytes are more likely to express MHC-I than tumor cells (which have a potential survival advantage from not expressing it) and tumor cells may actually have more dsDNA than lymphocytes due to changes in ploidy (or original differences in ploidy from e.g. liver cells). 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The manuscript proposed a new method to model the data generated by multiplexed ion beam imaging by time-of-flight (MIBI-TOF). Essentially, the model leans the many-to-many mapping between the cell types and different protein markers' expression levels. Compared with the other mainstream GAN methods, the authors show the proposed method, CCIGAN, can outperform them in terms of generating the expression map of different protein markers given the segmentation of cell types. The manuscript also has an in-depth discussion of the biological meaning of the learned model as well as the learned vectors. 

I personally like this manuscript a lot, considering the novelty and the thoroughness of the manuscript. However, I have the following concerns: 

Major concern (The score will be significantly improved if the authors can handle these two concerns during revision): 
1. My largest concern is how useful the proposed method is. It seems the wet experiments are mature. How can the computational synthesized image help the biologists then? Are there any cases that only the CCIGAN can do while the real experiment can not do? I am not an expert in MIBI-TOF, but I guess all the results in Section 6.2 can be easily achieved with the real data, right? 
2. CCIGAN is indeed better than the other methods, but would it be accepted by the biologists in terms of performance? Would they believe in the results? In fact, in the last column of Figure 2, we can see clear artifacts for the results generated by CCIGAN.


Minor concern:
1. What's the resolution of the MIBI-TOF and CCIGAN?
2. The introduction of the background could be further refined. The many-to-many mapping between different cell types and different protein markers is not emphasized explicitly. Readers can get lost easily. 
3. The data part is not clear. How many M*2048*2048 images do the authors have for training as well for testing? 
4. Why do the authors choose 64*64 as the image path size?
5. Can the model be generalized well for data collected in different experiments (i.e., different tissues) but from the same machine? Can the model be generalized well across different machines with the same imaging experimental setting?
6. Is the model sensitive to the preprocessing of the data, like normalization? As for as I know, the baseline expression level of tissue can vary significantly at different time points within one day. If the model is sensitive to that, it will affect the usage of the model.
review_title: Official Blind Review #2

Meta-review: This paper proposes a dedicated deep models for analysis of multiplexed ion beam imaging by time-of-flight (MIBI-TOF).

The reviewers appreciated the contributions of the paper but not quite enough to make the cut.

Rejection is recommended. 
Decision: Reject