Exploring Cellular Protein Localization Through Semantic Image Synthesis
Authors: Daniel Li
Qiang Ma
Andrew Liu
Justin Cheung
Dana Pe’er
Itsik Pe’er
Venue:  (2020)
Keywords: Computational biology
image synthesis
GANs
exploring multiplex images
attention
interpretability

Abstract: Cell-cell interactions have an integral role in tumorigenesis as they are critical in governing immune responses. As such, investigating specific cell-cell interactions has the potential to not only expand upon the understanding of tumorigenesis, but also guide clinical management of patient responses to cancer immunotherapies. A recent imaging technique for exploring cell-cell interactions, multiplexed ion beam imaging by time-of-flight (MIBI-TOF), allows for cells to be quantified in 36 different protein markers at sub-cellular resolutions in situ as high resolution multiplexed images. To explore the MIBI images, we propose a GAN for multiplexed data with protein specific attention. By conditioning image generation on cell types, sizes, and neighborhoods through semantic segmentation maps, we are able to observe how these factors affect cell-cell interactions simultaneously in different protein channels. Furthermore, we design a set of metrics and offer the first insights towards cell spatial orientations, cell protein expressions, and cell neighborhoods. Our model, cell-cell interaction GAN (CCIGAN), outperforms or matches existing image synthesis methods on all conventional measures and significantly outperforms on biologically motivated metrics. To our knowledge, we are the first to systematically model multiple cellular protein behaviors and interactions under simulated conditions through image synthesis.

Under review as a conference paper at ICLR 2020
EXPLORING CELLULAR PROTEIN LOCALIZATION THROUGH SEMANTIC IMAGE SYNTHESIS
Anonymous authors Paper under double-blind review
ABSTRACT
Cell-cell interactions have an integral role tumorigenesis as they are critical in governing immune responses. As such, investigating specific cell-cell interactions has the potential to not only expand upon the understanding of tumorigenesis, but also guide clinical management of patient responses to cancer immunotherapies. recent imaging technique for exploring cell-cell interactions, multiplexed ion beam imaging by time-of-flight (MIBI-TOF), allows for cells to be quantified in 36 different protein markers at sub-cellular resolutions in situ as high resolution multiplexed images. To explore the MIBI images, propose GAN for mul- tiplexed data with protein specific attention. By conditioning image generation cell types, sizes, and neighborhoods through semantic segmentation maps, we are able to observe how these factors affect cell-cell interactions simultaneously in different protein channels. Furthermore, design set of metrics and offer the first insights towards cell spatial orientations, cell protein expressions, and cell neighborhoods. Our model, cell-cell interaction GAN (CCIGAN), outperforms matches existing image synthesis methods all conventional measures and significantly outperforms on biologically motivated metrics. To our knowledge, we are the first to systematically model multiple cellular protein behaviors and interactions under simulated conditions through image synthesis.
INTRODUCTION
1.1 BIOLOGICAL ROLE OF CELLULAR PROTEINS
Cell-cell interactions within the tumor microenvironment have been implicated in many facets of cancer pathogenesis and treatment. Most prominently, tumor cell evasion of immunc surveillance (iiag al. 2019), tumor metastasis (Nishida-Aoki & Gujral 2019). and efficacy of cancer im- munotherapies (Lau et al., 2017) have all been closely linked to the relationships between immune and cancer cells. These types of cell-cell relationships are generally governed by the interactions of cell surface proteins which drive cell behavior, gene expression, and survival. One of the most prominent examples of cellular proteins influencing disease progression is the case of PD-I/PD-LI. PD-L1 is a protein often overexpressed on tumor cells and has the capacity to bind PD-1 on local T cells downregulate their anti-tumor immune responses (Iwai et al. 2002).
Antibodies which interrupt the interaction of PD-L1 and PD-1 and allow the immune system to attack tumor cells, have become clinically influential treatments for a variety of cancers (Pardoll) 2012). This example highlights the value of accurately predicting cellular protein patterns which play key roles in disease processes. Exploring protein localizations sents challenge for which deep learning models are uniquely suited. However, to our knowledge no image-based deep generative models have utilized semantic image synthesis produce accurate predictions of these biological phenomena.
multi-cellular system repre-
1.2 FRAMEWORK AND NOVELTY OF A PREDICTIVE MODEL
meaningful exploration of cell-cell interactions, particularly in the tumor microenvironment, re- quires thorough understanding of the proteins expressed and within cell and its neighborhood. Multiplexed ion beam imaging by time-of-fline flight (MIBI-TOF) represents a novel technology that can accurately quantify and spatially resolve cellular protein expressions at the single cell level within

---

Under review as a conference paper at ICLR 2020
tissue samples. Given tissue sample that is first stained with protein-specifer antibodies tethered to elemental metals, MIBI-TOF bombards the sample with atomic ions (i.e. O22) from primary ion beam. This causes the release of clemental isotopes and tissue-specific material which can be quantified a mass spectrometer (Angelo et al.) 2014). The cellular proteins characterized by this technique indicate specific cell types (i.e. immune cells, tumor cells), cell status (i.e. markers of pro- liferation), and immunomodulation. Figure (A) shows an example spatial orientation of cell types and some selected cellular protein expressions. Here, we propose novel protein based attention
(A) Examples of TOF Data
(8) Direclly Manipulated Segmentations
Segmentation Patches
Pan-Keratin
Iterative
Non-ifiitrteed
(compartieettaliee)
Infilrated (mixed)
Endothekill
Tumor
Keratin bumner
CD4T
CD8
CD3
Macrophages
Figure 1: (A) Example segmentation patches (left column) and protein expressions (other columns). (B) Examples of how segmentation maps can be synthetically altered to pose different counterfactual biological scenarios for in silico hypothesis testing.
mechanism for a convolutional Generative Adversarial Network (GAN) with the capacity to provide accurate conditioned predictions of cellular protein localizations. The model, Cell-Cell Interaction GAN (CCIGAN), learns many to many mapping between different cell types and different protein markers. CCIGAN is trained on semantic segmentation maps of cell tissue samples to identify and associate cell type, shape, and the identity of its cellular neighbors with celfular protein expressions for each cell. The network was then applied to simulated segmentation maps to understand how cells vary their protein localizations.
CCIGAN demonstrates its ability to probe elements of oncologic processes by:
Independently recapitulating established patterns biological phenomena, thereby demon- strating the accuracy of its predictive power.
2. Yielding quantitative and spatial information on specific cellular protein modulations as result immune cell cancer cell interactions.
3. Allowing for biological data to be generalized beyond traditional in vitro scenarios where specific, rare, or hypothetical cell-cell events can be posed in counterfactual scenarios and the results of their interactions predicted.
The ability for CCIGAN generate subcellular protein predictions represents step forward in the ability to understand cellular relationships within microenvironment. Rather than assessing in vitro incidence of cell interaction phenomena as is done by MIBI-TOF, CCIGAN allows for hypothetical biological situations to be generated. Figure I (B) illustrates examples how a segmentation map can be directly manipulated to pose various biological scenarios. In other words, individual cellular re- sponses to challenge or proximity of cells of any identity can be assessed without having to seek this specific occurrence within the available biological tissue sample. These capabilities allow the model to provide insight on hypothetical interaction events which shed insight on disease pathogenesis and which cannot be feasibly achieved by biological investigation alone.
MIBI-TOF is labor intensive process which can make the data collection process cumbersome. Deep learning models such as CCIGAN have the potential to be useful tools which can extend on MIBI-TOF capabilities and result in more rapidly synthesized data. CCIGAN, once trained on MIBI-TOF data sets and properly replicating known cell-cell interaction patterns, can be used as robust means to provide high throughput readouts myriad single cell scenarios which may normally take many iterations of MIBI-TOF experiments to investigate. Recent work by (Wu et al. 2019) demonstrated the utility of conditional GANs for adding depth dimension to images;
2

---

Under review as a conference paper at ICLR 2020
demonstrate learning contextual cell-cell interactions, facilitating rapid hypothesis testing to assess biological environments. This provides valuable insight into a complex system that normally would have taken resource-intensive wet lab experiments to interrogate.
2 RELATED WORK
We are interested in the task of generating biologically consistent expression patterns of cellular proteins given a segmentation map of cell neighborhoods. Specifically, want to learn a gen- erative model that simultaneously produces high quality maps of protein expression for individual cells that are probabilistically consistent when conditioned on the same factors, e-g- similar cell neighborhoods should produce similar expression patterns. Such generative models typically take the framework of a generative adversarial network (GAN) (Goodfellow al., 2014 Brock et al. 2018) or a variational autoencoder ( (VAE) (Kingma & Welling|2013)
Within these generative modeling techniques, image translation focuses on learning many to many mapping to transform data from one domain to another. One approach is to model the translation task, known as image synthesis, with conditioning such as Pix2Pix (Isola et al. 2016), Pix2PixHD (Wang et al., 201 and CycleGAN (Zhu et al., (017). By using image synthesis, model able to distill more information from a segmentation map to various protein channels. As MIBI-TOF measures information such as protein localizations at subcellular resolution, framing the problem through an image synthesis framework allows a model to see how predictive neighbor- hoods and cell types are of cell's phenotype.
Recently, Park et al. (2019) proposed spatial adaptive normalization (SPADE) to synthesize images using segmentation maps by learning fully convolutional normalization parameters based their segmentation conditioning. For each layer, semantic information is retained by allowing the network to learn from the segmentation map directly and modulate the current layer. They demonstrate impressive quality and diversity in generated images, especially modeling objects in context. Using SPADE normalization, CCIGAN able condition on surrounding cell neighbors and capture cell- cell interactions in a local receptive field. Despite image generation, there has been no work done towards understanding such generations in context, particularly in biological images.
Attention layers (Vaswani et al.) 2017] Wang et al. 20188] Zhhang et al. 2018) have also been added to generative models with great success. Self-attention was initially proposed in machine translation tasks to help model long distance dependencies that occur frequently in language (Sutskever et al.) 2014), which led to the idea of external memories as a persistent state to model long range depen- dencies (Sukhbaatar et al., 2015). While convolutional neural networks are apt at exploiting local structures of patches, they may struggle to model global structures. For this reason visual attention allows the network to enhance activations in interesting parts of image. Goodfellow et al. (2014) and Brock et al. (2018) both achieve state-of-the-aar unconditional generative modeling using self attention GAN (Zhang al. 2018). We propose a specialized attention module conditioned on different proteins to mimic real world protein markers.
3 DATASET
3.1
MULTIPLEXED ION BEAM IMAGES (MIBI)
All experiments are performed on data obtained through MIBI-TOF characterized tissue samples collected from triple-negative breast cancer (TNBC) patients. By simultaneously imaging over 36 protein markers, MIBI-TOF is able to identify cell type as well as provide detailed information of sub-cellular structure, cell neighbors, and interactions in the tumor microenvironment across these different marker settings. Each of these markers m E {I,..., J. given as a channel taking on real values continuous in [0, 1] at each (I, coordinate, demarcates different cellular subtype and furthermore, is indicative of the functional properties of a cell. While MIBI-TOF is capable of 36 different markers, we chose markers exhibiting cellular protein localization and disregarded markers that were blank or used simply for cell-typing resulting in M = 24. A full list of these markers is given in A.8
Some channels were completely blank and others were indicator channels, i.c. CD45, binary indicator for immune cell.
3

---

Under review as a conference paper at ICLR 2020
Segmentation
Pan-Karattin
de-DNA
CD3
CDB
HLA-Caas
PD-L1
Real Data
CCIGAN
SPADE
PH2PaHD
OydkGAN
Figure 2: Examples generated from a segmentation for certain channels for different models. The segmentation patch is the one hot encoded patch collapsed and colored into channel. The horizon- tal labels represent protein markers and the vertical labels are each of the generative models.
The MIBI-TOF data collected from the TNBC tissue allows for the differentiation between wide variety of cell types within the samples. For instance, cells positive for the marker CD3 could be identified as cells, and then subdivided into cytotoxic or helper lineages by the presence of markers CD8 or CD4, respectively. Tumor cells could be identified by markers such nan-keratin and overexpressed beta-catenin. Along these lines, a wide variety of cellular proteins identified by MIBI-TOF could characterize cell interactions well as immunomodulatory processes occurring within the microenvironment.
MIBI-TOF data is fundamentally different than typical RGB images. This poses unique challenges in image modeling and characterization, Such challenges stem from each marker being conditionally expressed the cell type in its respective channel Using simplified 3 channel multiplex setting, a T cell expresses signals in the CD3, CD8 channels (indicators for immune cells) but not in a pan- keratin channel (indicator for tumor cells). Another problem is the sparsity of the data, meaning either some expressions for rare cell types are rarely observed or have weak signals. For example, Figure 2 includes segmentation of CD8 cell (orange, top right corner), where other models fail in generating correct CD3 expression, if at al Lastly, the noisy nature of the data leads to inaccurate cell type classifications, creating inconsistent pairs of labels and outputs during training.
These issues make it especially difficult for an RGB multihead decoder to output multiple channels in biologically accurate manner. Without addressing multiplexed data, a decoder would equally attend to every location of the current latent representation, even if it is irrelevant to the current protein. Furthermore each protein channel in the output has its own sensitivities to signal intensi- ties and noise, suggesting each channel requires unique prior and that equal attention would be problematic. It follows that special care must be given towards modeling specific channels and multiplexed nature of MIBI-TOF images.
3.2 DATA PROCESSING
MIBI-TOF images represented as high dimensional tensor T E R (M,20448,008)) These images then further processed at a cell by cell basis into Y E (MM,61,64) patches, where cell is at the center of the patch along with its neighbors. Next, we construct semantic segmentation maps E (CC++16166)) where a vector S... is one-hot encoded based on a cell type 17, and the C + 1-th channel denotes empty segmentation space. The data is train-test split 9:1 ratio the MIBI-TOF image level to avoid cell neighborhood bias. We also use a synthetic test set where cells
Thhe dark green cell is extremely rare, classified "other immune" and noise class label.
4

---

Under review as a conference paper at ICLR 2020
and their neighbors sequentially modified to observe how varying cell type, position, and size affects the progressive changes in protein localizations
METHODS
4.1
MODEL ARCHITECTURE
Z
S
R
R
Attention Module
R
Output
128, 0,0
120, 10,
120, 32,
120, 64, 64
SPADE ReiBlocka
Figure 3: CCIGAN
We use SPADE residual blocks (Park et al. 2019) as our generative backbone and DCGAN's dis- criminator's architecture (Figure 3 A.II) (Radford al. 2015). Park al. (2019) have shown SPADE to be an effective way to inject conditioning into generative model. The SPADE nor- malization layer serves as replacement for previous layer normalization techniques. Instead of learning universally shared per channel affine transformation, like in Batch Normalization (loffe Szegedy 2015) or Instance Normalization (Ulyanov et al., 2016). SPADE learns predict affine transformations based on segmentation maps; each feature is uniquely transformed based its cell type, size, and neighboring cells. The ability for SPADE to modulate activations based on the context of adjacent cell segmentations allows the network to effectively model the behaviors and interactions of cells. The input CCIGAN is a noise vector z E 1218 and segmentation map S. denotes a linear layer R 128 + 2048 R' are feature map representations from SPADE resblocks and X denotes the final output of M cell expressions. Below, each layer's output dimensions given next to their respective equations.
E R (128,4,4) f(=)
(1)
R E (128,32.32) SPADE_RFSHIKKR" S)
(4)
SPADE.RESBLKFZ S) R I (128,8,8)
(2)
R' E R (128,64,64) SPADE (5)
R² R (128.16.16)
SPADE RESBLK (3)
R (M.61.64) ATTENTION(R', S) (6)
4.2 ATTENTION MODULE
Our architectural contribution is a protein marker dependent attention module in the final output layer. The goal of the attention module is condition the final output of channel protein marker m and S's cell types. For example the protein marker, pan-keratin Mypk is expressed exclu- sively in tumor cells but not in other cells. Appropriately, tumor cells and ignore irrelevant cells in S for map. To replicate a marker searching for specific cell types that express it, we define learned persistent vector for each marker denoted by SmEM R that undergo series of operations (Figure 4 with the final feature map representation attending to vn's mn's specific cell types. It is also worthwhile note that these persistent vectors offer a degree of model interpretability that mimic real world markers. The current input dimensions to the attention module are g (125, 64,64) following the last resblock Rd and m indexes from 1,.,,M M.
attention mechanism should attend to
"The test dataset will be released along with trained models for reproducibility after the reviewing period.

---

Under review as a conference paper at ICLR 2020
Expand channels independently
2
Modulate current rapresentation with with SPADE normalization
Outar product with cornepoondnng search vector pixel dimension
C
Scale
Sme M
C
C
C
KMA 64,
64
convolution
apply
5
Add newly learned attantion to residual fenture maps
A
B M....M
1,64,
84,
M, 84,
M. 64,
Figure 4: Attention module. This illustrates an instance, focusing on the light blue block Ct.
(.M.64.64) = CONV2D(R*)
(7)
A, E [2(4)K,,4664 = 8 8m
(10)
C E R (4,6) = CONV2D(O)
(8)
B (R(6616)) =o(CONV2D(A.)) (11)
C. G.R
S) (9) X EER (M...8)) = BI.....
(12)
Shown in Figure after R a bottleneck convolution applied to match the original data's di- mension O (step 1), which is used in a residual manner with the final output. Intuitively this stage, O's feature maps resemble the target Y, but we wish to further refine the output channels. We convolve into MK channeled features for each protein marker where K 8. Considering each C, where i E [1....M as group of K channels, the model spatially adaptive normal- izes each C, and computes an outer product with the corresponding persistent vector 8; and C., The resulting matrix is flattened and convolved (with a kernel size of 1 on the pixel level) from ( x 64)
A;
g! (1,644,64 followed by sigmoid o(-) activation. Lastly, the attentions B BI....M M. are added to to obtain the output X.
Initially, the model has priors over the interaction of protein markers and cell types. The pro- posed outer product attention layer (outer product and x convolution) excels at modeling these relationships and interactions between specific markers and cell types. By using outer product, the model forces attention at pairwise pixel level comparison for all combinations of elements be- tween 8mn and A, As training progresses, both the learned features the learned persistent vectors 8m. improve, in turn allowing the learned x I convolution to reason about positive negative relationships from the pairwise pixel combinations.
segmentation patches and
4.3 IMPLEMENTATION DETAILS AND TRAINING REGIMEN
Our implementation The discriminator's input the output of the generator concatenated with the segmentation patch [X S) and [Y, S] for the ground truth. Finally CCIGAN uses ADAM (IrG = 0.0004, Irp = 0.0001) with GAN loss and feature matching loss. Full training details and loss functions given in A.2
of the generator applies Spectral Norm to all layers (Miyato et al. 2018).
5
EVALUATION
To conduct fair experiments, all models were optimized, tuned, and set with similar parameters. They were also taken from their official online implementations and trained for 120 epochs or until convergence (max 150). CCIGAN is identical to our designed SPADE comparison baseline with the exception of the attention module. Three experiments were conducted to validate the trained model's utility in generating biologically meaningful cellular proteins in the tumor microenvironment and ability to recapitulate and quantify previously established biological phenomena. Each subsection describes the experiment and the relevant metrics used in evaluation. Full mathematical definitions given in section A.3

---

Under review as a conference paper at ICLR 2020
5.1 IMAGE EVALUATION AND RECONSTRUCTION
First, we use the following evaluation metrics in order to compare with baseline results: adjusted L1 and MSE score, 11 and MSE score, structural similarity (SSIM) index (Wang et al.) 2004) and cell based mutual information (Mf) shown Table 1 Bolded scores indicate the best scores.
Equations and explanations are given in A.3.1
Metrics
CCIGAN
SPADE
Pix2PixHD
CycleGAN
Adjusted I, Score
0.613
0.618
0.875
4.745
1,1 Score
0.594
0.602
0.745
3.959
Adjusted MSE Score
0.026
0.031
0.061
1.841
MSE Score
0.026
0.031
0.055
1.523
SSIM
0.810
0.802
0.709
0.394
Cell Mutual Information
10.46
10.25
9.26
7.96
Table 1: Comparison of conventional reconstruction metrics between different models.
Experiment
Metrics
CCIGAN
SPADE
Pix2PixHD
CycleGAN
COM Score
10.46
10.69
12.81
13.38
PD1-PDL1 (Tumor)
Random COM Score
12.27
12.16
12.96
12.45
EM Distance
151.56
-12.83
-12.24
-0.01
Positive EMD
278.38
149.81
162.97
0.004
PDI-PDLI (Tumor)
Projected EMD
115.77
1.83
11.89
-0.008
Random EMD
-123.43
-83.11
-12.50
-0.006
EM Distance
-22.68
-18.88
-92.08
0.001
PD1-PDL1 (Endothelial)
Positive EMD
35.92
22.46
109.82
0.01
Projected EMD
-9.29
-9.9
-70.96
0.003
f-test
-1381
2.68
25.20
14.27
Tumor-CD8 (Exp.)
p-value
< 10-8
0.007
10-8
10-8
t-test
1.32
2.21
37.99
6.60
Tumor- Tumor (Control)
p-value
0.187
0.027
< 10-8
< 10--
Table Biologically motivated experiments. Random, Endothelial, and Tumor-Tumor experiments controls.
5.2 PD-1/PD-LI RELATIONSHIPS BETWEEN T CELLS AND TUMOR CELLS
In the first step of biologically verifying the accuracy of CCIGAN, PD-I/PD-LI expression relationships between CD8 T cells and tumor cells were assessed. Many T cells located within tumor microenvi- ronment have upregulated expression of PD-1 sug- gesting that the tumor milicu exerts influence on the protein localization and expression of infiltrating T cells (Ahmadzadeh et al. 2009). We assessed if in- creased expression of PD-LI neighboring tumor
Segmentation
PD-1
Histogram
t=2
cells would result in increased directional PD-1 ex- pression in CD8 T cell (Figure 55). We also deter- mined if there was shift the cell surface localiza- tion of the PD-1. To do so. we computed the Earth Mover's ' Distance between CD8 T cell's PD-1 ex- pression (represented as histogram on a cell's polar coordinates) and itself before and after in different tumor scenarios. In addition to the mass, we com- puted the expected center of mass (COM) of PD-1 in a CD8 cell with respect to the neighboring tu- mor PD-L1 expressions. Equations and explanations are given in A.3.2 and |A.333] Additional figures il- lustrating this process given in A.4
t=3
t=4
Figure 5: Example illustration of how CD8 T cell's (orange) PD-1 histogram changes as function of iteratively added tumor cells.

---

Under review as a conference paper at ICLR 2020
We expected that the properties of the PD-L1 expressing tumor cell would result in increased directional PD-1 expression in a neighboring cell. As a control, a PD-1 expressing T Cell was surrounded with endothelial cells (a normal tissue lining cell). As shown in Table 2 our initial hypothesis was confirmed in CCIGAN's predictions, wherein the presence of a PD-L.T expressing tumor cell, PD-1 expression in the CD8 T cell increased and moved towards the PD-L1 COM. On the other hand, the endothelial cell presence yielded no effect on the T cell's PD-1 expression, a result which is biologically expected. While both tumor and endothelial cells express PD-LI, the tumor microenvironment exercises more complex suppressive effects on CD8 T cells, which are more likely to modulate the T cell expression than endothelial cells. These findings verify CCIGAN captures the biological relationship of tumor cells inducing immunomodulatory changes neigh- boring cell. None of the other models succeeded in capturing this protein relationship. This serves to highlight the capacity of CCIGAN to recapitulate established cell interaction phenomena within the tumor microenvironment.
5.3 PAN-KERATIN AND CD8 EXPRESSION
Secondly, we assessed the effect of immune cell presence on tumor cell status markers. Keratins are a class of intracellular proteins that play an important role in ensuring cell structure. Further- more, when CD8 cells mediate tumor killing, they release enzymes which cleave the tumor cell's pan-keratin, disrupting the tumor cell structure (Oshimal 2002). We explored how the presence of neighboring CD8 T cells to a tumor cell would affect pan-keratin levels within the tumor cell, hypothesizing that T cell mediated tumor killing would result in drop in pan- pan-keratin tumor expres- sion.
In this experiment, we used Student's f-test the statistical hypothesis test evaluate the cor- relations between the pan-keratin expression in tumor cells and the area/number of CD8 T cells in contact with the tumor cell surface. Given generated pan-keratin channel X; E IR (H.WW) and the segmentation map channel for CD8 T cells S, E (2(HW),, we compute the total area of the cells a, - - ΣW == S., and the total expression level of pan-keratin e; = >i Σw X,. We then regress (ei i=1 on {ai and assess significance of the slope using a (-test against the null of no change pan-keratin expression as a function of CD8 T cell-tumor contact.
CD8 T cells were placed adjacent to a tumor cell in increasing number, similar to Figure 1 (B)'s first row and Figure experiment. was observed that the presence of neighboring CD8 T cells to a tumor cell tended to decrease the pan-keratin expression in the tumor cell. This effect became more dramatic as the number of surrounding CD8 T cells was increased AA.5). As a control, when the tumor cell of interest was surrounded with adjacent tumor cell(s) instead of CD8 T cells, no change was noted in that tumor cell's pan keratin for CCIGAN. Table 2]s fourth row shows the result of the 1-test where the slope of the linear regression of pan-keratin expression with respect the number of surrounding cells is compared to a V 0 flat baseline. A statistically significant difference between the slope of the linear regression for the T cell scenario vs. the baseline was found in the main experiment, but was not present in the control scenario. These results indicate that the T cell presence mitigates a decrease in pan- pan-keratin expression the tumor cell, which suggestive of T cell mediated tumor killing. Other models reported contrasting results where a significant change in the tumor cell pan-keratin expression was incorrectly reported under the control conditions. The resulting graphs are given in A..5) Further protein interaction patterns identified by CCIGAN are also reported in A.777 The findings from this experiment serve to highlight the robustness and accuracy of CCIGAN compared to existing image synthesis techniques in probing cell-cell interactions.
5.4 TUMOR INFILTRATED AND COMPARTMENTALIZE] MICROENVIRONMENT
As a final biological evaluation, we consider the variability of PD-L1 expression in immune and tumor cells across TNBC patient groups. Keren et al. (2018) determined that in situations of mixed tumor-immune architecture, where immune cells freely infiltrated the tumor, the tumor cells pre- dominantly expressed PD-L1. Conversely, in situations of compartmentalized tumors, where there is greater degree of delineation between immune and tumor cells, macrophages were the predom- inant source of expressed PD-LI, particularly at the tumor boundary. Figure (B)ss row 2. 3 show examples of how directly manipulated segmentation maps can simulate the two microenvironments.
8

---

Under review as a conference paper at ICLR 2020
We used CCIGAN to predict on 200 directly manipulated mixed and non-mixed tumor environment segmentation patches. Similar to Section 5.3's experimental settings, we then compute the average expression of specific marker for the cells of interest for all patches. For each experiment, use endothelial cells as control cells to show our result has biological significance.
These findings were recapitulated by CCIGAN in Table 3) For patient with mixed tumor environ- ment, when trained with mixed patient samples, CCIGAN reported increased PD-L1 expression on tumor cells. Furthermore, CCIGAN was able quantify this difference in expression at the single cell level, reporting tumor to macrophage PD-LI expression ratio (bolded) of approximately 3.2 and 1.75 for patients A and B respectively.
Experiment
Microenvironmenns
Patient A
Patient B
cell /TTmoo/ Macrophages
0.01886
0.00131
PD-1 (elll))
T cell Endothelial / Macrophages
0.00558
0.00107
T cell / Tumor Macrophages
0.00649
0.00100
PD-L1 (Tumor)
Endothelial / Tumor Macrophages
0.00279
0.00046
T cell Tumor Macrophages
0.00204
0.00057
PD-LI (Macrophages)
T cell Endothelial Macrophages
0.00068
0.00047
Table 3:
Average PD-I/PD-LI expression on the mixed tumor environment. The bolded cells indi- cate which cells are being measured.
Conversely, when trained with compartmentalized patient samples, CCIGAN reported increased PD-LI expression on macrophages adjacent to tumor cells as compared to macrophages adjacent to normal endothelial (inert) cells for patients C and D. This difference was quantified as ratio of PD- L1 expression of tumor-adjacent macrophages to endothelial-adjacen macrophages, approximately 1.85 and 2.7 for patient C and patient D respectively. Moreover, using the trained compartmentalized model to predict on mixed segmentation patches, CCIGAN still reports a 26% (patient C) and 19% (patient D) increase of macrophage PD-L1 expression when compared to mixed microenvironments (Table 4
Experiment (Macrophages)
Mieroenvironments
Patient C
Patient D
T cell / Tumor Macrophages
0.00408
0.00608
PD-LI (Compartmenntlizzed)
T cell Endothelial Macrophages
0.00220
0.00225
PD-Li (Mixed)
cell Tumor Macrophages
0.00324
0.00510
Table 4: Average PDL1 expression of macrophages//monocytes on the compartmentalizee tumor environment.
The PD-L1 ratios for the above two scenarios indicate that CCIGAN has appropriately captured previously reported biological outcomes and is capable of quantifying these phenomena at single cell levels. Furthermore, the model is adaptable to various different types of tumor architecture depending on its training set to produce different hypothesis testing environments. The agreement between CCIGAN data and those reported in Keren al. (201)) serve as an important control and demonstrate the fidelity of the CCIGAN output towards true biological results. More importantly, this provides strong evidence to support the accuracy of the predictions made by CCIGAN in the assessment of hypothetical cellular scenarios which cannot be tested via in vitro tissue study alone.
5.5 MODEL INTERPRETABILIT
Examining the model's persistent vectors Sm, we can try to understand there match between real world protein markers and the representations of 8m. For example, the vector 8ps for pan- keratin attends to tumor cells and $CDs attends to CD8 T cells at pixel pairwise levels. It follows that in simple experiment where corresponding SCDs ++ Spk. vectors are exchanged internally in the attention module (Eq. 10, Figure 4 Step 3, outer product) we may observe a lower expression for tumor cells in channel mpp and lower expression for CD8 T cells in channel McDS since tumor cells do not express CD8 and CD8 T cells do not express pan-keratin. As a control, we also switch surface membrane markers HLA Class and dsDNA markers they present in all cells and have very similar average expression values (SHLAel + $&DNA). Accordingly, for our control, we expect to see negligble changes. We define the expression ratio before 1.
after

---

Under review as a conference paper at ICLR 2020
Protein Markers
CDS
pan keratin
HLA Class
dsDNA -0.0012
Expression Ratios
-0.373
-0.145
-0.054
Table 5: 8m persistent vector interpretability experiments.
larger magnitude decrease of the expression ratios in the SCDS < 8pk experiment and a minute difference in the $IIAAc1 ++ AdsDNA- Further visualizations (Figure 13 and discussion (model generativeness, Figure 14) are given A.6
In Table 5
CONCLUSION AND DISCUSSION
We introduced the idea of applying image synthesis to understanding and exploring cell-cell inter- actions in various and different contexts. To do so we use protein attention based GAN, CCIGAN, which can provide accurate characterizations of cellular protein localization phenomena from con- ditioned counterfactual cell-cell scenarios. Additionally, the architecture of the attention module we propose can be generalized other multiplexed datasets that require real world priors.
Furthermore, CCIGAN outperforms variety of current methods in biological modeling. We demonstrate this through biological consistency where CCIGAN recapitulates, discovers, and quan- tifies meaningful cellular interactions through 3 different experiments in tumor environment un- recognized by other models. This highlights the potential for CCIGAN identify cellular protein interactions which account for variation in patient responses to cancer therapy, providing frame- work for biological hypotheses which explain clinical outcomes on a cellular level.
REFERENCES
Mojgan Ahmadzadeh, Laura A. Johnson, Bianca Heemskerk, John R. Wunderlich, Mark E. Dudley, Donald E. White, and Steven Rosenberg. Tumor antigen-specifi ed8 cells infiltrating the tumor express high levels of pd-1 and are functionally impaired. Blood, 1 11::1337-1544, 2009.
S. C. Angelo, M.and Bendall, R. Finck, M. B. Hale, C. Hitzman, A. D. Borowsky, R. M. Levenson, B. Lowe, S. D. Liu, S. Zhao, Y. Natkunam, and G. Nolan. Multiplexed ion beam imaging of human breast tumors. Nature Medicine, 20:436 442, 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. CoRR, abs/1809.11066, 2018. URL http://arxiv.org//00
18C9.11098
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. arXiv e-prints, art. arXiv:1 106.2666 Jun 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv e-prints, art. arXiv:1502.03167, Feb 2015.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016.
Yoshiko Iwai, Masayoshi Ishida, Yoshimasa Tanaka, Taku Okazaki, Tasuku Honjo, and Nagahiro Minato. Involvement of pd-ll on tumor cells in the escape from host immune system and tumor immunotherapy by pd-11 blockade. PNAS, 19:12293-12221 2002.
Xianjie Jiang, Jie Wang, Xiangying Deng, Fang Xiong, Junshang Ge, Bo Xiang, Xu Wu, Jian Ma, Ming Zhou, Xiaoling Li, Yong Li, Guiyuan Li, Wei Xiong, Can Guo, and Zhaoyang Zeng. Role of the tumor microenvironmen in pd-11/pd-1 http//dd--mmdittee -mediated tumor immune escape. Molecular Cancer, 18, 2019.
10

---

Under review as a conference paper at ICLR 2020
Leeat Keren, Mare Bosse, Diana Marquez, Roshan Angoshtari, Samir Jain, Sushama Varma, Soo- Ryum Yang. Allison Kurian, David Van Valen, Robert West, et al. A structured tumor- tumor-immune immune microenvironment in triple negative breast cancer revealed by multiplexed ion beam imaging. Cell, 17446):133--138, 2018.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints, art. arXiv:1312.6114, Dec 2013.
Janet Lau, Jeanne Cheung, Armando Navarro, Steve Lianoglou, Benjamin Haley, Klara Totpal, Laura Sanders, Hartmut Koeppen, Patrick Caplazi, Jacqueline McBride, Henry Chiu, Rebecca Hong, Jane Grogan, Vincent Javinal, Robert Yauch, Bryan Irving, Marcia Belvin, Ira Mellman, Jeong M. Kim, and Maike Schmidt. Tumour and host cell pd-11 is required mediate suppression of anti-tumour immunity in mice. Nature Communications, 8, 2017.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 2794-2802, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. CoRR, abs/1802.05957, 2018. URL http:://axxii...rrr.
abs/1802. 05957
Nao Nishida-Aoki and Taranjit S. Gujral. Emerging approaches to study cell-cell interactions in tumor microenvironment. Oncotarget, 10:785-797, 2019.
RG Oshima. Apoptosis and keratin intermediate filaments. Cell Death and Differentiation, 9:486 492, 2002.
Drew Pardoll. The blockade of immune checkpoints in cancer immunotherapy. Nature Reviews Cancer, 12:252-264, 2012.
Taesung Park, Ming-Yi Yu Liu, Ting-Chun Wang, and Jun- Zhu. Semantic image synthesis with apatially-adaptiv normalization. CoRR, abs/1990,,0721, 2019. URL httlatt:p/xxxi..rrrt abs/1903. 07291
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv e-prints, art. arXiv:1511.06433 Nov 2015.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as metric for image retrieval. International journal of computer vision, 40(2):99-121, 2000.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. Weakly supervised memory networks. CoRR, 500.0955,,, 2015. utLpttt://aaxxt http://axxxvvvorrrbs////aa httpg//bs/15033 08895
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks. CoRR, abs/1409.32 2014. URL
Dmitry Ulyanow. Andrea Vedaldi, and Victor Lempitsky. Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv e-prints, art. arXiv:1607.08022. Jul 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-666869 6008. Curran Associates, Inc., 2017. URL http: /ttt:://paper....m nips cttp/ppppp///B.attttttoooooo
781 attention
all you-oouuudeee pdf
Ting Chhg Wang. Ming- Yu Liu, Jun- Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018a.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. CVPR, 2018b.
11

---

Under review as a conference paper at ICLR 2020
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, al. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Yichen Wu, Yair Rivenson, Hongda Wang, Yilin Luo, Eyal Ben-David, Laurent A Bentolila, Chris- tian Pritz, and Aydogan Ozean. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nature Methods, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv:
2018.
Jan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE Interna- fional Conference on, 2017.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1
ARCHITECTURE DETAILS
The detailed architecture of our generator is shown on Table 6
Layers
Output Size
Generator
Linear
(128, 4, 4) (128, 8,
Linear 128 x 2048
Upsampling
Upsampling 2 x 2
SPADE 128, Leaky ReLU
Convolution 3 x
SPADE ResBlk-1
(128, 8,
SPADE 128, Leaky ReI.U
Convolution 3 x
Upsampling
(128, 16, 16)
Upsampling 2 x 2
SPADE 128, Leaky ReLU
Convolution 3 x
SPADE ResBlk-2
(128, 16, 16)
SPADE 128, Leaky ReLU
Convolution 3 x 3
Upsampling
(128, 32, 32)
Upsampling 2 x 2
SPADE 128, Leaky ReLU
Convolution 3 x 3
SPADE 64, Leaky ReLU
SPADE ResBIk-3
(64, 32, 32)
Convolution 3 x
SPADE 64, Leaky ReLU
Shorteut Convolution 3 3
Upsampling
(64, 64, 64)
Upsampling 2 x 2
SPADE 64, Leaky ReLU
Convolution 3 x 3
SPADE ResBIk-4
(64, 64, 64)
SPADE 64, Leaky ReLU
Convolution 3 x
Convolution
(24, 64, 64)
Leaky ReLU, Convolution 5 x 5
Convolution
(24 * 8, 64, 64)
Leaky ReL.U, Convolution 5 x
Group SPADE
(24 8, 64, 64)
[SPADE 8] * 24
(24 # 64, 64, 64)
[Outer Product 8 & 8] * 24
Modulation
(24, 64, 64)
Convolution 1, Sigmoid
Output
(24, 64, 64)
Sum residual, Sigmoid
Table 6: Architecture details of CCIGAN's generator
where ResBlk is the residual block with skip connection used in ResNet (He et al. 2016). and SPADE is the spatially-adaptive normalization layer. The detailed architecture of our discriminator
is shown on Table
Layers
Output Size
Discriminator
Convolution 4 x 4, stride 2 Instance Norm, Leaky ReLU
Conv-1
(32, 32, 32)
Convolution 4 x 4, stride 2 Instance Norm, Leaky ReLU
Conv-2
(64, 16, 16)
Convolution 4 4, stride 2 Instance Norm, Leaky ReLU Convolution 4 stride Instance Norm, Leaky ReLU Convolution 4 4, stride Instance Norm, Leaky ReLU
Conv-3
(128, 8, 8)
Conv-4
(256, 4,4)
Conv-5
(512, 2, 2)
Convolution 3 x 3, stride 2 Sigmoid
Conv-6
1,)
Table 7: Architecture details of CCIGAN's discriminator
13

---

Under review as a conference paper at ICLR 2020
For all baseline models, use the architecture based the their original implementation. Due to the size of the cell patch is (64, 64), we reduce the size of hidden layers to fit our dataset. For fair comparison, we use the same reduction of hidden layers and the same discriminator architecture for SPADE, pix2pixHI,, and CCIGAN.
A.2 MODEL TRAINING
G' is the generator and D is the discriminator for CCIGAN. Given segmentation map S. ground truth and noise 5, the generated image G(S, 8). The input the discriminator is the cell image conditioned on the segmentation map S. We use LSGAN loss (Mao et al. 2017) in CCIGAN, which is defined as follows:
LGAN D) Ey s D(, 1) Es [|| - D(G(S) 6). S)
(13)
In addition to GAN loss, also use feature matching loss expressed as:
(Wang al. 2018a) during training
1 N3 [I (D LG, Byy E
(14)
where D, is j-th layer feature map of the discriminator for j E {I,..., and Nj is the number of elements in j-th layer. Consequently, the objective function for training is given as follows:
min (maxx Di) ATTMMG,
(15)
where A = 10. Due to the size of cell patch is (64, 64), we do not use multi-scale discriminators and perceptual loss in CCIGAN and other baseline models e.g. SPADE and pix2pixHD.
In training, discriminator learning rate is (rp = 0.0001. We train CCIGAN 120 epochs with training set of 5648 cell patches. We train other baseline models for 120 epochs until they converge (max 150). The full details of training of CCIGAN and baselines are shown as Table g The hyperparameters of each model are fine-tuned get better performance. The training time roughly equal for all models. In particular, CCIGAN was around 1.2 times slower than the SPADE baseline on a single Tesla V V100 GPU.
ADAM as the optimizer. The generator learning rate is bra 0.0004 and the
Metrics Ira
Ours
SPADE
Pix2PixHD
CycleGAN
0.0004
0.0008
0.0002
0.0002
IND
0.0001
0.0001
0.0002
0.0002
Table 8: Hyperparameters of models
A.3 EVALUATION METRICS
A.3.1 RECONSTRUCTION METRICS
Given the generated image set X' (x) and the ground truth set y (Y.),, with X,, Y,
(M,..W)) the L,/MSE score is defined as follows,
N
M
L(, = Σ
sort( UU 2 X.m) - sort(U; o Yim
(16)
i= m=1
where || || can be either L1 or L2 norm, the clement-wise product, X m and Y Ysm m are the m-th channel of the i-th cell patch, U, E {0, (H.W) is the mask matrix which masks all the cells i-th patch. For any matrix A, sort( A) is the sort function that sorts all entries of A. The sorting function ensures our metrics are position independent and only measures the intensity the generated image and ground truth. The score function T(X,), ))) only computes the loss of sorted expression inside of
14

---

Under review as a conference paper at ICLR 2020
the cells. Then add penalization for expression outside of cells. The adjusted L/MMSE score is introduced as follows,
M
())) = Σ [|ssor(U; O Xi,m sort(U; O Yim)
(17)
[ss - U.) 2 x.m sort( U.) Yem)))
where 1a the matrix with all entries equal to I. The (adjusted) L,/MSE scores of CCIGAN and baseline models are shown Table[] A smaller score means a better result.
For any two images X, Y E [0, 1)(,,) the SSIM and MI are defined as:
(2μx + 01 (2xxy + 02) (1* m x oo (2)
I H(Y) H(X,Y Y)
SSIM (X Y)=
(18)
(19)
where H() entropy, μx and ox are the mean and standard deviation X, c1 C2 are constants. Then the SSIM between X, is
SSIM 3)) N EE SSIM(XI,m Y,m))
(20)
In cell based MI, test patches processed computed with the corresponding cell in the ground truth. For the generated image X; of the i-th patch, we assume there are T; cells in the i-th patch. Then for each cell t, the pixels of m-th channel of the 1-th cell in the i-th patch can be expressed as vector a..m Hence, the cell based MI is formulated as:
cell-cell basis where their mutual information is
N
M
((x:))
-
Σ [a vl.))
(21)
EE i=1 m=1
We report (((;)) on Table D The SSIM measures the similarity between the generated image and the ground truth. For SSIM, we use HLA Class 1 and dsDNA due to the their expressions in all considered, the SSIM would be uninformative due to the majority of the
cells. If all channels channels being blank or sparse. The MI measures the information shared between generated image and ground truth cell by cell basis where consider all channels. Consider the example where model generates no expression in marker m but the real data has expression in m, the MI would be and vice versa. Higher SSIM and MI values mean better results. Table demonstrates that CCIGAN outperforms or matches all other baselines on all reconstruction metrics.
A.3.2 WEIGHTED CENTROID
For a generated cell image, its centroid is the mean position all the points in the cell. We use pixel values as weights in computing the weighted centroid, now referred to center of mass (COM). Given cell image X E (H.W,, with indices of the segmented cell V C (1...., H) (1....,Wi W), the COM p (5. y) is defined
L(swev ax,.> EiEyjev Xx,, and Eiaeev 9X... L(ayee X
In the PD-I/PD-LI experiment, we compute the COM of the CD8 T cell (cell of interest) weighted by PD-1 expression, given as CCD8 and the COM of all tumor cells weighted by PD-LI expression, given as p Tummor Since T cells located within the tumor microenvironment often have upregulated expression of PD-L1, tumor cells Tunoor The center of mass score is defined below as the relative distance between p pcDs and p Tumor where N is defined as the number patches:
assume that pCD8 should have the same COM as all of its surrounding
N
COMprojectio projection
Σ
lp
CD8 PRojeDE (p. Tumor ||
(22)
N
The projection function Proj(-) is used project D Tunor onto the CD8 T cell to ensure the expected COM of the tumor cells is inside of the CD8 T cell. As reference we choose random position p RRnndmm the CD8 T cell (PD-1) which replaces pCD8 in Eq. 22 and compute the random COM score show the effectiveness of the result. An example illustration is given in F
15

---

Under review as a conference paper at ICLR 2020
A.3.3 EARTH MOVER'S DISTANCE
For each segmentation map i, we iteratively add its 7; tumor cells around one CD8 T cell. The COM is defined for the t-th tumor cell as Pi,- for E {1...., T.J. We omit the subscript i when it is clear from context. The subsequent instances of the PD-1 COMs in the CD8 T cell by adding the 1-th tumor are given by p. Initially when there are no tumor cells, define Po o as the centroid of the CD8 T cell. Based on the above setting, we define a vector which points from the centroid of the CD8 cell po to the COM of the t-th tumor cell P;
Tumor We define vector u, which points
from the previous COM p to the current COM p. of the CD8 T cell. We define 6g as the angle between v;; U,, If cos O2 > that is say if the cosine similarity is positive, the COM of CD8 T cell p.D moves correctly towards the COM of the added tumor cell p!umorr An illustration of the points and vectors is given in Figure Formally:
Pt
Tumor
Po
CD8
U.
Pt
CDS
CD8 P
COS 82
| |
2h
(23)
After obtaining the directional information, we use Earth Mover's Distance (EMD) Rubner et al, 2000) to evaluate the changes in PD-1 expression of the CD8 T cell. The EMD, which measures the dissimilarity of two distributions, is used in this context to measure the protein localization shifts in PD-1 before and after adding tumor cell. We consider each cell X in polar coordinates (r: O) with respect to its centroid. integrate its expression along the radius coordinates, and evaluate the resulting histogram hist(X) along the angle coordinate. This allows for the definition of distance for moving one histogram to another, i.e. em( [ x demstist generated PD-1 expression of the CD8 T cell X when adding the 1-th tumor cell. The final EMI) score is defined as:
tist(XX for the
1
N
T
EMD
1
|X/- em X x!-1)
(24)
Σ T
t=
where the indicator function 11-) if and only if | |X|- otherwise 11-) 0. This ensures that the biological constraint of PD-1 expression increasing as response to added tumor cells is met. Recall, if cos 0 > 0. pt has moved in the direction of p. TTmmorr immplynng the shift in PD-1 expression is correct, and in turn increases the EMD. By contrast, the EMD score decreases when p, cC88 moves in the opposite direction. Example figures and illustrations showing this process Using the EMD we define a randomized search algorithm for discovering other cell-cell interactions; their results and discussion given A.7
are given in Figure
Based the definition of EMD score, the positive EMD score is defined as:
EMD positive
1(|
|X
(em. XX, x!-1) max//cos0,, 0}
(25)
ΣN
The positive EMD score only evaluates the change in PD-1 expression when the COM of a CD8 cell moves towards the COM of the added tumor cell. The projected EMD score is defined
EMD projected
1(|x II) em ( x(-)) -|uu,,el el cos
(26)
Σ1=1
i=1
The projected EMD score is the EMD score weighted by |lur, i.c. the shift from the previous COM to the current COM of the CD8 T cell.
A.4 EVALUATION VISUALIZZATIONS
Here provide example visualizations and illustrations center of mass nomenclature, mass movement of PD-1 as a function of neighboring PD-LI, and process of computing EM distance.
We can observe in Figure 7 that the mass inside of the T cell in the PD-1 channel shifts as response to surrounding tumor cell expressions of PD-L1. The surrounding tumor PD-LI expressions p, are shown in the third row a cell by cell basis for t E {1. T 4). Note that the 3rd column in PD-LI has sparse expression. Finally the last row shows the PD-1 and PD-LI channels superimposed into one channel.
Tumor
16

---

Under review as a conference paper at ICLR 2020
Segmentation
PD-1
PD-L1
p
Tumorr
Proj(p Tumor)
Figure 6: An example illustration of the center of mass (COM) nomenclature from section 5.1. Note the projection onto the CD8 T cell. This provides more consistent measurement across different patches by projecting p Tumer onto the CD8 T cell.
t=1
t=2
t=3
t=4
Segmentation
1 CD3
CD8
CD8
CDB
PD-1
Tumor
PD-L1
Tumor
P
Tumor
PD-1 + PD-L1
Figure 7: Process of iteratively adding tumor cells. The added red cells are tumor cells (PD-L1) and the center orange cell indicates CD8 T cell (cell of interest, PD-1). For this process, focus each instance of an added tumor.
We give example in Figure illustrate the vectors and after adding t-th tumor cell for
t = 1,2 2 computing EMD score, where V; and u. defined in Eq. 23
A.5 RESULTS GRAPHS
The pan-keratin/CDg experiment is similar to Figure 7 orientation except the center cell (cell of interest) is tumor cell (red) and the adjacent neighboring cells are CD8 cells (orange).
17

---

Under review as a conference paper at ICLR 2020
t=0
t=1
t=2
P
Figure 8: An example illustration of the points and vector nomenclature from section 5.2. The blue dots are the expression of PD-1 and PD-LI proteins. The eyan arrows show the vectors Vr and u. Note the shift in expression of the PD-1 as response to the added tumor's PD-L1 expression.
CCIGAN
0.022
Pan Kerain Expresion
Eipresion
0.255
0.008
T Cell
80000
0.006
8.05
8.005
8:000
0.000
0.87
0.0755
0.56
Earcssinn
0.150
Control
0.04
0.0000
0.83
0.075
8.00
00025
0.00
00000
Figure 9: CCIGAN experiment for adding CD8 T cells and tumor cells (control) around tumor cell.
CCIGAN predicted a decrease in tumor cell pan-keratin expression with respect to increasing CD8 T cell area/number (Figure This is juxtaposed to the tumor cell control where there is no change the pan-keratin level as the number of neighboring tumor cells is increased.
SPADE does not predict decrease in tumor cell pan-keratin expression with respect to increasing CD8 cell area/number and shows no difference in pan-keratin expression trends between the cell and control groups (Figure 10).
18

---

Under review as a conference paper at ICLR 2020
Spade
Cell
0.02
00075
0.01
00850
00825
0.00
00000
Arcu Colla
Namber
0.025
Pan-trrdh Exrrosiin
804-
8.055
Control
683
0.000
0:2
0:03
0:00
500
2590
3000
0.000
Anea Calic
Mamber Call
Figure 10: SPADE experiment for adding CD8 T cells and tumor cells (control) around a tumor cell.
pix2pixHD
0.035
Pas.tratt Expersion
8.000
0.022
T Cell
0.000
Pan-eeattn
0.008
0.15
0.066
0.04
8.005
0002
500
3000
1500
2000
2500
3008
Area of Cetls
Kenetn Expresion
0.3-
Control
0.02
0.02
500
3000
1500
2000
2500
3000
Figure 11: pix2pixHD experiment for adding CD8 cells and tumor cells (control) around a tumor cell.
pix2piXHD erroneously predicts an increase tumor cell pan-keratin expression with respect to increasing CD8 T cell area/number and shows no difference in pan-keratin expression trends between the cell and control groups (Figure IT].
CycleGAN fails to predict decrease in tumor cell pan-keratin expression with respect to increasing CD8 T cell area/number and shows no difference in pan- pan-keratin expression trends between the T cell and control groups (Figure 12).
19

---

Under review as a conference paper at ICLR 2020
CycleGan
0.06
Pan.terth Depesson
0.05
PasaGordin Expression
0.012
0.10
Cell
0.08
0.2
0004
003
0.000
500
1000
1900
2500
3000
0.08
0.020
Expessin
0.05
0.015
Control
Pan-Ceratnn
0.00
500
3000
1500
2000
2500
3000
Area of Catss
Figure 12: CycleGAN experiment for adding CD8 T cells and tumor cells (control) around a tumor cell.
A.6 MODEL. INTERPRETABILITY AND GENERATIVENESS
Figure 13 shows the persistent vectors 8; for all proteins. Note the similarity between CD3 and CD8 T cell protein markers and the similarity between dsDNA and HLA Class surface membrane proteins (expressed in all cells). It is also important to make the distinction that sparse markers (while different) are similar state. This is due the lack of training data for rare cell types, making it difficult for the model to reason on such small sample size.
Pan-Koratin
dsDNA
Lag3
Figure 13: Persistent vectors for various channels.
Figure 14 shows the generativeness of CCIGAN through an uncertainty map over 100 instances (random noise). An uncertainty map shows the differences per pixel (x, y) location. The higher intensities indicate a higher probability of changing at the specified (ar, y) location.
20

---

Under review as a conference paper at ICLR 2020
dsDNA
CD3
CD8
HLA Class
Segmentation
Uncertainty Map
Fake
(Instance)
Real Data
Figure 14: Uncertainty maps illustrating model generativeness.
A.7
SEARCH ALGORITHM
Here we provide a randomized search algorithm to try to discover further cell-cell interactions in
other channels.
Index
Fixed Cell Type
Added Cell Type
Channels
Expression
CD8 T Cell
Acratin-positivy tumor
PD-1, CD3, Vimentin
Increase
2
CD8 T Cell
CD8 T Cell Ceratin-positiv tumor
PD-1
Decrease
3
CD3 T Cell
CD3, CD4
Increase
4
Keratin-positive tumor
CD8 T Cell
Pan-Keratin
Decrease
5
veratin-positive tumor
CD3 T Cell
Pan-Keratin Pan-Keratin
Decrease
6
Ceratin-positiig tumor
Macrophages
Decrease
7
Macrophages
Keratin postiivve tumor
Vimentin
Increase 1
Table 9: Additional cell interaction trends captured by CCIGAN.
The experimental findings Indices 4, 5 and 6 support those reported in 6,2.2, demonstrating that immune cell presence adjacent tumor cells causes a decrease in the tumor's 's pan-keratin regardless of immune cell identity. In addition to confirming the results of 6.2.1, findings in indices and also indicate that tumor cells increase the expression of the T cell co-receptor, although this is of unclear functional significance. Index 2 suggests cell clustering reduces the expression of the immune suppressive PD-1 marker. Lastly, Index demonstrates expressed vimentin when macrophages are placed adjacent to tumor cells. Since vimentin is secreted as a pro-inflammatory marker macrophages, this suggests an early macrophage inflammatory response to its tumor neighbor.
increase in macrophage
A.8 MARKERS
The markers we used (total 24) in our experiments are: Pan-Keratin, EGFR, Beta catenin, dsDNA, Ki67, CD3, CD8, CD4, FoxP3, MPO, HLA-DR, HLA-Class-1, CD209, CD1lb, CD1le, CD68, CD63, Lag3, PDI, PD-LI, IDO, Vimentin, SMA, CD31.
The markers didn't use (total 12) in experiments CD16, B7H3, CD45, CD45RO, Ker- atin CD20, CD163, CD56, Keratin6, CSF-1R, p53, CD138
21

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Search Algorithm
Input:
Cell segmentation list {S:) the channel index m, rotation angle A0, fixed noise 8, threshold / and the generator
Randomly chose initial cell index in E n1,
Input segmentation SINPUT = Sip:
Mask for the initial cell U = - ;
Generated image X, G(d,, SINPUT);
Specified channel Mo U * Xo|m]:
Show SINPUT and Mo:
for & = to n 1 do
Random index is (I, n// us/ (b):
SINPUT = SINPUT + Si.
X, = G(,, SINUUT:
Mj Xx(m];
Show SINPUT and Mxi Es dEM(MR- Ma);
Ex > 8 Ei, Uni then Log Significance:
if Σx, Ma-1As < Ens Ma.dy then Log Increase;
else
Log Decrease;
22


Reviews:
[{"review_id": "ryxXuMHAtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies the SPADE semantic image synthesis technique (with a custom attention mechanism) to MIBI-TOF data to examine hypotheses about cell-to-cell interactions in the context of an immune infiltrated tumor sample. I think that this is potentially an interesting application of GANs -- to generalize beyond specific gathered data and instead start engaging with counterfactual scenarios like \"what effect does it have to add cell type X next to cell type Y\". \n\nUnfortunately, I think this paper is not yet finished with regard to both (1) conveying sufficient motivation for the use of image synthesis for distilling biological insights and (2) evaluating the success or failure of the technique primarily in terms of generating useful biological insights. Simultaneously, there are enough red flags with regard to knowledge of the data generation and underlying biology that it's unclear if the authors could correctly sanity check any insights they extract from their model. \n\nMore specific criticisms & suggestions\n\n1) It would help to provide much clearer motivation for applying image synthesis to MIBI-TOF data. Section 1.2 skips straight from describing the MIBI-TOF instrument to “we made a new kind of GAN”.  Again in the beginning of the Related Work the paper states: \"We are interested in the task of generating biologically consistent expression patterns of cellular proteins given a segmentation map of cell neighborhoods”. But, why are synthetic images interesting given that we can actually look at real MIBI-TOF data. I start getting a sense of why this model might be interesting or useful only in Section 5 — more rationale is required earlier in the paper. \n\n2) The evaluation section is hard to follow. I think it would be helpful to more clearly describe a larger set of biological phenomena that a practitioner would expect to see in the data, choose a single metric for each case, and show that that these phenomena are recapitulated. No one is going to trust a GAN to give them scientific insights unless they're very confident that all known / simple cell-to-cell interactions have a clear signal. \n\n3) \"MIBI-TOF bombards a tissue sample with elemental metals tethered to respective antibodies for dozens of distinct cellular proteins and detects each to obtain image channels” — this is not an accurate description of MIBI-TOF, at least not the instrument I'm familiar with. Typically the tissue is first stained with antibodies tagged with heavy metals and the instrument then bombards the tissue sample with simple ions (like O2+), causing the release of metal ions. It seems very unlikely that bombarding anything with antibody/metal conjugates could be informative. \n\n4) \"Tumor cells could be identified by markers such as pan-keratin and beta-catenin” — I guess in the context of a tumor sample beta-catenin could be over-expressed but it's present in pretty much every cell type, including lymphocytes (https://www.proteinatlas.org/ENSG00000168036-CTNNB1)\n\nSmall nits:\n\n* Repeated use of “antibodies” in \"Engineered antibodies for PD-1/PD-L1 antibodies” — maybe better to write “Antibodies which block the interaction of PD-1/PD-L1\"\n\n* \"While MIBI-TOF is capable of 36 different markers, we discarded uninformative and irrelevant markers1 resulting in M = 24.” — I’m really surprised that someone put 12/36 uninformative markers in a MIBI-TOF panel. Aren’t these tagged antibodies expensive? Can we at least get a list of what got discarded?\n\n* The model interpretability control seems weak in that lymphocytes are more likely to express MHC-I than tumor cells (which have a potential survival advantage from not expressing it) and tumor cells may actually have more dsDNA than lymphocytes due to changes in ploidy (or original differences in ploidy from e.g. liver cells). ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SyO0QgHqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a GAN for multiplexed imaging (MIBI-TOF) data called CCIGAN. They propose an interesting architecture design with protein-specific attention to find association between cell types and neighboring pattens and cell-cell interactions. They also propose new and biologically interpretable metrics including a reconstruction metric, projected EMD and regressing of expression on neighbors.\n\nThey present improved reconstruction of interactions compared to other models in the context of PD-1 and PD-L1 interactions. It would be great to extend the evaluation to other interactions and tissue types.\n\nOverall the paper is well written, the application and especially the focus on cell-cell interactions is novel. The model is properly justified and evaluated, and there is a high demand for this framework in the multiplexed imaging field.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HygLks1AFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript proposed a new method to model the data generated by multiplexed ion beam imaging by time-of-flight (MIBI-TOF). Essentially, the model leans the many-to-many mapping between the cell types and different protein markers' expression levels. Compared with the other mainstream GAN methods, the authors show the proposed method, CCIGAN, can outperform them in terms of generating the expression map of different protein markers given the segmentation of cell types. The manuscript also has an in-depth discussion of the biological meaning of the learned model as well as the learned vectors. \n\nI personally like this manuscript a lot, considering the novelty and the thoroughness of the manuscript. However, I have the following concerns: \n\nMajor concern (The score will be significantly improved if the authors can handle these two concerns during revision): \n1. My largest concern is how useful the proposed method is. It seems the wet experiments are mature. How can the computational synthesized image help the biologists then? Are there any cases that only the CCIGAN can do while the real experiment can not do? I am not an expert in MIBI-TOF, but I guess all the results in Section 6.2 can be easily achieved with the real data, right? \n2. CCIGAN is indeed better than the other methods, but would it be accepted by the biologists in terms of performance? Would they believe in the results? In fact, in the last column of Figure 2, we can see clear artifacts for the results generated by CCIGAN.\n\n\nMinor concern:\n1. What's the resolution of the MIBI-TOF and CCIGAN?\n2. The introduction of the background could be further refined. The many-to-many mapping between different cell types and different protein markers is not emphasized explicitly. Readers can get lost easily. \n3. The data part is not clear. How many M*2048*2048 images do the authors have for training as well for testing? \n4. Why do the authors choose 64*64 as the image path size?\n5. Can the model be generalized well for data collected in different experiments (i.e., different tissues) but from the same machine? Can the model be generalized well across different machines with the same imaging experimental setting?\n6. Is the model sensitive to the preprocessing of the data, like normalization? As for as I know, the baseline expression level of tissue can vary significantly at different time points within one day. If the model is sensitive to that, it will affect the usage of the model.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a dedicated deep models for analysis of multiplexed ion beam imaging by time-of-flight (MIBI-TOF).

The reviewers appreciated the contributions of the paper but not quite enough to make the cut.

Rejection is recommended. 
Decision: Reject