Title: EXPLORING CELLULAR PROTEIN LOCALIZATION THROUGH SEMANTIC IMAGE SYNTHESIS

Abstract: Cell-cell interactions have an integral role in tumorigenesis as they are critical in governing immune responses. As such, investigating specific cell-cell interactions has the potential to not only expand upon the understanding of tumorigenesis, but also guide clinical management of patient responses to cancer immunotherapies. A recent imaging technique for exploring cell-cell interactions, multiplexed ion beam imaging by time-of-flight (MIBI-TOF), allows for cells to be quantified in 36 different protein markers at sub-cellular resolutions in situ as high resolution multiplexed images. To explore the MIBI images, we propose a GAN for multiplexed data with protein specific attention. By conditioning image generation on cell types, sizes, and neighborhoods through semantic segmentation maps, we are able to observe how these factors affect cell-cell interactions simultaneously in different protein channels. Furthermore, we design a set of metrics and offer the first insights towards cell spatial orientations, cell protein expressions, and cell neighborhoods. Our model, cell-cell interaction GAN (CCIGAN), outperforms or matches existing image synthesis methods on all conventional measures and significantly outperforms on biologically motivated metrics. To our knowledge, we are the first to systematically model multiple cellular protein behaviors and interactions under simulated conditions through image synthesis.

1. INTRODUCTION:

1.1 BIOLOGICAL ROLE OF CELLULAR PROTEINS Cell-cell interactions within the tumor microenvironment have been implicated in many facets of cancer pathogenesis and treatment. Most prominently, tumor cell evasion of immune surveillance (Jiang et al., 2019) , tumor metastasis (Nishida-Aoki & Gujral, 2019) , and efficacy of cancer immunotherapies (Lau et al., 2017) have all been closely linked to the relationships between immune and cancer cells. These types of cell-cell relationships are generally governed by the interactions of cell surface proteins which drive cell behavior, gene expression, and survival. One of the most prominent examples of cellular proteins influencing disease progression is the case of PD-1/PD-L1. PD-L1 is a protein often overexpressed on tumor cells and has the capacity to bind PD-1 on local T cells to downregulate their anti-tumor immune responses (Iwai et al., 2002) .

Antibodies which interrupt the interaction of PD-L1 and PD-1 and allow the immune system to attack tumor cells, have become clinically influential treatments for a variety of cancers (Pardoll, 2012) . This example highlights the value of accurately predicting cellular protein patterns which play key roles in disease processes. Exploring protein localizations in a multi-cellular system represents a challenge for which deep learning models are uniquely suited. However, to our knowledge no image-based deep generative models have utilized semantic image synthesis to produce accurate predictions of these biological phenomena.

1.2. FRAMEWORK AND NOVELTY OF A PREDICTIVE MODEL:

A meaningful exploration of cell-cell interactions, particularly in the tumor microenvironment, requires a thorough understanding of the proteins expressed on and within a cell and its neighborhood. Multiplexed ion beam imaging by time-of-flight (MIBI-TOF) represents a novel technology that can accurately quantify and spatially resolve cellular protein expressions at the single cell level within tissue samples. Given a tissue sample that is first stained with protein-specific antibodies tethered to elemental metals, MIBI-TOF bombards the sample with atomic ions (i.e. O + 2 ) from a primary ion beam. This causes the release of elemental isotopes and tissue-specific material which can be quantified in a mass spectrometer (Angelo et al., 2014) . The cellular proteins characterized by this technique indicate specific cell types (i.e. immune cells, tumor cells), cell status (i.e. markers of proliferation), and immunomodulation. Figure 1 (A) shows an example spatial orientation of cell types and some selected cellular protein expressions. Here, we propose a novel protein based attention mechanism for a convolutional Generative Adversarial Network (GAN) with the capacity to provide accurate conditioned predictions of cellular protein localizations. The model, Cell-Cell Interaction GAN (CCIGAN), learns a many to many mapping between different cell types and different protein markers. CCIGAN is trained on semantic segmentation maps of cell tissue samples to identify and associate cell type, shape, and the identity of its cellular neighbors with cellular protein expressions for each cell. The network was then applied to simulated segmentation maps to understand how cells vary their protein localizations.

CCIGAN demonstrates its ability to probe elements of oncologic processes by: 1. Independently recapitulating established patterns of biological phenomena, thereby demonstrating the accuracy of its predictive power.

2. Yielding quantitative and spatial information on specific cellular protein modulations as a result of immune cell -cancer cell interactions.

3. Allowing for biological data to be generalized beyond traditional in vitro scenarios where specific, rare, or hypothetical cell-cell events can be posed in counterfactual scenarios and the results of their interactions predicted.

The ability for CCIGAN to generate subcellular protein predictions represents a step forward in the ability to understand cellular relationships within a microenvironment. Rather than assessing in vitro incidence of cell interaction phenomena as is done by MIBI-TOF, CCIGAN allows for hypothetical biological situations to be generated. Figure 1 (B ) illustrates examples how a segmentation map can be directly manipulated to pose various biological scenarios. In other words, individual cellular responses to challenge or proximity of cells of any identity can be assessed without having to seek this specific occurrence within the available biological tissue sample. These capabilities allow the model to provide insight on hypothetical interaction events which shed insight on disease pathogenesis and which cannot be feasibly achieved by biological investigation alone.

MIBI-TOF is a labor intensive process which can make the data collection process cumbersome.

Deep learning models such as CCIGAN have the potential to be useful tools which can extend on MIBI-TOF capabilities and result in more rapidly synthesized data. CCIGAN, once trained on MIBI-TOF data sets and properly replicating known cell-cell interaction patterns, can be used as a robust means to provide high throughput readouts on myriad single cell scenarios which may normally take many iterations of MIBI-TOF experiments to investigate. Recent work by (Wu et al., 2019) demonstrated the utility of conditional GANs for adding a depth dimension to images; we demonstrate learning contextual cell-cell interactions, facilitating rapid hypothesis testing to assess biological environments. This provides valuable insight into a complex system that normally would have taken resource-intensive wet lab experiments to interrogate.

2. RELATED WORK:

We are interested in the task of generating biologically consistent expression patterns of cellular proteins given a segmentation map of cell neighborhoods. Specifically, we want to learn a generative model that simultaneously produces high quality maps of protein expression for individual cells that are probabilistically consistent when conditioned on the same factors, e.g. similar cell neighborhoods should produce similar expression patterns. Such generative models typically take the framework of a generative adversarial network (GAN) (Goodfellow et al., 2014; Brock et al., 2018) or a variational autoencoder (VAE) (Kingma & Welling, 2013) .

Within these generative modeling techniques, image translation focuses on learning a many to many mapping to transform data from one domain to another. One approach is to model the translation task, known as image synthesis, with conditioning such as Pix2Pix (Isola et al., 2016) , Pix2PixHD (Wang et al., 2018a) and CycleGAN (Zhu et al., 2017) . By using image synthesis, a model is able to distill more information from a segmentation map to various protein channels. As MIBI-TOF measures information such as protein localizations at a subcellular resolution, framing the problem through an image synthesis framework allows a model to see how predictive neighborhoods and cell types are of a cell's phenotype.

Recently, Park et al. (2019) proposed spatial adaptive normalization (SPADE) to synthesize images using segmentation maps by learning fully convolutional normalization parameters based on their segmentation conditioning. For each layer, semantic information is retained by allowing the network to learn from the segmentation map directly and modulate the current layer. They demonstrate impressive quality and diversity in generated images, especially modeling objects in context. Using SPADE normalization, CCIGAN is able to condition on surrounding cell neighbors and capture cellcell interactions in a local receptive field. Despite image generation, there has been no work done towards understanding such generations in context, particularly in biological images.

Attention layers (Vaswani et al., 2017; Wang et al., 2018b; Zhang et al., 2018) have also been added to generative models with great success. Self-attention was initially proposed in machine translation tasks to help model long distance dependencies that occur frequently in language (Sutskever et al., 2014) , which led to the idea of external memories as a persistent state to model long range dependencies (Sukhbaatar et al., 2015) . While convolutional neural networks are apt at exploiting local structures of patches, they may struggle to model global structures. For this reason visual attention allows the network to enhance activations in interesting parts of an image. Goodfellow et al. (2014) and Brock et al. (2018) both achieve state-of-the-art unconditional generative modeling using self attention GAN (Zhang et al., 2018) . We propose a specialized attention module conditioned on different proteins to mimic real world protein markers.

3. DATASET:



3.1. MULTIPLEXED ION BEAM IMAGES (MIBI):

All experiments are performed on data obtained through MIBI-TOF characterized tissue samples collected from triple-negative breast cancer (TNBC) patients. By simultaneously imaging over 36 protein markers, MIBI-TOF is able to identify cell type as well as provide detailed information of sub-cellular structure, cell neighbors, and interactions in the tumor microenvironment across these different marker settings. Each of these markers m ∈ {1, ..., M }, given as a channel taking on real values continuous in [0, 1] at each (x, y) coordinate, demarcates a different cellular subtype and furthermore, is indicative of the functional properties of a cell. While MIBI-TOF is capable of 36 different markers, we chose markers exhibiting cellular protein localization and disregarded markers that were blank or used simply for cell-typing1 resulting in M = 24. A full list of these markers is given in A.8. The MIBI-TOF data collected from the TNBC tissue allows for the differentiation between a wide variety of cell types within the samples. For instance, cells positive for the marker CD3 could be identified as T cells, and then subdivided into cytotoxic or helper lineages by the presence of markers CD8 or CD4, respectively. Tumor cells could be identified by markers such as pan-keratin and overexpressed beta-catenin. Along these lines, a wide variety of cellular proteins identified by MIBI-TOF could characterize cell interactions as well as immunomodulatory processes occurring within the microenvironment.

MIBI-TOF data is fundamentally different than typical RGB images. This poses unique challenges in image modeling and characterization. Such challenges stem from each marker being conditionally expressed on the cell type in its respective channel m. Using a simplified 3 channel multiplex setting, a T cell expresses signals in the CD3, CD8 channels (indicators for immune cells) but not in a pankeratin channel (indicator for tumor cells). Another problem is the sparsity of the data, meaning either some expressions for rare cell types are rarely observed or have weak signals. For example, Figure 2 includes a segmentation of a CD8 T cell (orange, top right corner), where other models fail in generating correct CD3 expression, if at all2 . Lastly, the noisy nature of the data leads to inaccurate cell type classifications, creating inconsistent pairs of labels and outputs during training.

These issues make it especially difficult for an RGB multihead decoder to output multiple channels in a biologically accurate manner. Without addressing multiplexed data, a decoder would equally attend to every location of the current latent representation, even if it is irrelevant to the current protein. Furthermore each protein channel in the output has its own sensitivities to signal intensities and noise, suggesting each channel requires a unique prior and that equal attention would be problematic. It follows that special care must be given towards modeling specific channels and the multiplexed nature of MIBI-TOF images.

3.2. DATA PROCESSING:

MIBI-TOF images are represented as a high dimensional tensor T ∈ R (M,2048,2048) . These images are then further processed at a cell by cell basis into Y ∈ R (M,64,64) patches, where a cell is at the center of the patch along with its neighbors. Next, we construct semantic segmentation maps S ∈ R (C+1,64,64) , where a vector S :,i,j is one-hot encoded based on a cell type C = 17, and the C + 1-th channel denotes empty segmentation space. The data is train-test split at a 9:1 ratio at the MIBI-TOF image level to avoid cell neighborhood bias. We also use a synthetic test set where cells and their neighbors are sequentially modified to observe how varying cell type, position, and size affects the progressive changes in protein localizations3 . Z ∈ R (128,4,4) 

4. METHODS:



4.1. MODEL ARCHITECTURE:

EQUATION

4.2. ATTENTION MODULE:

Our architectural contribution is a protein marker dependent attention module in the final output layer. The goal of the attention module is to condition the final output of a channel on a protein marker m and S's cell types. For example the protein marker, pan-keratin m pk , is expressed exclusively in tumor cells but not in other cells. Appropriately, an attention mechanism should attend to tumor cells and ignore irrelevant cells in S for m pk . To replicate a marker searching for specific cell types that express it, we define a learned persistent vector for each marker denoted by s m∈M ∈ R 8 that undergo a series of operations (Figure 4 ) with the final feature map representation attending to m's specific cell types. It is also worthwhile to note that these persistent vectors s m offer a degree of model interpretability that mimic real world markers. The current input dimensions to the attention module are R (128,64,64) following the last resblock R 4 and m indexes from 1, .., M . 

EQUATION

Shown in Figure 4 , after R 4 , a bottleneck convolution is applied to match the original data's dimension as O (step 1), which is used in a residual manner with the final output. Intuitively at this stage, O's feature maps resemble the target Y, but we wish to further refine the output channels.

We convolve O into M K channeled features for each protein marker where K = 8. Considering each C i where i ∈ {1, ..., M } as a group of K channels, the model spatially adaptive normalizes each C i and computes an outer product with the corresponding persistent vector s i and C i . The resulting matrix is flattened and convolved (with a kernel size of 1 on the pixel level) from A i ∈ R (|s|×K,64,64) → R (1,64,64) followed by a sigmoid σ(•) activation. Lastly, the attentions B 1,...,M are added to O to obtain the output X.

Initially, the model has no priors over the interaction of protein markers and cell types. The proposed outer product attention layer (outer product and 1 × 1 convolution) excels at modeling these relationships and interactions between specific markers and cell types. By using an outer product, the model forces attention at a pairwise pixel level comparison for all combinations of elements between s m and A i . As training progresses, both the learned features over segmentation patches and the learned persistent vectors s m improve, in turn allowing the learned 1 × 1 convolution to reason about positive or negative relationships from the pairwise pixel combinations.

4.3. IMPLEMENTATION DETAILS AND TRAINING REGIMEN:

Our implementation (A.1) of the generator applies Spectral Norm to all layers (Miyato et al., 2018) . The discriminator's input is the output of the generator concatenated with the segmentation patch [X, S] and [Y, S] for the ground truth. Finally CCIGAN uses ADAM (lr G = 0.0004, lr D = 0.0001) with GAN loss and feature matching loss. Full training details and loss functions are given in A.2.

5. EVALUATION:

To conduct fair experiments, all models were optimized, tuned, and set with similar parameters. They were also taken from their official online implementations and trained for 120 epochs or until convergence (max 150). CCIGAN is identical to our designed SPADE comparison baseline with the exception of the attention module. Three experiments were conducted to validate the trained model's utility in generating biologically meaningful cellular proteins in the tumor microenvironment and ability to recapitulate and quantify previously established biological phenomena. Each subsection describes the experiment and the relevant metrics used in evaluation. Full mathematical definitions are given in section A.3.

5.1. IMAGE EVALUATION AND RECONSTRUCTION:

First, we use the following evaluation metrics in order to compare with baseline results: adjusted L 1 and MSE score, L 1 and MSE score, structural similarity (SSIM) index (Wang et al., 2004) In the first step of biologically verifying the accuracy of CCIGAN, PD-1/PD-L1 expression relationships between CD8 T cells and tumor cells were assessed. Many T cells located within the tumor microenvironment have upregulated expression of PD-1 suggesting that the tumor milieu exerts influence on the protein localization and expression of infiltrating T cells (Ahmadzadeh et al., 2009) . We assessed if increased expression of PD-L1 on neighboring tumor cells would result in increased directional PD-1 expression in a CD8 T cell (Figure 5 ). We also determined if there was a shift in the cell surface localization of the PD-1. To do so, we computed the Earth Mover's Distance between a CD8 T cell's PD-1 expression (represented as a histogram on a cell's polar coordinates) and itself before and after in different tumor scenarios. In addition to the mass, we computed the expected center of mass (COM) of PD-1 in a CD8 T cell with respect to the neighboring tumor PD-L1 expressions. Equations and explanations are given in A.3.2 and A.3.3. Additional figures illustrating this process are given in A.4.

We expected that the properties of the PD-L1 expressing tumor cell would result in an increased directional PD-1 expression in a neighboring T cell. As a control, a PD-1 expressing T Cell was surrounded with endothelial cells (a normal tissue lining cell). As shown in Table 2 , our initial hypothesis was confirmed in CCIGAN's predictions, wherein the presence of a PD-L1 expressing tumor cell, PD-1 expression in the CD8 T cell increased and moved towards the PD-L1 COM. On the other hand, the endothelial cell presence yielded no effect on the T cell's PD-1 expression, a result which is biologically expected. While both tumor and endothelial cells express PD-L1, the tumor microenvironment exercises more complex suppressive effects on CD8 T cells, which are more likely to modulate the T cell expression than endothelial cells. These findings verify CCIGAN captures the biological relationship of tumor cells inducing immunomodulatory changes on a neighboring T cell. None of the other models succeeded in capturing this protein relationship. This serves to highlight the capacity of CCIGAN to recapitulate established cell interaction phenomena within the tumor microenvironment.

5.3. PAN-KERATIN AND CD8 EXPRESSION:

Secondly, we assessed the effect of immune cell presence on tumor cell status markers. Keratins are a class of intracellular proteins that play an important role in ensuring cell structure. Furthermore, when CD8 T cells mediate tumor killing, they release enzymes which cleave the tumor cell's pan-keratin, disrupting the tumor cell structure (Oshima, 2002) . We explored how the presence of neighboring CD8 T cells to a tumor cell would affect pan-keratin levels within the tumor cell, hypothesizing that T cell mediated tumor killing would result in a drop in pan-keratin tumor expression.

In this experiment, we used a Student's t-test as the statistical hypothesis test to evaluate the correlations between the pan-keratin expression in tumor cells and the area/number of CD8 T cells in contact with the tumor cell surface. Given a generated pan-keratin channel X i ∈ R (H,W ) and the segmentation map channel for CD8 T cells S i ∈ R (H,W ) , we compute the total area of the cells a i = H h=1 W w=1 S i , and the total expression level of pan-keratin e i = H h=1 W w=1 X i . We then regress {e i } N i=1 on {a i } N i=1 and assess significance of the slope using a t-test against the null of no change in pan-keratin expression as a function of CD8 T cell-tumor contact.

CD8 T cells were placed adjacent to a tumor cell in increasing number, similar to Figure 1 (B)'s first row and Figure 5 's experiment. It was observed that the presence of neighboring CD8 T cells to a tumor cell tended to decrease the pan-keratin expression in the tumor cell. This effect became more dramatic as the number of surrounding CD8 T cells was increased (A.5). As a control, when the tumor cell of interest was surrounded with adjacent tumor cell(s) instead of CD8 T cells, no change was noted in that tumor cell's pan keratin for CCIGAN. Table 2 's fourth row shows the result of the t-test where the slope of the linear regression of pan-keratin expression with respect to the number of surrounding cells is compared to a y = 0 flat baseline. A statistically significant difference between the slope of the linear regression for the T cell scenario vs. the baseline was found in the main experiment, but was not present in the control scenario. These results indicate that the T cell presence mitigates a decrease in pan-keratin expression in the tumor cell, which is suggestive of T cell mediated tumor killing. Other models reported contrasting results where a significant change in the tumor cell pan-keratin expression was incorrectly reported under the control conditions. The resulting graphs are given in A.5. Further protein interaction patterns identified by CCIGAN are also reported in A.7. The findings from this experiment serve to highlight the robustness and accuracy of CCIGAN compared to existing image synthesis techniques in probing cell-cell interactions.

5.4. TUMOR INFILTRATED AND COMPARTMENTALIZED MICROENVIRONMENTS:

As a final biological evaluation, we consider the variability of PD-L1 expression in immune and tumor cells across TNBC patient groups. Keren et al. (2018) determined that in situations of mixed tumor-immune architecture, where immune cells freely infiltrated the tumor, the tumor cells predominantly expressed PD-L1. Conversely, in situations of compartmentalized tumors, where there is a greater degree of delineation between immune and tumor cells, macrophages were the predominant source of expressed PD-L1, particularly at the tumor boundary. We used CCIGAN to predict on 200 directly manipulated mixed and non-mixed tumor environment segmentation patches. Similar to Section 5.3's experimental settings, we then compute the average expression of a specific marker for the cells of interest for all patches. For each experiment, we use endothelial cells as control cells to show our result has biological significance. These findings were recapitulated by CCIGAN in Table 3 . For a patient with a mixed tumor environment, when trained with mixed patient samples, CCIGAN reported increased PD-L1 expression on tumor cells. Furthermore, CCIGAN was able to quantify this difference in expression at the single cell level, reporting a tumor to macrophage PD-L1 expression ratio (bolded) of approximately 3.2 and 1.75 for patients A and B respectively. Conversely, when trained with compartmentalized patient samples, CCIGAN reported increased PD-L1 expression on macrophages adjacent to tumor cells as compared to macrophages adjacent to normal endothelial (inert) cells for patients C and D. This difference was quantified as a ratio of PD-L1 expression of tumor-adjacent macrophages to endothelial-adjacent macrophages, approximately 1.85 and 2.7 for patient C and patient D respectively. Moreover, using the trained compartmentalized model to predict on mixed segmentation patches, CCIGAN still reports a 26% (patient C) and 19% (patient D) increase of macrophage PD-L1 expression when compared to mixed microenvironments (Table 4 ). The PD-L1 ratios for the above two scenarios indicate that CCIGAN has appropriately captured previously reported biological outcomes and is capable of quantifying these phenomena at single cell levels. Furthermore, the model is adaptable to various different types of tumor architecture depending on its training set to produce different hypothesis testing environments. The agreement between CCIGAN data and those reported in Keren et al. (2018) serve as an important control and demonstrate the fidelity of the CCIGAN output towards true biological results. More importantly, this provides strong evidence to support the accuracy of the predictions made by CCIGAN in the assessment of hypothetical cellular scenarios which cannot be tested via in vitro tissue study alone.

Experiment:



Experiment (Macrophages) Microenvironments:



5.5. MODEL INTERPRETABILITY:

Examining the model's persistent vectors s m , we can try to understand if there is a match between real world protein markers and the representations of s m . For example, the vector s pk for pankeratin attends to tumor cells and s CD8 attends to CD8 T cells at pixel pairwise levels. It follows that in a simple experiment where corresponding s CD8 ↔ s pk vectors are exchanged internally in the attention module (Eq. 10, Figure 4 Step 3, outer product) we may observe a lower expression for tumor cells in channel m pk and a lower expression for CD8 T cells in channel m CD8 since tumor cells do not express CD8 and CD8 T cells do not express pan-keratin. As a control, we also switch surface membrane markers HLA Class 1 and dsDNA markers as they are present in all cells and have very similar average expression values (s HLAc1 ↔ s dsDNA ). Accordingly, for our control, we expect to see negligble changes. We define the expression ratio as after before -1.

Protein Markers CD8 pan keratin HLA Class 1 dsDNA Expression Ratios -0.373 -0.145 -0.054 -0.0012 Table 5 : s m persistent vector interpretability experiments.

In Table 5 , we can see a larger magnitude decrease of the expression ratios in the s CD8 ↔ s pk experiment and a minute difference in the s HLAc1 ↔ s dsDNA . Further visualizations (Figure 13 ) and discussion (model generativeness, Figure 14 ) are given in A.6.

6. CONCLUSION AND DISCUSSION:

We introduced the idea of applying image synthesis to understanding and exploring cell-cell interactions in various and different contexts. To do so we use a protein attention based GAN, CCIGAN, which can provide accurate characterizations of cellular protein localization phenomena from conditioned counterfactual cell-cell scenarios. Additionally, the architecture of the attention module we propose can be generalized to other multiplexed datasets that require real world priors.

Furthermore, CCIGAN outperforms a variety of current methods in biological modeling. We demonstrate this through biological consistency where CCIGAN recapitulates, discovers, and quantifies meaningful cellular interactions through 3 different experiments in a tumor environment unrecognized by other models. This highlights the potential for CCIGAN to identify cellular protein interactions which account for variation in patient responses to cancer therapy, providing a framework for biological hypotheses which explain clinical outcomes on a cellular level.

A APPENDIX:

A.1 ARCHITECTURE DETAILS

The detailed architecture of our generator is shown on Table 6 . For all baseline models, we use the architecture based on the their original implementation. Due to the size of the cell patch is (64, 64), we reduce the size of hidden layers to fit our dataset. For fair comparison, we use the same reduction of hidden layers and the same discriminator architecture for SPADE, pix2pixHD, and CCIGAN.

Layers:



A.2 MODEL TRAINING:

G is the generator and D is the discriminator for CCIGAN. Given segmentation map S, ground truth Y and noise δ, the generated image is X = G(S, δ). The input of the discriminator is the cell image conditioned on the segmentation map S. We use LSGAN loss (Mao et al., 2017) in CCIGAN, which is defined as follows:

EQUATION

In addition to GAN loss, we also use feature matching loss (Wang et al., 2018a) during training expressed as:

EQUATION

where D j is j-th layer feature map of the discriminator for j ∈ {1, ..., J}, and N j is the number of elements in j-th layer. Consequently, the objective function for training is given as follows:

EQUATION

where λ = 10. Due to the size of cell patch is (64, 64), we do not use multi-scale discriminators and perceptual loss in CCIGAN and other baseline models e.g. SPADE and pix2pixHD.

In training, we use ADAM as the optimizer. The generator learning rate is lr G = 0.0004 and the discriminator learning rate is lr D = 0.0001. We train CCIGAN 120 epochs with a training set of 5648 cell patches. We train other baseline models for 120 epochs or until they converge (max 150).

The full details of training of CCIGAN and baselines are shown as Table 8 . The hyperparameters of each model are fine-tuned to get better performance. The training time was roughly equal for all models. In particular, CCIGAN was around 1.2 times slower than the SPADE baseline on a single Tesla V100 GPU. 

Metrics:

EQUATION

)

where • * can be either L 1 or L 2 norm, is the element-wise product, X i,m and Y i,m are the m-th channel of the i-th cell patch, U i ∈ {0, 1} (H,W ) is the mask matrix which masks all the cells in i-th patch. For any matrix A, sort(A) is the sort function that sorts all entries of A. The sorting function ensures our metrics are position independent and only measures the intensity of the generated image and ground truth. The score function L(X , Y) only computes the loss of sorted expression inside of the cells. Then we add penalization for expression outside of cells. The adjusted L 1 /MSE score is introduced as follows,

EQUATION

where 1 d is the matrix with all entries equal to 1. The (adjusted) L 1 /MSE scores of CCIGAN and baseline models are shown on Table 1 . A smaller score means a better result.

For any two images X, Y ∈ [0, 1] (H,W ) , the SSIM and MI are defined as:

EQUATION

where H(•) is entropy, µ X and σ X are the mean and standard deviation of X, c 1 , c 2 are constants.

Then the SSIM between

EQUATION

In cell based MI, test patches are processed at a cell-cell basis where their mutual information is computed with the corresponding cell in the ground truth. For the generated image X i of the i-th patch, we assume there are T i cells in the i-th patch. Then for each cell t, the pixels of m-th channel of the t-th cell in the i-th patch can be expressed as a vector x t i,m . Hence, the cell based MI is formulated as:

EQUATION

We report I(X ; Y) on Table 1 . The SSIM measures the similarity between the generated image and the ground truth. For SSIM, we use HLA Class 1 and dsDNA due to the their expressions in all cells. If all channels were considered, the SSIM would be uninformative due to the majority of the channels being blank or sparse. The MI measures the information shared between generated image and ground truth at a cell by cell basis where we consider all channels. Consider the example where a model generates no expression in marker m but the real data has expression in m, the MI would be 0 and vice versa. Higher SSIM and MI values mean better results. Table 1 demonstrates that CCIGAN outperforms or matches all other baselines on all reconstruction metrics.

A.3.2 WEIGHTED CENTROID:

For a generated cell image, its centroid is the mean position of all the points in the cell. We use pixel values as weights in computing the weighted centroid, now referred to as center of mass (COM). Given a cell image X ∈ R (H,W ) , with indices of the segmented cell V ⊆ {1, . . . , H}×{1, . . . , W }, the COM p = (x, ȳ) is defined as x = (x,y)∈V xXx,y (x,y)∈V Xx,y and ȳ = (x,y)∈V yXx,y (x,y)∈V Xx,y . In the PD-1/PD-L1 experiment, we compute the COM of the CD8 T cell (cell of interest) weighted by PD-1 expression, given as pCD8 , and the COM of all tumor cells weighted by PD-L1 expression, given as pTumor . Since T cells located within the tumor microenvironment often have upregulated expression of PD-L1, we assume that pCD8 should have the same COM as all of its surrounding tumor cells pTumor . The center of mass score is defined below as the relative distance between pCD8 and pTumor , where N is defined as the number of patches:

EQUATION

The projection function Proj(•) is used to project pTumor onto the CD8 T cell to ensure the expected COM of the tumor cells is inside of the CD8 T cell. As a reference we choose a random position pRandom in the CD8 T cell (PD-1) which replaces pCD8 in Eq. 22 and compute the random COM score to show the effectiveness of the result. An example illustration is given in Figure 6 A.4.

A.3.3 EARTH MOVER'S DISTANCE:

For each segmentation map i, we iteratively add its T i tumor cells around one CD8 T cell. The COM is defined for the t-th tumor cell as pTumor t,i for t ∈ {1, ..., T i }. We omit the subscript i when it is clear from context. The subsequent instances of the PD-1 COMs in the CD8 T cell by adding the t-th tumor are given by pCD8 t . Initially when there are no tumor cells, we define pCD8 0 as the centroid of the CD8 T cell. Based on the above setting, we define a vector v t which points from the centroid of the CD8 T cell pCD8 0 , to the COM of the t-th tumor cell pTumor t . We define vector u t which points from the previous COM pCD8

t-1 to the current COM pCD8 t of the CD8 T cell. We define θ t as the angle between v t , u t . If cos θ t > 0, that is to say if the cosine similarity is positive, the COM of a CD8 T cell pCD8 t , moves correctly towards the COM of the added tumor cell pTumor t . An illustration of the points and vectors is given in Figure 8 . Formally:

EQUATION

After obtaining the directional information, we use Earth Mover's Distance (EMD) (Rubner et al., 2000) to evaluate the changes in PD-1 expression of the CD8 T cell. The EMD, which measures the dissimilarity of two distributions, is used in this context to measure the protein localization shifts in PD-1 before and after adding a tumor cell. We consider each cell X in polar coordinates (r, θ) with respect to its centroid, integrate its expression along the radius coordinates, and evaluate the resulting histogram hist(X) along the angle coordinate. This allows for the definition of distance for moving one histogram to another, i.e. em(

X t i , X t-1 i ) = d EM (hist(X t i ), hist(X t-1 i

)), for the generated PD-1 expression of the CD8 T cell X t i when adding the t-th tumor cell. The final EMD score is defined as:

EQUATION

where the indicator function 1

(•) = 1 if and only if X t i > X t-1 i

, otherwise 1(•) = 0. This ensures that the biological constraint of PD-1 expression increasing as a response to added tumor cells is met. Recall, if cos θ t > 0, pCD8 t has moved in the direction of pTumor t , implying the shift in PD-1 expression is correct, and in turn increases the EMD. By contrast, the EMD score decreases when pCD8 t moves in the opposite direction. Example figures and illustrations showing this process are given in Figure 7 A.4. Using the EMD we define a randomized search algorithm for discovering other cell-cell interactions; their results and discussion are given in A.7. Based on the definition of EMD score, the positive EMD score is defined as:

EQUATION

The positive EMD score only evaluates the change in PD-1 expression when the COM of a CD8 T cell moves towards the COM of the added tumor cell. The projected EMD score is defined as:

EQUATION

The projected EMD score is the EMD score weighted by u t,i , i.e. the shift from the previous COM to the current COM of the CD8 T cell.

A.4 EVALUATION VISUALIZATIONS Here we provide some example visualizations and illustrations center of mass nomenclature, mass movement of PD-1 as a function of neighboring PD-L1, and process of computing EM distance.

We can observe in Figure 7 that the mass inside of the T cell in the PD-1 channel shifts as a response to surrounding tumor cell expressions of PD-L1. The surrounding tumor PD-L1 expressions pTumor t are shown in the third row on a cell by cell basis for t ∈ {1, ..., T i = 4}. Note that the 3rd column in PD-L1 has sparse expression. Finally the last row shows the PD-1 and PD-L1 channels superimposed into one channel. For this process, we focus on each instance of an added tumor.

We give an example in Figure 8 to illustrate the vectors v t and u t after adding t-th tumor cell for t = 1, 2 in computing EMD score, where v t and u t defined in Eq. 23.

A.5 RESULTS GRAPHS

The pan-keratin/CD8 experiment is similar to Figure 7 's orientation except the center cell (cell of interest) is a tumor cell (red) and the adjacent neighboring cells are CD8 T cells (orange). CCIGAN predicted a decrease in tumor cell pan-keratin expression with respect to increasing CD8 T cell area/number (Figure 9 ). This is juxtaposed to the tumor cell control where there is no change in the pan-keratin level as the number of neighboring tumor cells is increased.

SPADE does not predict a decrease in tumor cell pan-keratin expression with respect to increasing CD8 T cell area/number and shows no difference in pan-keratin expression trends between the T cell and control groups (Figure 10 ). pix2pixHD erroneously predicts an increase in tumor cell pan-keratin expression with respect to increasing CD8 T cell area/number and shows no difference in pan-keratin expression trends between the T cell and control groups (Figure 11 ).

CycleGAN fails to predict a decrease in tumor cell pan-keratin expression with respect to increasing CD8 T cell area/number and shows no difference in pan-keratin expression trends between the T cell and control groups (Figure 12 ). 

A.6 MODEL INTERPRETABILITY AND GENERATIVENESS:

Figure 13 shows the persistent vectors s i for all proteins. Note the similarity between CD3 and CD8 T cell protein markers and the similarity between dsDNA and HLA Class 1 surface membrane proteins (expressed in all cells). It is also important to make the distinction that sparse markers (while different) are similar in state. This is due to the lack of training data for rare cell types, making it difficult for the model to reason on such a small sample size. Figure 14 shows the generativeness of CCIGAN through an uncertainty map over 100 instances (random noise). An uncertainty map shows the differences per pixel (x, y) location. The higher intensities indicate a higher probability of changing at the specified (x, y) location.

:

Some channels were completely blank and others were indicator channels, i.e. CD45, a binary indicator for an immune cell.

The dark green cell is extremely rare, classified as "other immune" and is a noise class label.

The test dataset will be released along with trained models for reproducibility after the reviewing period.

Figures and Tables:

Figure 1: (A) Example segmentation patches (left column) and protein expressions (other columns). (B) Examples of how segmentation maps can be synthetically altered to pose different counterfactual biological scenarios for in silico hypothesis testing.

Figure 2: Examples generated from a segmentation for certain channels for different models. The segmentation patch is the one hot encoded patch collapsed and colored into 1 channel. The horizontal labels represent protein markers and the vertical labels are each of the generative models.

Figure 3: CCIGAN

Figure 4: Attention module. This illustrates an instance, focusing on the light blue block C 1 .

Figure 5: Example illustration of how a CD8 T cell's (orange) PD-1 histogram changes as a function of iteratively added tumor cells.

Figure 1 (B)'s row 2, 3 show examples of how directly manipulated segmentation maps can simulate the two microenvironments.

Average PDL1 expression of macrophages/monocytes on the compartmentalized tumor environment.

Figure 6: An example illustration of the center of mass (COM) nomenclature from section 5.1. Note the projection onto the CD8 T cell. This provides a more consistent measurement across different patches by projecting p Tumor onto the CD8 T cell.

Figure 7: Process of iteratively adding tumor cells. The added red cells are tumor cells (PD-L1) and the center orange cell indicates a CD8 T cell (cell of interest, PD-1). For this process, we focus on each instance of an added tumor.

Figure 8: An example illustration of the points and vector nomenclature from section 5.2. The blue dots are the expression of PD-1 and PD-L1 proteins. The cyan arrows show the vectors v t and u t . Note the shift in expression of the PD-1 as a response to the added tumor's PD-L1 expression.

Figure 9: CCIGAN experiment for adding CD8 T cells and tumor cells (control) around a tumor cell.

Figure 10: SPADE experiment for adding CD8 T cells and tumor cells (control) around a tumor cell.

Figure 11: pix2pixHD experiment for adding CD8 T cells and tumor cells (control) around a tumor cell.

Figure 12: CycleGAN experiment for adding CD8 T cells and tumor cells (control) around a tumor cell.

Figure 13: Persistent vectors s for various channels.

and cell based mutual information (MI) shown in Table 1. Equations and explanations are given in A.3.1. Bolded scores indicate the best scores. Comparison of conventional reconstruction metrics between different models.

Table content: <table><tr><td>Metrics</td><td/><td colspan="4">CCIGAN SPADE Pix2PixHD CycleGAN</td></tr><tr><td colspan="2">Adjusted L 1 Score</td><td>0.613</td><td>0.618</td><td>0.875</td><td>4.745</td></tr><tr><td>L 1 Score</td><td/><td>0.594</td><td>0.602</td><td>0.745</td><td>3.959</td></tr><tr><td colspan="2">Adjusted MSE Score</td><td>0.026</td><td>0.031</td><td>0.061</td><td>1.841</td></tr><tr><td>MSE Score</td><td/><td>0.026</td><td>0.031</td><td>0.055</td><td>1.523</td></tr><tr><td>SSIM</td><td/><td>0.810</td><td>0.802</td><td>0.709</td><td>0.394</td></tr><tr><td colspan="2">Cell Mutual Information</td><td>10.46</td><td>10.25</td><td>9.26</td><td>7.96</td></tr><tr><td>Experiment</td><td>Metrics</td><td/><td colspan="4">CCIGAN SPADE Pix2PixHD CycleGAN</td></tr><tr><td>PD1-PDL1 (Tumor)</td><td colspan="2">COM Score Random COM Score</td><td>10.46 12.27</td><td>10.69 12.16</td><td>12.81 12.96</td><td>13.38 12.45</td></tr><tr><td/><td colspan="2">EM Distance</td><td>151.56</td><td>-12.83</td><td>-12.24</td><td>-0.01</td></tr><tr><td>PD1-PDL1 (Tumor)</td><td colspan="2">Positive EMD Projected EMD</td><td>278.38 115.77</td><td>149.81 1.83</td><td>162.97 11.89</td><td>0.004 -0.008</td></tr><tr><td/><td colspan="2">Random EMD</td><td>-123.43</td><td>-83.11</td><td>-12.50</td><td>-0.006</td></tr><tr><td/><td colspan="2">EM Distance</td><td>-22.68</td><td>-18.88</td><td>-92.08</td><td>0.001</td></tr><tr><td>PD1-PDL1 (Endothelial)</td><td colspan="2">Positive EMD</td><td>35.92</td><td>22.46</td><td>109.82</td><td>0.01</td></tr><tr><td/><td colspan="2">Projected EMD</td><td>-9.29</td><td>-9.9</td><td>-70.96</td><td>0.003</td></tr><tr><td>Tumor-CD8 (Exp.)</td><td>t-test p-value</td><td/><td>-13.81 &lt; 10 -8</td><td>2.68 0.007</td><td>25.20 &lt; 10 -8</td><td>14.27 &lt; 10 -8</td></tr><tr><td>Tumor-Tumor (Control)</td><td>t-test p-value</td><td/><td>1.32 0.187</td><td>2.21 0.027</td><td>37.99 &lt; 10 -8</td><td>6.60 &lt; 10 -8</td></tr></table>



Table content: <table/>

Average PD-1/PD-L1 expression on the mixed tumor environment. The bolded cells indicate which cells are being measured.

Table content: <table><tr><td/><td>Microenvironments</td><td colspan="2">Patient A Patient B</td></tr><tr><td>PD-1 (T cell)</td><td>T cell / Tumor / Macrophages T cell / Endothelial / Macrophages</td><td>0.01886 0.00558</td><td>0.00131 0.00107</td></tr><tr><td>PD-L1 (Tumor)</td><td colspan="2">T cell / Tumor / Macrophages Endothelial / Tumor / Macrophages 0.00279 0.00649</td><td>0.00100 0.00046</td></tr><tr><td>PD-L1 (Macrophages)</td><td>T cell / Tumor / Macrophages T cell / Endothelial / Macrophages</td><td>0.00204 0.00068</td><td>0.00057 0.00047</td></tr></table>

Architecture details of CCIGAN's generatorwhere ResBlk is the residual block with skip connection used in ResNet(He et al., 2016), and SPADE is the spatially-adaptive normalization layer. The detailed architecture of our discriminator is shown on Table7.

Table content: <table><tr><td/><td colspan="2">Output Size</td><td>Generator</td></tr><tr><td>Linear</td><td>(128, 4, 4)</td><td/><td>Linear 128 × 2048</td></tr><tr><td>Upsampling</td><td>(128, 8, 8)</td><td/><td>Upsampling 2 × 2</td></tr><tr><td/><td/><td/><td>SPADE 128, Leaky ReLU</td></tr><tr><td>SPADE ResBlk-1</td><td>(128, 8, 8)</td><td/><td>Convolution 3 × 3 SPADE 128, Leaky ReLU</td></tr><tr><td/><td/><td/><td>Convolution 3 × 3</td></tr><tr><td>Upsampling</td><td colspan="2">(128, 16, 16)</td><td>Upsampling 2 × 2</td></tr><tr><td/><td/><td/><td>SPADE 128, Leaky ReLU</td></tr><tr><td>SPADE ResBlk-2</td><td colspan="2">(128, 16, 16)</td><td>Convolution 3 × 3 SPADE 128, Leaky ReLU</td></tr><tr><td/><td/><td/><td>Convolution 3 × 3</td></tr><tr><td>Upsampling</td><td colspan="2">(128, 32, 32)</td><td>Upsampling 2 × 2</td></tr><tr><td/><td/><td/><td>SPADE 128, Leaky ReLU</td></tr><tr><td/><td/><td/><td>Convolution 3 × 3</td></tr><tr><td>SPADE ResBlk-3</td><td colspan="2">(64, 32, 32)</td><td>SPADE 64, Leaky ReLU Convolution 3 × 3</td></tr><tr><td/><td/><td/><td>SPADE 64, Leaky ReLU</td></tr><tr><td/><td/><td/><td>Shortcut Convolution 3 × 3</td></tr><tr><td>Upsampling</td><td colspan="2">(64, 64, 64)</td><td>Upsampling 2 × 2</td></tr><tr><td/><td/><td/><td>SPADE 64, Leaky ReLU</td></tr><tr><td>SPADE ResBlk-4</td><td colspan="2">(64, 64, 64)</td><td>Convolution 3 × 3 SPADE 64, Leaky ReLU</td></tr><tr><td/><td/><td/><td>Convolution 3 × 3</td></tr><tr><td>Convolution</td><td colspan="2">(24, 64, 64)</td><td>Leaky ReLU, Convolution 5 × 5</td></tr><tr><td>Convolution</td><td colspan="3">(24  *  8, 64, 64) Leaky ReLU, Convolution 5 × 5</td></tr><tr><td>Group SPADE</td><td colspan="2">(24  *  8, 64, 64)</td><td>[SPADE 8] * 24</td></tr><tr><td>Modulation</td><td colspan="2">(24  *  64, 64, 64) (24, 64, 64)</td><td>[Outer Product 8 ⊗ 8] * 24 Convolution 1 × 1, Sigmoid</td></tr><tr><td>Output</td><td colspan="2">(24, 64, 64)</td><td>Sum residual, Sigmoid</td></tr><tr><td colspan="2">Layers Output Size</td><td/><td>Discriminator</td></tr><tr><td colspan="2">Conv-1 (32, 32, 32)</td><td colspan="2">Convolution 4 × 4, stride 2 Instance Norm, Leaky ReLU</td></tr><tr><td colspan="2">Conv-2 (64, 16, 16)</td><td colspan="2">Convolution 4 × 4, stride 2 Instance Norm, Leaky ReLU</td></tr><tr><td colspan="2">Conv-3 (128, 8, 8)</td><td colspan="2">Convolution 4 × 4, stride 2 Instance Norm, Leaky ReLU</td></tr><tr><td colspan="2">Conv-4 (256, 4, 4)</td><td colspan="2">Convolution 4 × 4, stride 2 Instance Norm, Leaky ReLU</td></tr><tr><td colspan="2">Conv-5 (512, 2, 2)</td><td colspan="2">Convolution 4 × 4, stride 2 Instance Norm, Leaky ReLU</td></tr><tr><td>Conv-6</td><td>(1, 1, 1)</td><td colspan="2">Convolution 3 × 3, stride 2 Sigmoid</td></tr></table>

Architecture details of CCIGAN's discriminator

Table content: <table/>

Peer review (iclr2020):
This paper applies the SPADE semantic image synthesis technique (with a custom attention mechanism) to MIBI-TOF data to examine hypotheses about cell-to-cell interactions in the context of an immune infiltrated tumor sample. I think that this is potentially an interesting application of GANs -- to generalize beyond specific gathered data and instead start engaging with counterfactual scenarios like *what effect does it have to add cell type X next to cell type Y*. Unfortunately, I think this paper is not yet finished with regard to both (1) conveying sufficient motivation for the use of image synthesis for distilling biological insights and (2) evaluating the success or failure of the technique primarily in terms of generating useful biological insights. Simultaneously, there are enough red flags with regard to knowledge of the data generation and underlying biology that it*s unclear if the authors could correctly sanity check any insights they extract from their model. More specific criticisms & suggestions 1) It would help to provide much clearer motivation for applying image synthesis to MIBI-TOF data. Section 1.2 skips straight from describing the MIBI-TOF instrument to “we made a new kind of GAN”. Again in the beginning of the Related Work the paper states: *We are interested in the task of generating biologically consistent expression patterns of cellular proteins given a segmentation map of cell neighborhoods”. But, why are synthetic images interesting given that we can actually look at real MIBI-TOF data. I start getting a sense of why this model might be interesting or useful only in Section 5 — more rationale is required earlier in the paper. 2) The evaluation section is hard to follow. I think it would be helpful to more clearly describe a larger set of biological phenomena that a practitioner would expect to see in the data, choose a single metric for each case, and show that that these phenomena are recapitulated. No one is going to trust a GAN to give them scientific insights unless they*re very confident that all known / simple cell-to-cell interactions have a clear signal. 3) *MIBI-TOF bombards a tissue sample with elemental metals tethered to respective antibodies for dozens of distinct cellular proteins and detects each to obtain image channels” — this is not an accurate description of MIBI-TOF, at least not the instrument I*m familiar with. Typically the tissue is first stained with antibodies tagged with heavy metals and the instrument then bombards the tissue sample with simple ions (like O2+), causing the release of metal ions. It seems very unlikely that bombarding anything with antibody/metal conjugates could be informative. 4) *Tumor cells could be identified by markers such as pan-keratin and beta-catenin” — I guess in the context of a tumor sample beta-catenin could be over-expressed but it*s present in pretty much every cell type, including lymphocytes (https://www.proteinatlas.org/ENSG00000168036-CTNNB1) Small nits: * Repeated use of “antibodies” in *Engineered antibodies for PD-1/PD-L1 antibodies” — maybe better to write “Antibodies which block the interaction of PD-1/PD-L1* * *While MIBI-TOF is capable of 36 different markers, we discarded uninformative and irrelevant markers1 resulting in M = 24.” — I’m really surprised that someone put 12/36 uninformative markers in a MIBI-TOF panel. Aren’t these tagged antibodies expensive? Can we at least get a list of what got discarded? * The model interpretability control seems weak in that lymphocytes are more likely to express MHC-I than tumor cells (which have a potential survival advantage from not expressing it) and tumor cells may actually have more dsDNA than lymphocytes due to changes in ploidy (or original differences in ploidy from e.g. liver cells).