Exploring Cellular Protein Localization Through Semantic Image Synthesis

1 INTRODUCTION . 1.1 BIOLOGICAL ROLE OF CELLULAR PROTEINS . Cell-cell interactions within the tumor microenvironment have been implicated in many facets of cancer pathogenesis and treatment . Most prominently , tumor cell evasion of immune surveillance ( Jiang et al. , 2019 ) , tumor metastasis ( Nishida-Aoki & Gujral , 2019 ) , and efficacy of cancer immunotherapies ( Lau et al. , 2017 ) have all been closely linked to the relationships between immune and cancer cells . These types of cell-cell relationships are generally governed by the interactions of cell surface proteins which drive cell behavior , gene expression , and survival . One of the most prominent examples of cellular proteins influencing disease progression is the case of PD-1/PD-L1 . PD-L1 is a protein often overexpressed on tumor cells and has the capacity to bind PD-1 on local T cells to downregulate their anti-tumor immune responses ( Iwai et al. , 2002 ) . Antibodies which interrupt the interaction of PD-L1 and PD-1 and allow the immune system to attack tumor cells , have become clinically influential treatments for a variety of cancers ( Pardoll , 2012 ) . This example highlights the value of accurately predicting cellular protein patterns which play key roles in disease processes . Exploring protein localizations in a multi-cellular system represents a challenge for which deep learning models are uniquely suited . However , to our knowledge no image-based deep generative models have utilized semantic image synthesis to produce accurate predictions of these biological phenomena . 1.2 FRAMEWORK AND NOVELTY OF A PREDICTIVE MODEL . A meaningful exploration of cell-cell interactions , particularly in the tumor microenvironment , requires a thorough understanding of the proteins expressed on and within a cell and its neighborhood . Multiplexed ion beam imaging by time-of-flight ( MIBI-TOF ) represents a novel technology that can accurately quantify and spatially resolve cellular protein expressions at the single cell level within tissue samples . Given a tissue sample that is first stained with protein-specific antibodies tethered to elemental metals , MIBI-TOF bombards the sample with atomic ions ( i.e . O+2 ) from a primary ion beam . This causes the release of elemental isotopes and tissue-specific material which can be quantified in a mass spectrometer ( Angelo et al. , 2014 ) . The cellular proteins characterized by this technique indicate specific cell types ( i.e . immune cells , tumor cells ) , cell status ( i.e . markers of proliferation ) , and immunomodulation . Figure 1 ( A ) shows an example spatial orientation of cell types and some selected cellular protein expressions . Here , we propose a novel protein based attention mechanism for a convolutional Generative Adversarial Network ( GAN ) with the capacity to provide accurate conditioned predictions of cellular protein localizations . The model , Cell-Cell Interaction GAN ( CCIGAN ) , learns a many to many mapping between different cell types and different protein markers . CCIGAN is trained on semantic segmentation maps of cell tissue samples to identify and associate cell type , shape , and the identity of its cellular neighbors with cellular protein expressions for each cell . The network was then applied to simulated segmentation maps to understand how cells vary their protein localizations . CCIGAN demonstrates its ability to probe elements of oncologic processes by : 1 . Independently recapitulating established patterns of biological phenomena , thereby demonstrating the accuracy of its predictive power . 2 . Yielding quantitative and spatial information on specific cellular protein modulations as a result of immune cell - cancer cell interactions . 3 . Allowing for biological data to be generalized beyond traditional in vitro scenarios where specific , rare , or hypothetical cell-cell events can be posed in counterfactual scenarios and the results of their interactions predicted . The ability for CCIGAN to generate subcellular protein predictions represents a step forward in the ability to understand cellular relationships within a microenvironment . Rather than assessing in vitro incidence of cell interaction phenomena as is done by MIBI-TOF , CCIGAN allows for hypothetical biological situations to be generated . Figure 1 ( B ) illustrates examples how a segmentation map can be directly manipulated to pose various biological scenarios . In other words , individual cellular responses to challenge or proximity of cells of any identity can be assessed without having to seek this specific occurrence within the available biological tissue sample . These capabilities allow the model to provide insight on hypothetical interaction events which shed insight on disease pathogenesis and which can not be feasibly achieved by biological investigation alone . MIBI-TOF is a labor intensive process which can make the data collection process cumbersome . Deep learning models such as CCIGAN have the potential to be useful tools which can extend on MIBI-TOF capabilities and result in more rapidly synthesized data . CCIGAN , once trained on MIBI-TOF data sets and properly replicating known cell-cell interaction patterns , can be used as a robust means to provide high throughput readouts on myriad single cell scenarios which may normally take many iterations of MIBI-TOF experiments to investigate . Recent work by ( Wu et al. , 2019 ) demonstrated the utility of conditional GANs for adding a depth dimension to images ; we demonstrate learning contextual cell-cell interactions , facilitating rapid hypothesis testing to assess biological environments . This provides valuable insight into a complex system that normally would have taken resource-intensive wet lab experiments to interrogate . 2 RELATED WORK . We are interested in the task of generating biologically consistent expression patterns of cellular proteins given a segmentation map of cell neighborhoods . Specifically , we want to learn a generative model that simultaneously produces high quality maps of protein expression for individual cells that are probabilistically consistent when conditioned on the same factors , e.g . similar cell neighborhoods should produce similar expression patterns . Such generative models typically take the framework of a generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ; Brock et al. , 2018 ) or a variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) . Within these generative modeling techniques , image translation focuses on learning a many to many mapping to transform data from one domain to another . One approach is to model the translation task , known as image synthesis , with conditioning such as Pix2Pix ( Isola et al. , 2016 ) , Pix2PixHD ( Wang et al. , 2018a ) and CycleGAN ( Zhu et al. , 2017 ) . By using image synthesis , a model is able to distill more information from a segmentation map to various protein channels . As MIBI-TOF measures information such as protein localizations at a subcellular resolution , framing the problem through an image synthesis framework allows a model to see how predictive neighborhoods and cell types are of a cell ’ s phenotype . Recently , Park et al . ( 2019 ) proposed spatial adaptive normalization ( SPADE ) to synthesize images using segmentation maps by learning fully convolutional normalization parameters based on their segmentation conditioning . For each layer , semantic information is retained by allowing the network to learn from the segmentation map directly and modulate the current layer . They demonstrate impressive quality and diversity in generated images , especially modeling objects in context . Using SPADE normalization , CCIGAN is able to condition on surrounding cell neighbors and capture cellcell interactions in a local receptive field . Despite image generation , there has been no work done towards understanding such generations in context , particularly in biological images . Attention layers ( Vaswani et al. , 2017 ; Wang et al. , 2018b ; Zhang et al. , 2018 ) have also been added to generative models with great success . Self-attention was initially proposed in machine translation tasks to help model long distance dependencies that occur frequently in language ( Sutskever et al. , 2014 ) , which led to the idea of external memories as a persistent state to model long range dependencies ( Sukhbaatar et al. , 2015 ) . While convolutional neural networks are apt at exploiting local structures of patches , they may struggle to model global structures . For this reason visual attention allows the network to enhance activations in interesting parts of an image . Goodfellow et al . ( 2014 ) and Brock et al . ( 2018 ) both achieve state-of-the-art unconditional generative modeling using self attention GAN ( Zhang et al. , 2018 ) . We propose a specialized attention module conditioned on different proteins to mimic real world protein markers . 3 DATASET . 3.1 MULTIPLEXED ION BEAM IMAGES ( MIBI ) . All experiments are performed on data obtained through MIBI-TOF characterized tissue samples collected from triple-negative breast cancer ( TNBC ) patients . By simultaneously imaging over 36 protein markers , MIBI-TOF is able to identify cell type as well as provide detailed information of sub-cellular structure , cell neighbors , and interactions in the tumor microenvironment across these different marker settings . Each of these markers m ∈ { 1 , ... , M } , given as a channel taking on real values continuous in [ 0 , 1 ] at each ( x , y ) coordinate , demarcates a different cellular subtype and furthermore , is indicative of the functional properties of a cell . While MIBI-TOF is capable of 36 different markers , we chose markers exhibiting cellular protein localization and disregarded markers that were blank or used simply for cell-typing 1 resulting in M = 24 . A full list of these markers is given in A.8 . 1Some channels were completely blank and others were indicator channels , i.e . CD45 , a binary indicator for an immune cell . The MIBI-TOF data collected from the TNBC tissue allows for the differentiation between a wide variety of cell types within the samples . For instance , cells positive for the marker CD3 could be identified as T cells , and then subdivided into cytotoxic or helper lineages by the presence of markers CD8 or CD4 , respectively . Tumor cells could be identified by markers such as pan-keratin and overexpressed beta-catenin . Along these lines , a wide variety of cellular proteins identified by MIBI-TOF could characterize cell interactions as well as immunomodulatory processes occurring within the microenvironment . MIBI-TOF data is fundamentally different than typical RGB images . This poses unique challenges in image modeling and characterization . Such challenges stem from each marker being conditionally expressed on the cell type in its respective channelm . Using a simplified 3 channel multiplex setting , a T cell expresses signals in the CD3 , CD8 channels ( indicators for immune cells ) but not in a pankeratin channel ( indicator for tumor cells ) . Another problem is the sparsity of the data , meaning either some expressions for rare cell types are rarely observed or have weak signals . For example , Figure 2 includes a segmentation of a CD8 T cell ( orange , top right corner ) , where other models fail in generating correct CD3 expression , if at all2 . Lastly , the noisy nature of the data leads to inaccurate cell type classifications , creating inconsistent pairs of labels and outputs during training . These issues make it especially difficult for an RGB multihead decoder to output multiple channels in a biologically accurate manner . Without addressing multiplexed data , a decoder would equally attend to every location of the current latent representation , even if it is irrelevant to the current protein . Furthermore each protein channel in the output has its own sensitivities to signal intensities and noise , suggesting each channel requires a unique prior and that equal attention would be problematic . It follows that special care must be given towards modeling specific channels and the multiplexed nature of MIBI-TOF images .

Summary: The manuscript proposed a new method to model the data generated by multiplexed ion beam imaging by time-of-flight (MIBI-TOF). Essentially, the model leans the many-to-many mapping between the cell types and different protein markers' expression levels. Compared with the other mainstream GAN methods, the authors show the proposed method, CCIGAN, can outperform them in terms of generating the expression map of different protein markers given the segmentation of cell types. The manuscript also has an in-depth discussion of the biological meaning of the learned model as well as the learned vectors.