Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Authors: Dihan Zheng
Bo Huang
Venue: ICML 2025 poster (2025)
Keywords: AI for biology
Image generation
Diffusion model
MultiModal learning
Cell biology
Primary area: applications->everything_else

Abstract: Fluorescence microscopy is ubiquitously used in cell biology research to characterize the cellular role of a protein. To help elucidate the relationship between the amino acid sequence of a protein and its cellular function, we introduce CELL-Diff, a unified diffusion model facilitating bidirectional transformations between protein sequences and their corresponding microscopy images. Utilizing reference cell morphology images and a protein sequence, CELL-Diff efficiently generates corresponding protein images. Conversely, given a protein image, the model outputs protein sequences. CELL-Diff integrates continuous and diffusion models within a unified framework and is implemented using a transformer-based network. We train CELL-Diff on the Human Protein Atlas (HPA) dataset and fine-tune it on the OpenCell dataset. Experimental results demonstrate that CELL-Diff outperforms existing methods in generating high-fidelity protein images, making it a practical tool for investigating subcellular protein localization and interactions.

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Dihan Zheng'
Bo Huang 123
Abstract
development of generative models has enabled researchers to design functional proteins (Madani et al., 2023; Dau- paras et al., 2022) and drug-like molecules (Isigkeit et al., 2024). These computational methods allow for large-scale virtual screening, significantly reducing the costs and re- sources associated with experimental validation. The advent of those technologies presents significant opportunities for biomedical research, potentially accelerating advancements in therapeutic target identification, drug discovery, and the investigation of biochemical pathways (Palma et al., 2012).
Fluorescence microscopy is ubiquitously used in cell biology research to characterize the cellular role of protein. To help elucidate the relation- ship between the amino acid sequence of a protein and its cellular function, we introduce CELL-Diff. unified diffusion model facilitating bidirectional transformations between protein sequences and their corresponding microscopy images. Utilizing reference cell morphology images and protein sequence, CELL-Diff efficiently generates cor- responding protein images. Conversely, given a protein image, the model outputs protein se- quenees. CELL-Diff integrates continuous and diffusion models within a unified framework and is implemented using a transformer-base work. We train CELL-Diff on the Human Protein Atlas (HPA) dataset and fin-tune it on the Open- Cell dataset. Experimental results demonstrate that CELL-Diff outperforms existing methods in generating high-fidelity protein images, making it practical tool for investigating subcellular pro- tein localization and interactions.
In this work, we focus on the relationship between protein sequences and their cellular functions as characterized by microscopy images. Specifically, we focus on fluorescence microscopy. which is ubiquitously used in nearly all cell biology research. Fluorescence microscopy images provide extremely rich information for proteins of interest in the cellular context, such as their expression level, subcellular distribution, and molecular interactions as can be measured by spatial colocalization. Such information characterizes protein functional activities as well as the physiological and pathological state of cells. Disease-causing genetic muta- tions can alter the amino acid sequence of proteins, resulting in changes in image phenotypes by modifying gene expres- sion patterns, reshaping molecular interaction profiles, or globally perturbing cellular states. As first step towards building a model that connects the sequence of proteins and their cellular images, recently, Khwaja et al. (2024b) proposed CELL-EE, text-to-imaae transformer that predicts fluorescence protein images from sequence input and cell morphology condition. Furthermore, CELL-E2 (Khwaja et al., 2024a) was developed to enhance the image genera- tion speed of CELL-E by utilizing the idea of masked token prediction (Chang et al., 2022). Additionally, CELL-E2 fa- cilitates the bidirectional transformation between sequences and images. However, their image model only allowed out- puts of highly blurred images lacking fine details discern any of the subcellular structures other than the most promi- nent one (i.e., the nucleus), restricting their applicability only to the study of very limited set of sequence features (i.e., the nuclear localization signal).
net-
1. Introduction
Protein sequences inherently encode their functions, and predicting these functions solely from sequence information has become a critical area of research. With the develop- ment of artificial intelligence, learning-based methods are increasingly employed to predict wide range of protein properties, including structural conformation (Jumper et al., 2021; Back et al., 2021), I), interaction partners (Evans et al., 2021), and subcellular localization (Almagro Armenteros ct al., 2017; Khwaja et al., 2024b). Concurrently, the rapid
'Department of Pharmaceutical Chemistry, UCSE, San Fran- cisco, CA 94143* 'Department of Biochemistry and Bicphysics, UCSF, San Francisco, CA 94143 'Chan Zuckerberg Biohub San Francisco, San Francisco, CA 94158. Correspondence Huang whw..uunng@e...cc @ucsf.edu>>
expand the application of sequence-to--ell-imae gener- ative models, we introduce CELL-Diff, unified diffusion model that enables bidirectional transformation between
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Given as condittion
Bidircctional Generation
Cell dataset. Experimental results show that our model generates more detailed and sharper protein images than previous methods.
Microtubule
MISTLIASEURKRFID
CFII-DIf
TIKRNEITY
Dilicie
PLD SIDPPLLEE FLLGAAAG
Nucieus
WRKEPILNNIIPSSIRN AKLSRAAN AALLSRARARRIR
GOKINDLDDVOODD YHHTFFEMI GSWSFG
2. Related works
DYFKCLACKMA...
Cell images
Protein imsaee
Protein sequence
Multi-modal generative modeling can be formalized as learning the conditional or joint distribution between modal- itics. Representative applications include text-to-image gen- eration (Ramesh al., 2021; Ding et al., 2021; Nichol et al., 2022), image-to-text generation (image caption- ing) (Mokady al., 2021; Chen et al., 2023), text-to-video generation (Ho et al., 2022), and text-to-speech (Chen et al., 2021; Popov et al., 2021). Most of these approaches rely on diffusion models or auto-regressive models for the gener- ation and typically focus on unidirectional transformation. However, our goal is to achieve bidirectional generation, which requires learning joint distributions. To achieve this, Hu et al. (2023) proposed discrete diffusion-based model for learning the joint distribution between images and text, though its scalability remains unexplored. Bao et al. (2023) introduced Unidiffuser, a unified diffusion model capable of unconditional, conditional, and joint generation. The key observation of Unidiffuser' is that the learning objective of the diffusion score function can be unified in a general framework with multiple diffusion time steps. Furthermore, Zhou et al. (2024) developed Transfusion, which integrates auto-regressive and diffusion models for both single and cross-modality generation. Transfusion combines the auto- regressive loss with diffusion, training a single transformer model using an extended causal mask. These methods gen- erally depend on large pre-traines encoders for images and text, such as CLIP (Radford al., 2021) and VQGAN (Esser et al., 2021). However, for microscopy images, the variabil- ity in equipment and experimental conditions limits the availability of such robust image encoders, making the di- rect application of these models challenging. Indeed, the two previous CELL-E (Khwaja et al., 2024b) and CELL-E2 (Khwaja et al., 2024a), which both used VQGAN, only produce coarse-grain images that have too much blur to distinguish fine-scale subcellular structures such as the cytoskeleton. As for CELL-Diff, we combine continuous and discrete diffu- sion to enable bidirectional transformation between protein images and sequences, which achieves high-fidelity image generation while preserving the ability to resolve detailed subcellular structures.
Figure 1. Given cell morphology images as conditional input, CELL-Diff facilitates bidirectional generation between protein sequences and images.
protein sequences and their corresponding microscopy im- ages. Specifically, by utilizing cell morphology images including the nucleus and cytoplasmic markers such as en- doplasmic reticulum (ER) and microtubules as conditional input, CELL.-Diff can generate detailed protein images from given protein sequences. Conversely, it can also output protein sequences when provided with microscopy images, as shown in Figure 1. To enable this bidirectional trans- formation, CELL-Diff employs the continuous diffusion model for generating microscopy images and the discrete diffusion model for redesigning protein sequences, which can be integrated within a unified framework. Specifi- cally, we design a transformer-bait
U-Net model (Ron- neberger et al., 2015; Peebles & Xie, 2023) enhanced with the cross-attention mechanism from Stable Diffusion (Rom- bach et al., 2022). This model processes sequences and images as inputs, simultaneously modeling the conditional distributions for image-to songgetto-eqque sequeneeec and sequence-to -o-imagge---- transformations. The final objective function comprises the noise prediction loss for the continuous diffusion model and the masked value prediction loss for the discrete diffusion model. We evaluate CELL-Diff on HPA dataset (Digre & Lindskog, 2021), which provides cellular microscopy im- ages of human proteins based on fixed immunofluorescence staining. Subsequently, we fine-tune the model on the Open- Cell dataset (Cho et al., 2022), which offers live microscopy images of different human cell lines, each tagged with a single protein via CRISPR/Cas9 gene editing.
generators,
We present CELL-Diff, diffusioo--assd generative model that enables conditional bidirectional genera- tion of protein sequences and their corresponding mi- croscopy images. By integrating the continuous dif- fusion and discrete diffusion models, CELL-Diff can be trained within unified framework. We propose wranformmr-bbaee U-Net model for implementing CELL-Diff, which effectively integrates information from images and sequences.
3. Technical background
Before delving into our unified diffusion model, we briefly introduce the background of diffusion models applicable to continuous and discrete state spaces. Specifically, we employ the continuous and discrete diffusion models for
We train CELL-Diff on the HPA dataset using different conditional cell images and fune-tune on the Open-
2

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
generating mieroscopy images and protein sequences, re- spectively.
where (O, Ta) and eg is noise prediction network. In this formulation, Po can be parameterized by co. enabling the generation of new samples through the reverse process.
3.1. Diffusion model for continuous state spaces
3.2. Diffusion model for discrete state spaces
Let lo be a continuous random variable in RR where d de- notes the dimension, and let h..- [- be sequence of latent variables, with as the index for diffusion steps. The diffusion model involves two processes: the forward process and the reverse process. In the forward process, the diffusion model progressively injects noise into the initial data Io. transforming it into a Gaussian random variable Ir. In the reverse process, the model learns to invert the diffusion process through denoising model and generate new data by gradually eliminating the noise.
Several distinct diffusion models designed for discrete data (Austin et al., 2021; Hoogeboom et al., 2021). This sec- tion focuses on the order-agnostic Autoregressive Diffusion Models (OA-ARDM) (Hoogeboom et al., 2022).
Let S = able, where D denotes the maximum sequence length, VI [I,...,D). S, (1...., K} with K categories. Denote SD as the set of all permutations of (1, (1..... D}. and assume represents random ordering in Sp. Applying Jensen's 's inequality, we obtain:
be a multivariate random vari-
Forward process. The forward process involves inject- ing noise into the initial data. Given a variance schedule (3.)=1, the forward process is defined as:
(og((S) = log E. u(s 2 Bo- log (S
(5)
(L,, (1) = N(V 1,, 3ela), t = T.
where U(Sp) denotes the uniform distribution over Sp. Following order o, p(So()) can be factor- ized as Σ p where Sa(en) =
(1)
Let a = - Be and Cr = II 0g. Then, for any arbitrary time step t, it holds that - -NVa (1 (1-āa) La). Consequently, for sufficiently large T, this process will transform Ig into an isotropic Gaussian variable.
(So(1).... 1,)( Combining this with (5), we have:
logr(S) 2 Booe((s))
log
Reverse process. The goal of the reverse process is to generate new samples from p(Io) starting from Gaussian random variable Ir ~ (0, Ia). The reverse process is defined by a Markov Chain with trainable transitions:
[=1
(6)
Exco(2 log
D +++1
(2) Here, Hi represents parameterized neural networks designed to estimate the means from the current state, and of denotes the variance.
Po (0 =
( 1= La),
Therefore, denote fa as the neural network, C as the categor- ical distribution, the loss function for OA-ARDM is
Exer(Z)) log C(S fo (<))))
LOA-ARDM = -ES,t.
D-++1
Training objective. The training objective function can be derived using variational inference. Instead of optimizing the intractable log-likelihood function log P(To), the diffu- sion model maximize its ELBO:
(7)
where a ~ U(Sp) and l ~ u(l...., D).
The objective function of OA-ARDM corresponds to the "Masked Language Modeling" training objective proposed in BERT (Kenton & Toutanova, 2019) with reweighting term. At each training step, we first sample time step t from ((1, D), followed by random ordering from u(Sp). We then feed S Sa(<e) into the model, which predicts the remaining values So(20) In the generation step, we first sample a random ordering and then generate the values according to that order. These processes are facilitated through a masking operation, see Appendix A for the details.
By log (oo II) - DKi. (H)
Σ DKL (q
I.))
where q Le, Io) has an formulation as ( - 1o I-a,
|oll (1 (1 (1) Ia). Combining with (2), the KL I-ū.
1-o+
divergence of two Gaussian distributions has a closed-form formulation.
4. Methodology
To further simplify the computation, Ho et al. (2020) pro- posed a training objective based on a variant of the ELBO in (3) as
In this section, we introduce our unified diffusion model for generating microscopy images and protein sequences. Let Ip represent the protein image, I represent the cell morphology image, and S represent the protein sequence.
LDDMM = El..telleoo + VI- V1-aie, 1) - - (4)
3

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Noise prediction
In
Masked value prediction
is formulated
Eg
Σ
log (o P
Ir r
D - is +1 1==1
kEO(21s)
Model
Model
where ((D))
M
Using the same parametrization technique as shown in (4) and (7), let fa represents the neural network. The training objective for protein images is defined as:
fs
T.9
S
L1 = Exps p S,1,, fo (S, In, 11. rell)) cll
(10)
Figure 2. Training losses of CELL-Diff.. During each training itera- tion, the protein image Iee and sequence S are transformed using the forward processes of the continuous and discrete diffusion mad- els, with randomly sampled time steps f and is, respectively. The network model is tasked with predicting the noise in the protein image and the masked values from the protein sequence, corre- sponding to the noise prediction loss Lı and the masked value prediction loss Ls.
where 11, Van, I vI age. The training objective for protein sequences is defined as:
EmEr(2 log C(S; 1)
Ls = Eine, Jeell S,tg.o
(11)
D- is +
where + = (o((o((Cta) Ip .. D.
Finally, by combining (10) and (11), and introducing bal-
ancing coefficient X, the total loss for the proposed CELL- Diff model is given by:
The task of protein image prediction involves sampling from the conditional distribution [S, ) while the task of sequence generation involves sampling from P[S][rmn, ) To achieve these goals within unified diffusion model, we
LCELL-DIt = L1 + XLs.
(12)
The training strategy is illustrated in Figure 2. During the training phase, the protein image loss L1 is computed by applying the forward diffusion process the image, where Gaussian noise is incrementally added. The model then takes the corrupted image and the corresponding protein sequence as inputs to predict the noise introduced into the protein image. Similarly, the sequence loss L. is computed by partially masking the sequence using a randomly gen- erated mask. The model uses the masked sequence and microscopy images as inputs predict the masked values within the sequence.
simultaneously model these two conditional distributions, which involves modeling a continuous variable ppro and discrete variable S.
4.1. Proposed method
To simultaneously model p (S, reell) and p(S ppor p we aim maximize the conditional log-likelihoods log p( [pror] [S, Ieell and (() Icell).
For ( ) based on the continuous diffusion model assumption described in Section 3.1, we introduce latent variables I = (T Following the training objective of continuous diffuusion (3), the conditional ELBO for log p(Irro SS Ieell) is expressed as:
4.2. Model details
Latent diffusion model. To reduce computational costs, employ the latent diffusion model (Rombach al., 2022; Peebles & Xie, 2023) for modeling microscopy im- ages. Specifically, we first train a Variational Autoencoder (VAE) (Kingma, 2013) to compress microscopy images into lower-dimensional spatial representations. After training, the VAE model is fixed, and the CELL-Diff model is trained using the latent representations of microscopy images pro- duced by the VAE encoder. During image generation, latent representation is sampled from the diffusion model and subsequently decoded into a full-resolution image using the learned VAE decoder.
Bg
log Po [ S, ( - DKL (P)((o)))
DKL (q I,, Inoo) (I,, reell))
For (S | model approach described in Section 3.2, introducing a random ordering a for the decomposition of the sequence S. Based on (6), the conditional ELBO for log (S
[ (eell, we follow the discrete diffusion
Inference. After training. we can generate samples from two conditional distributions: Is reell) and
[ Tooo
roell)

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
tensors are split into image and sequence feature tensors. The image feature tensors are then upsampled and com- bined with the downsampling features to output the noise from the protein image. The sequence feature tensor is pro- cessed using a linear projector to predict the masked values. The upsampling and downsampling blocks in the U-Net consist of residual and attention blocks. To enhance the integration of sequence information within the image pro- cessing component, we utilize cross-attention mechanisms with the attention blocks from Stable Diffusion (Rombach et al., 2022). Furthermore, we employ the adaptive layer norm zero (adaLN-Zero) conditioning method (Peebles & Xie, 2023) for incorporating the diffusion time step f1. The network architecture illustrated in Figure
Cross attention
Linear Projection
Attention Block
Nx
Residual Block
adaLN-Zero
Ridirectional Transformer
Attention Block
Nx
Cross attention
Residual Block
Pre-trained * ESM2
tr
1 -g
M
M
M
5. Experiments
Figure Network architecture CELL-Diff. Microscopy images are embedded into a latent sequence through residual and attention blocks. The protein sequences are embedded using a pre-trained ESM2 model (Lin et al., 2022). These embeddings are concate- U-Net
5.1. Datasets
Human Protein Atlas. The Human Protein Atlas (HPA) dataset (Digre & Lindskog, 2021) includes immunofluo rescence images across various human cell lines with the proteins of interest stained by antibodies. It provides cel- lular images for 12,833 proteins and corresponding cell morphology images consisting of staining for the nucleus, ER, and microtubules. For each protein, the dataset includes multiple microscopy images from different cell lines. The corresponding protein sequences can be accessed from the UniProt dataset (UniProt Consortium, 2018). In total, we have collected 88,483 data points, each containing a protein sequence, a protein image, a nucleus image, an ER image, and a microtubule image.
nated and processed by a bidirectional transformer. architecture (Ronneberger et al., 2015) is employed output the noise the protein image, while a linear projection utilized to predict the masked values in the protein sequence. Cross-attention mechanisms are implemented to enhance information integration from images and sequences.
[S]][rr ) Specifically, to generate the protein im- age Irse, we utilize the conventional reverse diffusion pro- cess as shown in (2), conditioning on the unmasked pro- tein sequence S and the cell image Isell The network model employed for generation (S, Ir, I where I....,T. For the generation of the protein se- quence S, we utilize the reverse process of discrete dif- fusion OA-ARDM (Hoogeboom et al., 2022). We first sam- ple a random ordering a, and then generate sequence from ((Sa(tt)) (So(tts) I ] where is = -I,...,) The network model in this scenario is Im , I The sampling algorithm for OA-ARDM is shown in Algorithm
O OpenCell. The OpenCell (Cho al., 2022) dataset pro- vides library of 1,311 CRISPR-edited HEK293T human cell lines, each with target protein fluorescently tagged us- ing the SpiiemmNeonGrree22 system. For each target protein, OpenCell provides 4-5 confocal images along with a refer- ence nucleus image. The cells were imaged live, offering a more accurate representation of protein distribution than the immunoftuorescence images from HPA. Notably, 1,102 pro- teins are common between the HPA and OpenCell datasets. In total, we collected 6,301 data points, each containing protein sequence, a protein image, and a nucleus image.
Network architecture. As shown in (10) and (11), the net- work model f takes four inputs: the protein sequence S, the protein image 1root the cell image Isell, and the diffusion time step f1. To process the protein and cell images, we first concatenate them and then apply the commonly used U-Net architecture (Ronneberger et al., 2015). The concatenated images are fed into series of downsampling blocks, trans- forming into image embeddings. The protein sequences are embedded using a pre-trained ESM2 model (Lin et al., 2022), with its parameters held fixed during training. Then, the image and protein embeddings are concatenated and processed using a bidirectional transformer. After passing through the transformer module, the concatenated feature
Given the size limitations of the HPA and OpenCell datasets, particularly in the diversity of protein sequences, we ran- domly selected 100 proteins from the shared subset between the two datasets the test set, leaving the remainder for training. The test set for HPA and OpenCell contains 714 and 473 data points, respectively.
5.2. Implementation details
We begin by training VAE with the same network architec- ture from Stable Diffusion on the training sets of the HPA
5

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Protein
Real
CELL4
CELL-DIf
Figure 4, Visual results of protein image generation on HPA dataset.
Table Comparison protein image generation performance on HPA and OpenCell datasets. "Nucl" denotes the nucleus image, "ER" denotes the endoplasmic reticulum image, and "MT" denotes the microtubule image. "FID-T" indicates the FID computed using the thresholded protein image, and "FID-O" indicates the FID computed using the original protein image.
Dataset
Method
Cell image
MSFR (nm)
IoU
FID-T 4
FID-O 1
CELL-E2
Nucl
1824
0.462
77.6
166.4
HPA
CELL-DifT
Nucl
641
0.484
60.1
51.1
Nucl, ER
642
0.580
55.9
60.0
HPA
CELL-Diff
Nucl, MT
644
0.616
51.0
47.6
Nucl, ER, MT
644
0.635
50.4
45.6
CELL-E2
Nuel
1165
0.466
77.0
245.8
OpenCell
CELL-Diff
Nucl
529
0.492
62.4
68.9
and OpenCell datasets for compressing microscopy images into a latent representation. For the HPA dataset, images are randomly cropped to size 1024 and then resized to 256, while images from the OpenCell dataset are directly cropped to 256 pixels. Data augmentation is applied using random flips and rotations. The latent representation has dimensions of 4 x 64 x 64. The KL loss coefficient is set The learning rate is initialized using a linear warm-up strategy, increasing from ( to 3 x 10-4 over the first 1,000 iterations, followed by linear decay to zero. The batch size
attention. CELL-Diff is trained with 200 diffusion steps using the cosine noise schedules (Peebles & Xie, 2023), and use DDIM (Song et al., 2020) with 100 steps to accelerate the sampling speed. The weighting coefficient À in (12) is 1, and the maximum protein sequence length is 2,048. All models are trained using Nvidia H200 GPUs.
to x 10 10-5,
5.3. Protein image generation
We evaluate the protein image generation performance of CELL-Diff. Given that the protein image prediction prob- lem is relatively new, we compare CELL-Diff with the most closely related method, CELL-E2 (Khwaja et al., 2024a). The outputs for CELL-E2 are heat map that indicates the localization distribution and a binary protein image, see Figure 4.
192. The VAE is trained for a total of 50,000 steps the HPA dataset and fine-tuned for 20,000 steps the OpenCell dataset.
Next, we fix the VAE model and train the latent diffusion model. CELL-Diff is pre-trained on the HPA dataset and fine-tune on the OpenCell dataset. Both pre-training and 50,000 iterations using the
To provide quantitative comparison, introduce the Maximum Spatial Frequency Resolvability (MSFR) for mi- croscopy images to measure its capability to discern fine structural details. Given a microscopy image I, we de- fine the Fourier Ring Power Spectral Density (FRPSD) as FRPSD(r) = Er.er [i(r:))", where denotes the Fourier transform of and 7% denotes the pixel element at radius r.
fine-tuning are conducted Adam optimizer (Kingma& & Ba, 2014). The learning rate is initialized using linear warm-up strategy, increasing from 0 to 10 over the first 1,000 iterations, followed by linear decay to zero. The batch size is set to 64. The sequence embedding dimension is 1280, and the bidirec- tional transformer module consists layers with 8-head
6

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Microtubules
Protein: TUBB4A
Protein: TUBAIA
Protein: KIFISA
Protein: EML4
Overlaid image
Cell merphology images
Red: TUBAIA
Green: KIFISA
Blue: EML4
Nucleus
Figure 5. Virtual staining using HPA data. From identical cell images, CELL-Diff generates staining results for various proteins.
The MSFR is then defined as:
NLS
AVKRPAATKKA GOAKKKKLD
PAAKRVKLD
ROORKR
MSFR f f Image Size x Pixel Size"
(13)
Prodicted images
where i is defined the first frequency index that satisfies:
FRPSD(r) > 10-3, r<i FRPSD(r) < 10-3, r=i
(14)
10O
2.03
1.59
1.47
NES
EMI KELNEALEL
LOUPPLERKILD
LIRTLKELKY
We also employ the Intersection over Union (IoU) metric, which measures the similarity between two masks and is commonly used in image segmentation tasks. To calculate IoU, we apply median value thresholding to the original protein images to generate binary masks, while for CELL- E2, we use the predicted binary image. Additionally, we compute the Fréchet Inception Distance (FID) (Heusel et al., 2017) score evaluate the similarity between the real and predicted images. FID learning-based metric that eval- uates the quality of images generated by generative mod- els. It measures the similarity between the generated and real images regarding their feature distributions. Lower FID scores indicate that the generated images are more similar to the real images. To compute FID, we concate- nate the protein and nucleus images as input. In practice, compute FID-T and FID-O, representing the FID score based on thresholding and original protein images, respec- tively. The results are shown in Table 1. The results show that CELL-Diff generated images exhibit better MSFR than CELL-E2. In particular, the MSFR for the original HPA and OpenCell data are 640 nm and 463 nm, respectively. The results from CELL-Diff are approaching the resolv- ability of the original training data, allowing us discern finer details in protein distribution, such as various cytoplas- mic organelles. Regarding the prediction accuracy metric IoU, CELL-Diff outperform CELL-E2 when using only the nucleus image as the conditional cell image, which can be largely improved by incorporating additional cell mor- phology images, such as those of the ER and microtubules.
Predicted imsgee
1oO
0,86
0.98
0.90
Figure 6. Protein localization signal screening. Test sequences are tagged to the C-terminus of the protein BLVRA. The ratio "loO" represents the median protein intensity inside the nucleus relative to that outside the nucleus.
Regarding the learning -baed metric FID, CELL-Diff sig- nificantly outperforms CELL-E2, further demonstrating the superiority of the proposed method. Visual results are illus- trated in Figure 4. From the figure, we find that CELL-Difi accurately predicts protein images from unseen protein se- quences. Compared with CELL-E2, CELL-Diff generates more resolvable images, enabling the extraction of more de- tailed information from the generated images. More results are provided in Appendix B.
6. Discussions
6.1. Ablation on cross attention module
We employ the cross-attention mechanism to more effec- tively integrate information from sequences to images. To evaluate its efficiency, we conduct an ablation analysis of

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
distributed both inside and outside the nucleus. CELL-Diff is then employed to predict the images of the modified pro- tein. The resulting predicted images are analyzed to identify potential localization signals. As illustrated in Figure 6, we compute the median fluorescence intensity inside the nucleus relative to that outside the nucleus, referred to as the loO ratio. For the original BLVRA protein, the loO ratio 1.12. If the IoO ratio of the modified protein exceeds 1.12, the test sequence is likely to function an NLS, conversely, if the ratio is lower, the sequence is more likely to act as an NES. In Figure 6, we tested three NLSs and three NESs known from the literature. CELL-Diff successfully recog- nized these signals, proving its capability as a computational tool for screening potential protein localization signals.
Table 2. Ablation analysis of cross attention module HPA dataset. The nucleus image is used as the cell morphology image.
Metric
w/o CA
w CA
MSFR (nm) 4 IoU FID-T L
687
641
0.434
0.484
55.8
60.1
FID-O 4
41.3
51.1
this module on the HPA dataset, see Table 2. The results show that incorporating this module significantly improves the accuracy metric IoU, demonstrating the effectiveness of the cross-attention mechanism.
Localization signal generation. Using image-to-sequence generation, CELL-Diff can be applied generate novel protein localization signals, such as NL. and NES. Given a cell morphology image and corresponding protein image, CELL-Diff generates the protein sequences that should be located at the position indicated by the protein image. Con- ditioned on an image of either a mucleus-localizes protein or a nucleus-excluded protein (Figure 9), we generated 200 potential NLS and NES sequences, see Table 3 and Table 4, respectively. For NLS generation, the results indicate that the generated sequences exhibit key features of canonical NLS motifs, particularly clusters of basic residues (e.g.. K- K/R-X: X-K/R). The sequences are enriched with lysine (K) and arginine (R) residues, characteristic of functional NLS motifs. Similarly, for the NES generation, the predicted se- quences contain a high proportion of hydrophobic residues, including L, I, F, M. and V, which are known to be criti- cal for NES function. The results suggest that CELL-Diff can effectively generate peptide sequences with potential localization signals, offering a promising tool for studying intracellular protein targeting.
6.2. Potential applications
In this section, we present three potential applications of the proposed CELL-Diff method for biological discovery. Given that validation relies on biological knowledge and the dataset size is limited, we retrain all models using all the protein sequences from the HPA and OpenCell datasets.
Virtual staining. Typical fluorescence microscopes can only fit no more than four color channels in the visible spectrum. Because of this physical limitation, both HPA and OpenCell acquire the images of only one protein of interest per sample, with the other color channels occupied by morphological reference images. Consequently, it is challenging to identify the intracellular spatial relationships among multiple proteins of interest because their images are from different cells. With CELL-Diff, we solve this problem by generating images of these proteins conditioned on the same morphology reference images. These virtual staining images allow the subcellular distributions of arbitrary number of proteins to be directly compared and po- tential molecular interactions identified from colocalization, while entirely circumventing the color channel limitation of fluorescence microscopy experiments. We demonstrate using CELL-Diff to identify molecular interaction by virtu- ally staining two microtubule components (TUBAIA and TUBB4A) and two other proteins, CIF18A and EML4, from the same morphology image. The overlaid image clearly shows the association of KIFISA and EML4 with micro- tubules in the cell, consistent with their known biological function of microtubule binding, see Figure 5.
7. Conclusion
This paper proposes CELL-Diff, unified diffusion model that facilitates the transformation between protein sequences and microscopy images. Given cell morphology images as conditional inputs, CELL-Diff generates protein images from protein sequences. Conversely, it can generate protein sequences based on microscopy images. The objective func- tion of CELL-Diff is constructed by integrating continuous and discrete diffusion models. Experimental results on the HPA and OpenCell datasets demonstrate that CELL-Diff produces more accurate results with higher resolvability than previous methods. Potential applications, including vir- tual screening of protein localization signals, virtual staining, and protein localization signal generation, make CELL- Diff valuable tool for investigating subcellular protein localiza- tion and interactions.
Virtual sereening of protein localization signal. CELL- Diff' can be applied for the virtual screening of protein local- ization signals, such as Nuclear Localization Signals (NLS) and Nuclear Export Signals (NES). The NI. is short amino acid sequence that directs the import of proteins into the nucleus, while the NES facilitates their export from the nucleus. In this approach, the test peptide sequence is tagged to the C-terminus of the protein BLVRA, which is uniformly
8

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Impact Statement
ence Learning Representations, 2023. URL https: / //oppeeevee..eeeeeeee forum?id=33 JR9QxFF..
This paper introduces& a generative model that bridges pro- tein sequences with their corresponding microscopy images, offering novel tool for advancing biological research and discovery. The proposed approach has the potential to hance our understanding of protein sequences and their subcellular localizations, paving the way for breakthroughs in molecular and cellular biology.
Cho, N. H., Cheveralls, K. C., Brunner, A.-D., Kim, K., Michaelis, A. C., Raghavan, P., Kobayashi, H., Savy, L., Li, Y., Canaj. H., Kim, Y. S., Stew- art, E. M., Gnann, C., McCarthy, E., Cabrera, J. P., Brunetti, R. M., Chhun, B. B., Dingle, G., Hein, M. Y., Huang, B., Mehta, S. B., Weissman, J. S., Gómez- Sjöberg, R., Itzhak, D. N., Royer, L. A., Mann, M., and Leonetti, M. D. Opencell: Endogenous tagging for the cartography of human cellular organization. Sci- ence, r75 2022. doi: 10.1 1.1112//cience. abi6983, URL htwp:: /www.. scie eecienee.....oo orgg /doi/ abs /10
Acknowledgements
We thank Chan Zuckerberg Biohub San Francisco Scientific Compute Platform for their support. We thank Guoxun Zhang for valuable discussions. This work is supported by the Chan Zuckerberg Biohub San Francisco Investigator gift to B.H..
abi6983.
abi 6983.
Dauparas, J., Anishchenko, I., Bennett, N., Bai, H., Ragotte, J., Milles, F., Wicky, B. I., Courbet, A., de Haas, J.. Bethel, N., al. Robust deep learning- pro- tein sequence design using proteinmpnn. Science, 378 (6615):49-56, 2022.
References
Almagro Armenteros, J. Spnderby, C. K., Sønderby, S. K., Niclsen, H., and Winther, O. Deeploc: prediction of pro- tein subcellular localization using deep learning. Bioin- formatics, 33(21):3387-3395 2017.
Digre, A. and Lindskog, C. The human protein at- las-spatial localization of the human proteome in health and disease. Protein Science, 30(1):218-25) 233, 2021.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 14:17781-17999 2021.
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. Cogview: Mastering text-to-image generation via transformers. Ad- vances in neural information processing systems, 34: 19822-19835, 2021.
Back, M. DiMaio, F., Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G. R., Wang. J., Cong, Q., Kinch, L. N., Schaeffer, R. D., et al. Accurate prediction of pro- tein structures and interactions using a three-track neural network. Science, 373(6557::771-S77 2021.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883 2021.
Bao, F., Nie, S., Xue, K., Li, C., Pu, S., Wang, Y., Yue, G., Cao, Y.. Su, H., and Zhu, J. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference on Machine Learning, pp. 1692- 1717. PMLR, 2023.
Evans, R., O' O'ill,, M., Pritzel, A., Antropova, N., Senior, A., Green, T., Židek, A., Bates, R., Blackwell, S., Yim, J., al. Protein complex prediction with diphafoldmmltinne biorxiv, pp. 2021-10, 2021.
Chang, H., Zhang, H., Jiang, I.,, Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 11315-11325, 2022.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to local nash equilibrium. NIPS, 30, 2017.
Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., and Chan, W. Wavegrad: Estimating gradients for waveform generation. In International Conference on Leurning Representations, 2021. URL https: opppreviee.. net/forum? forum? MiN.dd
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Ho, J., Chan, W., Saharia, C., Whang. J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Flect, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22 2022.
Chen, T., ZHANG, R., and Hinton, G. Analog bits: Gen- erating discrete data using diffusion models with self- conditioning. In The Eleventh International Confer-

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 0-11244412465 2021.
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, I.., Xiong, C., Sun, Z. Z., Socher, R., et al. Large language models gener- ate functional protein sequences across diverse families. Nature Biotechnology, 41 41(8):1099-1106 2023.
Hoogeboom, E.,, Gritsenko, A. A., Bastings, J., Poole, B., van den Berg, R., and Salimans, T. Autoregressive dif- fusion models. In International Conference on Learning Representations, 2022. URL https:: htt///ppennt openreview..e net/ forummidd-m@T3
Mokady, R., Hertz, A., and Bermano, A. H. Clip- cap: Clip prefix for image captioning. arXiv preprint arXiv:2111.099 2021.
Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Megrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and edit- ing with text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804.
Hu, M., Zheng, C., Yang, Z., Cham, T.-J., Zheng, H., Wang, C., Tao, D., and Suganthan, N. Unified discrete diffu- sion for simultaneous vision-language generation. In The Eleventh International Conference Learning Represen- lations, 2023. URL https:/ http:::///peereeee //pp///peeee.eeeteeeeeee
PMLR, 2022.
Palma, C.-A., Cecchini, M., and Samori, P. Predicting self-assembly: from empirism to determinism. Chemical Society Reviews, (1(10))3713-377
Isigkeit, L.. Hörmann, T., Schallmayer, E., Scholz, K., Lillich, F. F., Ehrler, J. H., Hufnagel, B., Büchner, J., Marschner, J. A., Pabel, J., et al. Automated design of multi-target ligands by generative deep learning. Nature Communications, 15(1)79946 2024.
2012.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):833558, 2021.
Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudi- nov, M. Grad-tts: A diffusion probabilistic model for text-to-speech. International Conference on Machine Learning, pp. 8599-8608, PMLR, 2021.
Kenton, J. D. M.-W. C. and Toutanova, L. K. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Khwaja, E., Song, Y., Agarunov, A., and Huang, B. Celle-2: Translating proteins to pictures and back with a bidirec- tional text-to-image transformer. Advances Neural Information Processing Systems, 36, 2024a.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831, Pmir, 2021.
Khwaja, E., Song. Y. S., and Huang. B. Cell-e: A text- to-image transformer for protein image prediction. In International Conference Research in Computational Molecular Biology, pp. 185-200. Springer, 2024b.
Rombach, R., Blattmann, A., Lorenz, D.,, Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-asiisedd intervention-MICCAl 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 231-241. Springer, 2015.
Kingma, D. P. and Ba, Adam: A method for stochastic optimization. arXiv:1412.69980 2014.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S., et al. Language models of pro- tein sequences at the scale evolution enable accurate structure prediction. bioRxiv, 2022.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXtv:2010.02502 2020.
10

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
UniProt Consortium, T. Uniprot: the universal protein knowledgebase. Nucleic acids research, 46(5):2699- 2699, 2018.
Zhou, C., Yu, L.., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXiv:2408. 10039, 2024.

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
A. Implementation of Discrete Diffusion Model
The training and sampling process of the discrete diffusion model OA-ARDM (Hoogeboom et al., 2022) can be facilitated through a masking operation. Denote C as the categorical distribution, the training and sampling algorithms are shown in Algorithm and Algorithm respectively. For each training iteration, we first sample a time step from U(I,..... D). and random ordering from (())) Subsequently, apply the network fa, which takes m @ S as input, and predicts the masked values (1 m) ④ S.
generate mask based the index such that o < t. We then
Algorithm Training OA-ARDM
Algorithm 2 Sampling from OA-ARDM
Require: Network fg, datapoint S.
Require: Network fg.
Ensure: LOA-ARDM-
Ensure: Sample S.
1: Sample & U(1,......... D).
I: Initialize $ = 0, sample ~ ((D)).
2: Sample o L((D))
2: for t = 0, 1, .,,,,... D do
Compute m t (o t).
m
t) and n (or = t).
4: Compute 1 t -(1 m) @ ( S)).
4: S' C-C(S fo S)).
5: LONARDM D-1+1 D-r+1Summ sum(l).
5: S (1 O O
6: end for
B. Protein image generation
We present protein image generation results. The results the HPA and OpenCell datasets shown in Figure 7 and Figure 8, respectively. From these results, we observe that CELL-Diff is capable of generating realistic protein images with high accuracy, enabling the discernment of fine details. Compared to CELL-E2, CELL-Diff produces images with higher resolvability, which provides better clarity of detailed localization structures.
C. Localization signal generation
We CELL-Diff to generate protein localization signals. Specifically, the images in Figure 9 as the conditional input for generating NLS and NES signals. Using the CELL-Diff model, we generate short amino acid sequences. A total of 100 potential sequences are generated for each signal type. Generated NLS and NES sequences are summarized in Table 3 and Table 4, respectively.
12

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Protein
Nodl
Nud
Nacl FR
Nacl MT
Nud. FR MT
GYSI
TARDBP
GNL2
Figure 7. Visual results of protein image generation HPA dataset.
13

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Preesie
Rool
CHLL-DE
Protets
Real
CELI-IIT
GNL2
NSA2
PKM
RABII
ACAPS
Figure 8, Visual results of protein image generation on OpenCell dataset.
14

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Table 3. Generated NLS sequences.
Index
Sequence
Index
Sequence
1
PERKK
51
PKRKAEGDAKKDKAK
2
PKGKK
52
SKRKAEGDAKKIDKAKK
3
PKRSK
53
PKRKAAGIDAKGIDTKKK
4
PKRKK
PKRKAEGIDAKGIDKKKK
5
PEGKK
55
PERKAPGDAKGIKAKK
6
PKRKAE
56
PKRKAEGDAAKIDKKKKE
7
PKGAKK
57
AKRKAKAAAKKGDTAKKK
8
PKRAKK
58
PERKAEEIDAKKIDIAKKK
9
PKRAAK
59
SKRKAKKNDAKGDKAKERD
10
AKRKKK
60
PKEKAEGGAAKGKAAKKKO
11
PKRKAK
61
PKEKKENDAKKIDKAAKKKI
12
PKRKAKK
62
PERCAKADAKGGEVAEER)
13
PKRKKKK
63
PKRESPNIDAKGPTTIEKR
14
PKGKAKK
64
PKRKKKADAKKAKAKKKKO
15
PKRQAKK
65
PERKAKNDAKGPPAAKKK)
16
PKGKAKA
66
SKRKTPASAAKPPKAKKKDOO
17
PKRKAKAA
67
SKEKTTVKDAKKITTAKKKOOG
18
PKGKAKKA
68
SKRKTVASSAKGPAAKKKO
19
PKRKAKAD
69
SKEK.KKADAKGETAKKKKOO
20
PKRKAKGK
70
SKRKKEADAKGGDAKKKIGG
21
PKRKAKGD
71
SKRKTPAPAKGGKKAKKKOO
22
PKRKAKGA
72
PERKTKAPAAKIDAAKKKDDE
23
PKRKAKEDA
73
PERKTPAPAKKPPTAKKKDOC
24
PKRKAEKQA
74
SERKTTASAKKGPAAKKEDOPP
25
PKRKAKADK
75
PERKTTKAPAKGEETTKKKKOGP
26
PKRKAKRDA
76
KERKPEADEKKGDKAGKKOOHH
27
PKRKAEADAK
77
PERCAEADAKGDKAAKKOVVPR
28
PKRKAEASAK
78
SERKAPAAAAKGDRAXKKKOWWW
29
PKRKKEAPAK
79
PERKAPADAAKGDAAKKKORRR
30
PKRKKPADAK
80
PERKABADAKKODXAKKKOVVOD
31
PKRKKEADAKG
81
PARKAEEDAXGGDKAKKKOEPPP
32
PKRKPEAPAKG
82
PERKGAPDDOKGDXAXXKEOVVOO
33
PKRKKKAPAKG
83
PERKAEEDDAKGIKAAKKKEEPOO
34
PKRKSKASAKG
84
SKREAPPADAKGDXAKKKEDVVOPO
35
PKRKSEASAAGE
85
PERKAEADEKGGDAAKKKEDROO
36
PKGKTEADAAGG
86
SKEKAPADDAKGDKAAKKKKDMPOO
37
PKRKAPAPAKGD
87
AKCKABEADAKKDDXAAKKKKEPOOO
38
PKRKKEAPAKGE
88
PARKAECODAKGDYAAKKEEPPPR
39
PKRKAPAPAAGD
89
PERKAPADDKKGDXAEKKKGGPPOO
40
PKRKSKASAEGG
90
AKCKAAEBDAAKGDDAAKKKKDDPPOOO
41
PKRKAEEDAKGAK
91
AKCKKAERDDAKGDKAAKVVKEDDOOO
42
PKRKAEAAAKGAG
92
PERKATEDDAKCDAAKKKEEEPOO
43
PKRKAEADAKGET
93
AKRKAARDDAGGDDAKKKSDEPOOAA
44
PKRKAEADAAGEK
94
AEPKAAPRDAKCDDAAKKKEEPPOOA
45
PKRKAEADAPGIGG
95
AKRKAHEADDAKCOYAKKEEEPPPOOOOOOC
46
PKRKAEEDAKKIDKA
96
PERKPPLADDKKKOAAKKKKEEOOOOO
47
PKSKAEEDDRKGKA
97
AEPRAEERDAKKDKAKEKKEPPOOOO
48
PKRKAEEDAKGPKA
98
AERKAEERDAKGEPAKKERDPPOOOA
49
PRGKAEADAKGIDKA
99
PERKABBRDAKCODAKKKKEDPPOOOO
50
PKRKAEEDAKGDKAK
100
BKRKAETDAKKGDTANEEDDPPOOAAO
15

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Table Generated NES sequences.
Index
Sequence
Index
Sequence
AMPKALREMGG
51
AMFAARRLLGKAAQLLY
QDIIARIEOGK
52
VSFSARLLLGFMAQLDDI
GMIRARRENGE
53
AMFAAALLLLGEFAQRDY
QMIVYLEMMK
54
NWKLARLLL.KMAIQLDD
AMIRVRIEMGE
55
AGFIARLLLGGFFALLDY
6
ADIIALREMGKA
56
AMFIARLLLHGFFAAQLLDY
7
OKARAPEGELGA
57
AMIIAARRELDEFFQLLIIT
8
ADIIAAEEVGKA
58
AMIIARRRHLIKKFAOYYIIT
9
ADIIAAEEVIKA
59
AMITLARRSL.GEFAQLLIIT
10
ADIIAPRLHGKA
60
AMILARRSSLGKFAOYDDD
ADIIAPEEMGK
61
ANFIARRRS..GKFAQL.DD
12
GDIIAPEEYGKA
62
AMIIARREELOFFFFOLDD
13
ADIIAPRKVIKA
63
AMISARRENGKFAAOKDIT
14
ADIIARDEMGKA
64
AMIIARREHCGFFFOKDRRG
15
ADIIAPEEVGKA
65
AMIIARRIEHGFFAQLLDEO
16
GDFLAARKKGGIT
66
AMIIARKKLNGKFAVLDON
17
GDFLAPEKVGEEF
67
AMISAREDENYVEAALLOOP
18
GDQLAPLKYGKFF
68
AMISARREEHNKFALDDIEE
19
CDIEAPEKLPREF
69
AMISARREEOKFALLIDTEC
20
GDDLAPLRVGKEF
70
AMISARR
21
CDIRAAEQVGKFE
71
AMIIARRELTGFGADDITE
22
SDILAPRKHGEFT
72
AMISARREYDDAFAQLDOOO
23
PDKLAPLEVGGFER
73
AMISALLEHGGEANCCLKER
24
KDKKAPROVGGHFIL
74
AMITARKEHHTVPAHIDOFFFE
25
GDFIAPQLFGGETR
75
ADIALIKHIFFELDLFOR
26
SDFLAPELFGGFEE
76
SMIJARRENGGFFQIIDDEER
27
PDFKAPOLVKGFFA
77
ALIALLENOOGFEVIDOKKRR
28
PDFLVPLEFGEEFA
78
ALIIARKEHIFFAIIILLERR
29
PDFLAPLLVPLETA
79
ADISAVIEHIKFFAVIIDLFLL
30
PGFIAEQLKGLFAQE
80
ADIALLEENPVEAQSGLERR
31
PDFEAPOLKLLFAAL
81
GLISALIEHOOFFALDOFRR
32
CDFLYDLLLYLFANI
82
AQUALLEEDDGEATLLDOER
PGKEAPQLLLPKFAAI
83
GDIAAUJIKROOFEAVIGGNNNN
34
PGFEANQLFGEFFAY
84
SDFIALISFOGFAVLLLLOOY
35
PDFEAPOLOOLGFATI
85
SDIRALIRNDGEAALGLLKOPP
36
PDFEAELLKLLFAIE
86
SDIAALIRKGGFEAIGQOFRC
37
PGPLADLLHILLFOA
87
ADIRALLIENGKFAILCOOOOPPP
38
PDFEAPOLFGKMAAK
88
EDKKAALKEFGGGFEOLLLLFPEE
39
PDFEANQLKLLGAEV
89
SDIAALIEFPOGEALLLLPRYY
VGBSARRQLHGKKAQQI
90
GDDRAALIKFFOKEEVVCCCXOQZ::
SGESARQLKKKMAALII
91
SDISALIRNHGGVADLGOOORC
42
AGFSAIEQLLGKKAQQEI
92
GDISALISNKKIFAEIGOOOOCC
43
AGFAADOLLLGKMOLII
93
CKNLALLSFRVGFAWCCIIOOSDD
44
MGFSAADOLLLGMMAAU
94
CEKRALDRRNGFMAALLLKKONW
45
SGFSARQLLGGKTAEED
95
SDRRALKKITORTTAVLLGLGGGUUUT
46
AGFSADDOLLLKKFAOII
96
SKKRALLDEEVVGFAV..GLLRIVY
47
AGFLADOLLGKFAQQII
97
GMKKRALDKELLFFALLOTTORRT
48
AGESADOLLLGKMAQI
98
ANKKAALDSFOOEFNSSLLLEEER
49
PGFSAEELLLGKKAQVI
99
AKHIALDDNNKGFAALLLFOVEDDD
50
MAFFARLLL
100
AKIRALDXFLLGRVVDDLFOEEDD
16

---

Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models
Conditional image for generating NLS
Conditional image for generating NES
Figure 9, Conditional images for protein localization signal generation.
17


Reviews:
[{"review_id": "KFDt1XbSIW", "reviewer": "Reviewer_HAqj", "summary": "The paper, Bridging Protein Sequences and Microscopy Images with Unified Diffusion Models, presents CELL-Diff, a novel generative model that enables bidirectional transformations between protein sequences and fluorescence microscopy images. By leveraging a transformer-based U-Net architecture and integrating both continuous and discrete diffusion processes, CELL-Diff outperforms prior methods in generating high-resolution protein images. The model is trained on the Human Protein Atlas (HPA) dataset and fine-tuned on OpenCell, demonstrating its ability to reconstruct subcellular protein localization with improved fidelity. The proposed approach has significant implications for biomedical research, particularly in protein function prediction and cellular imaging.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1:** The potential for dataset biases or domain shifts between HPA and OpenCell is not explicitly explored.\n\n**Response:**  \nThe main difference between HPA and OpenCell is that HPA is larger, but OpenCell has more consistent labeling and higher image quality. The effect of domain shift has already been explored in the previous CELL-E2 paper by testing different pretraining and finetuning arrangements using the same two datasets. Therefore, we did not repeat this evaluation for CELL-Diff.\n\n---\n\n**Comment 2:** The paper covers relevant prior work but does not discuss alternative generative approaches, such as GAN-based models, which have also been applied to biological image synthesis. Including a comparison with these methods could provide a broader context for CELL-Diff’s contributions.\n\n**Response:**  \nThank you for the comment. The baseline model used in our comparison, CELL-E2, is based on a VQGAN architecture, making it a GAN-related model. We agree that comparing CELL-Diff with alternative generative approaches would provide valuable context. However, these existing generative models are not specifically designed for this task and require significant adaptation. Implementing and optimizing these models involves careful tuning of hyperparameters and architectural modifications to achieve their best performance.\n\nWe are currently exploring other generative approaches, including the consistency model[1], and have obtained some preliminary results, see the following Table. We did not observe a consistent trend of difference across metrics. We hope that the growing interest in biological image synthesis will lead to the development of more specialized models, providing a richer set of baselines for comprehensive evaluation.\n\n**Table 1: Comparison with consistency model.**\n\n| Method    | Cell image     | MSFR (nm) ↓ | IoU ↑   | FID-T ↓  | FID-O ↓  |\n|-----------|----------------|-------------|---------|----------|----------|\n| CM        | Nucl           | 650         | 0.430   | 51.3     | 38.6     |\n| CELL-Diff | Nucl           | 641         | 0.484   | 60.1     | 51.1     |\n| CM        | Nucl,ER        | 644         | 0.609   | 44.7     | 30.9     |\n| CELL-Diff | Nucl,ER        | 642         | 0.580   | 55.9     | 60.0     |\n| CM        | Nucl,MT        | 645         | 0.606   | 45.4     | 31.5     |\n| CELL-Diff | Nucl,MT        | 644         | 0.616   | 51.0     | 47.6     |\n| CM        | Nucl,ER,MT     | 645         | 0.619   | 44.6     | 31.8     |\n| CELL-Diff | Nucl,ER,MT     | 644         | 0.635   | 50.4     | 45.6     |\n\n---\n\n**Comment 3:** Limited discussion on potential biases in dataset selection and domain adaptation.\n\n**Response:**  \nSee Comment 1.\n\n---\n\n**Comment 4:** How does CELL-Diff handle proteins with highly disordered or ambiguous subcellular localization?\n\n**Response:**  \nCELL-Diff handles proteins with highly disordered or ambiguous subcellular localization by leveraging the conditional cellular morphology image and the stochastic nature of the diffusion model.\n\nThe cellular morphology image provides structural context, capturing the cellular environment, such as the nucleus, ER, and microtubules, as well as the cell type and cell state implicitly contained in the morphological information. This conditioning image acts as a spatial prior, guiding the model to place proteins in realistic and biologically plausible locations.\n\nAdditionally, the diffusion process naturally captures stochastic variations, enabling CELL-Diff to model the inherent uncertainty of disordered or ambiguously localized proteins. As a result, the model can generate multiple plausible localizations for the same sequence when sampled multiple times, reflecting the biological variability of such proteins.\n\n---\n\n**Comment 5:** Were any domain adaptation techniques used to mitigate differences between HPA and OpenCell datasets?\n\n**Response:**  \nWe currently address the domain shift between the HPA and OpenCell datasets by using the pre-training and fine-tuning approach. However, we recognize that more advanced domain adaptation techniques could further improve integration between the two datasets. Moving forward, we plan to explore additional domain adaptation methods to mitigate the differences and better align the datasets for improved model performance.\n\n---\n\n**Comment 6:** Could alternative generative architectures, such as GANs, be competitive with the proposed approach?\n\n**Response:**  \nSee Comment 2.\n\n[1] Song, Yang, et al. \"Consistency models.\" (2023).", "claims_and_evidence": "The paper claims that CELL-Diff facilitates accurate bidirectional transformation between protein sequences and their corresponding microscopy images, improving upon previous methods like CELL-E and CELL-E2. Experimental results support this claim, demonstrating that CELL-Diff produces higher-resolution images with better spatial fidelity. The authors provide quantitative metrics such as Maximum Spatial Frequency (MSF) resolvability, Intersection over Union (IoU), and Frechet Inception Distance (FID), all of which indicate that CELL-Diff outperforms baselines.", "methods_and_evaluation_criteria": "The methodology is well-defined, employing diffusion models in both continuous (for images) and discrete (for sequences) state spaces. The evaluation framework includes comparisons with prior models (CELL-E2) using established quantitative metrics.", "theoretical_claims": "The paper does not introduce new theoretical developments but builds upon existing diffusion models. The combination of continuous and discrete diffusion processes is well-motivated.", "experimental_designs_or_analyses": "The experiments are well-structured, with evaluations on multiple datasets and comparisons against prior work. The use of multiple quantitative metrics strengthens the findings. However, the potential for dataset biases or domain shifts between HPA and OpenCell is not explicitly explored.", "relation_to_broader_scientific_literature": "This work aligns with research on multimodal generative modeling, fluorescence microscopy, and protein function prediction. It extends prior work on text-to-image generation by applying diffusion models to biological data. The references to related work in protein structure prediction (e.g., AlphaFold) and generative models are appropriate.", "essential_references_not_discussed": "The paper covers relevant prior work but does not discuss alternative generative approaches, such as GAN-based models, which have also been applied to biological image synthesis. Including a comparison with these methods could provide a broader context for CELL-Diff’s contributions.", "other_strengths_and_weaknesses": "Strengths:\n\n- Introduces an innovative bidirectional generative model for protein sequences and microscopy images.\n- Demonstrates significant improvements over previous methods in image quality and sequence prediction.\n- Uses well-established benchmarks and evaluation metrics.\n\nWeaknesses:\n\n- Limited discussion on potential biases in dataset selection and domain adaptation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How does CELL-Diff handle proteins with highly disordered or ambiguous subcellular localization?\n2. Were any domain adaptation techniques used to mitigate differences between HPA and OpenCell datasets?\n3. Could alternative generative architectures, such as GANs, be competitive with the proposed approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "48UQDJ3VRw", "reviewer": "Reviewer_igYh", "summary": "This paper introduces CELL-Diff, a unified diffusion model that enables bidirectional generation between protein sequences and fluorescence microscopy images. It combines continuous diffusion for image synthesis and discrete diffusion for sequence prediction, integrating transformer-based cross-attention to fuse multimodal representations. The model is trained on Human Protein Atlas and OpenCell datasets, demonstrating improvements in protein localization prediction and potential applications in virtual staining and protein sequence design.\n\n## Update after rebuttal:\nI do not see sufficient justification to change my original rating. Please see my reply to rebuttal below.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to character limitation, we provide a general response to the comments. \n\nWe understand the reviewer’s concerns about the concept of sequence-to-image mapping, especially with limited data from HPA and OpenCell. While a universal sequence-to-image model requires massive data, CELL-Diff, as a smaller model, can still make practically relevant predictions for general cell biology. Specifically:\n### 1. Feasibility of sequence-to-image mapping.\nWhile the subcellular localization of a protein is influenced by cell type, state, and post-translational modifications, many proteins have a defined localization, often determined by sequence motifs (e.g., NLS for nuclear localization, signal peptide for ER translocation, and CAAX motif for lipid modification and subsequent plasma membrane localization). This premise underpins localization prediction models like DeepLoc[1] and MultiLoc[2], and CELL-Diff extends this concept by using images rather than discrete annotations, allowing quantitative description of multi-localization. For example, Figure 6 shows that CELL-Diff can characterize the non-binary effectiveness of different NLSs and NESs using the nuclear-to-cytoplasmic signal ratio from segmented images. The importance of understanding this quantitative effectiveness is illustrated by the need to add three separate NLSs to Cas9 for sufficient nuclear localization in CRISPR/Cas9 genome editing. Moreover, the input condition image implicitly conveys information about cell type and state. For instance, the CELL-E paper demonstrated the ability to predict the spherical shape of a cytoplasmic protein in a mitotic cell using a DNA-stain condition image.\n### 2. The dimension of the image space. \nPractical cell biology research often focuses on high-level image features like morphology and colocalization, rather than pixel-by-pixel data. In this sense, the meaningful image state space is likely smaller than the atomic coordinate structural space. As an illustration of this point, previous work (cytoSelf[3]) demonstrated the correlation between dimension-reduced image representations and stoichiometric protein-protein interactions and identified a new component of a protein complex solely by image similarity.\n### 3. Dataset size and the validation of sequence-to-image prediction.\nAlthough CELL-Diff's training data is smaller than that for structure prediction models, the HPA dataset covers over 60% of human proteins. Combined with ESM2 embeddings, CELL-Diff makes meaningful predictions. Figure 6 shows accurate localization predictions for NLS- and NES-fused biliverdin reductases, including the KLKIKRPVK sequence from E. coli protein Tus, acting as a mammalian NLS. We also tested GFP (from jellyfish and non-homologous to human proteins) fused to various yeast NLSs and demonstrated the additive effect of multiple NLSs. These results showcase CELL-Diff's ability to extract biological knowledge about nuclear importin recognition from limited data (Reviewer sVh2, 1. The Dataset Size and Model Generalizability.).\n\nWe must state that CELL-Diff is not a \"true sequence-to-image generative model\" capable of translating any random sequence into pixel-perfect cellular images, as structure prediction models do. Instead, it is built as a virtual experiment tool for typical sequence variabilities in cell biology research (mostly mutations of endogenous proteins). We will revise the manuscript to clarify its application and limitations.\n### 4. Comparison with CELL-E2.\nThe intention for us to develop CELL-Diff is indeed to improve the image clarity over CELL-E2 which is the major limitation for CELL-E2 and prevents it from resolving finer subcellular structures other than the nucleus and the nucleolus. We used the standard image similarity metrics to compare the generated images in the test set against the ground truth (Table 1). The improvement in these metrics should correlated with biological accuracy. \n### 5. Sequence generation.\nFor NLS sequence generation, our validation through amino acid composition analysis and clustering was exactly based on our expert knowledge of NLS. In addition, we have now further validated the generated sequences using the annotation-based localization prediction model DeepLoc 2.1[4], which indicates that 78% of the generated sequences are recognized as legitimate NLSs (Reviewer nmZP, Comment 1). Furthermore, our recent wet lab experiments testing 20 generated sequences confirmed that 10 of them exhibited NLS activity, providing direct experimental validation.\n\n[1] DeepLoc: prediction of protein subcellular localization using deep learning.\n\n[2] MultiLoc: prediction of protein subcellular localization using N-terminal targeting sequences, sequence motifs and amino acid composition.\n\n[3] Self-supervised deep learning encodes high-resolution features of protein subcellular localization.\n\n[4] DeepLoc 2.1: multi-label membrane protein type prediction using protein language models.", "claims_and_evidence": "- The paper claims that CELL-Diff, a unified diffusion model, can bidirectionally transform between protein sequences and their corresponding fluorescence microscopy images by integrating continuous diffusion (for images) and discrete diffusion (for sequences) within a single framework.\n\n- Quantitative metrics show better image generation than prior models. The concept of bidirectional transformation is novel in this context. Case studies suggest potential biological applications.\n\n- Unseen proteins were not rigorously tested. CELL-Diff is most likely memorizing dataset-specific patterns. The paper oversells its claims. While diffusion models are powerful, they cannot overcome the fundamental limitation of tiny training datasets in an enormous search space. This is likely more of an interpolation method rather than a true sequence-to-image generative model.", "methods_and_evaluation_criteria": "- CELL-Diff conditions image generation on reference cell morphology images (e.g., nucleus, ER, microtubules). Uses latent diffusion models to process microscopy images efficiently. The training objective combines: Noise prediction loss for continuous diffusion (image generation). Masked value prediction loss for discrete diffusion (sequence prediction). \n\n- Method uses several metrics that measures image clarity and fine structural details (MSF), assesses how well the generated protein images align with ground truth (IoU), evaluates similarity between generated and real images (Frechet Inception Distance)", "theoretical_claims": "- The paper does not have any strong theoretical claims. Section 3.1 presents the standard formulation of continuous diffusion models. Section 3.2 discusses OA-ARDM for discrete data (Hoogeboom et al. 2022). Uses random ordering to make the model agnostic to token positions. \n- Section 4 of the paper introduces the CELL-Diff model and describes how it integrates continuous and discrete diffusion models for bidirectional transformation between protein sequences and microscopy images. The authors claim to integrate continuous diffusion and discrete diffusion within a single framework. Continuous diffusion follows standard diffusion formulations. Discrete diffusion follows OA-ARDM (Hoogeboom et al., 2022). Not a novel theoretical contribution. The loss formulations are also standard for diffusion models and masked language modeling.", "experimental_designs_or_analyses": "- The authors compare CELL-Diff to CELL-E2, a previous protein-to-image generation model. Cell-Diff improves image clarity over CELL-E2. Although visual inspection shows more detailed subcellular structures, none of the metrics used for comparison would necessarily correlate with biological accuracy. \n- The image generation results seem promising within the dataset, but there is no proof of generalization to novel proteins. There is no held-out test set of completely unseen protein families, making it impossible to evaluate generalization. The sequence prediction claim is not validated by experts or in wet-lab studies. The methodological contribution (unified diffusion) is not rigorously tested. Molecular interaction predictions could be misleading. If two proteins often appear together in the dataset, the model might just learn their cooccurrence rather than discovering true interactions.\n- While ambitious in scope, this paper attempts to tackle an inherently intractable problem without the necessary scale of data, established benchmarks, or biological validation, making its claims more speculative than substantive.", "relation_to_broader_scientific_literature": "- The paper positions itself at the intersection of protein sequence modeling, fluorescence microscopy, and generative AI. Although the scope might be slightly different, there are many methods that use diffusion to predict structure from sequence (AlphaFold3, RoseTTAFold, Baek et al. 2021, RFDiffusion Waston et al., 2023, FrameDiff Wu et al. 2023). RFdiffusion and FrameDiff use diffusion models, but they generate structured, atomic-level representations, whereas CELL-Diff tries to map sequences directly to microscopy images. There are also diffusion-based generative models for de novo protein sequence design.\n\n- While ambitious, structure prediction and protein design have strong theoretical foundations and are being tackled by leading labs with rigorous validation, making them credible scientific pursuits. In contrast, sequence-to-microscopy image generation lacks structural constraints, relies on limited data, and has no established evaluation metrics, making it far more speculative. Moreover, the state space of high-resolution microscopy images is vastly larger than that of 3D protein structures, further highlighting the impracticality of learning a direct mapping from sequence to image.", "essential_references_not_discussed": "These are not essential but literature review would be more complete with some of these diffusion-based structure prediction methods. \n\nAlphaFold3 (Abramson et al., 2024) - AlphaFold that uses diffusion\nRFdiffusion (Watson et al., 2023) – The first diffusion model for protein structure generation.\nFrameDiff (Wu et al., 2023) – Rigid-body diffusion for protein structure prediction.\nProteinSGM (Trippe et al., 2022) – Diffusion for protein sequence design.\nFoldFlow (Anand et al., 2022) – Normalizing flows for protein backbone generation.", "other_strengths_and_weaknesses": "Strengths:\n- The paper introduces an ambitious multimodal diffusion framework that, if validated, could open new directions in bridging protein sequences and cellular imaging\n\nWeaknesses:\n- The Training data is woefully small: The Human Protein Atlas (HPA) dataset contains around 10K proteins with fluorescence microscopy images. Even though prominent structure prediction methods also use hundreds of thousands of protein structures protein folding problem is not random meaning the number of viable structures is vastly smaller than the full combinatorial space. On the other hand microscopy images have no equivalent constraints. The mapping from sequence to fluorescence image is far less structured than sequence-to-structure. The same protein can exhibit multiple localizations depending on cell type, environment, modifications. No universal, physics-driven constraints like those in protein folding.\n- Microscopy Images Are Not Sufficient to Capture Protein Function: Protein function is not solely determined by sequence; it depends on post-translational modifications, cellular environment, and binding interactions. Even if a model memorized all available images, it would not generalize to unseen proteins effectively.\n- Bidirectional Mapping Between Sequence and Image is Ill-Defined: A single protein sequence can adopt multiple conformations and localizations depending on: Cell type, Post-translational modifications, Interaction partners. One-to-one mapping between protein sequences and fluorescence images does not exist.\n- Diffusion Models May Be Overfitting: The authors claim CELL-Diff outperforms prior methods like CELL-E2, but if the model is trained on such a small dataset, it could be: Memorizing protein localizations rather than learning meaningful structure-function relationships. Hallucinating plausible but incorrect images, which may still look visually appealing but lack biological relevance.\n- No Evidence of Generalization to Unseen Proteins: A true sequence-to-image model should be tested on novel sequences never seen in training, but the paper does not provide convincing results for this. Without proper benchmarking on completely held-out protein families, this model might just be fitting noise or dataset-specific patterns.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWcaFNUoeU", "reviewer": "Reviewer_sVh2", "summary": "The paper introduces CELL-Diff, a unified diffusion model designed for bidirectional transformations between protein sequences and their corresponding microscopy images.  Given cell morphology images and a protein sequence, CELL-Diff generates corresponding protein images.  Conversely, it can also output protein sequences from protein images.  CELL-Diff integrates continuous and discrete diffusion models within a unified framework and is implemented using a transformer-based network.  The model is trained on the Human Protein Atlas (HPA) dataset and fine-tuned on the Open-Cell dataset.  Experimental results demonstrate that CELL-Diff outperforms existing methods in generating high-fidelity protein images.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to the limitation on the number of characters, we provide a general response to the reviewer's comment.\n\n### 1. The dataset size and model generalizability.\n\nThe generalizability of the sequence-to-image task depends on the downstream application.\n\nOne application is **virtual staining**. Traditional fluorescence microscopy images typically have four color channels, limiting the ability to visualize the spatial relationships between multiple proteins of interest. With CELL-Diff, we can virtually stain all proteins from the training dataset using the same cell morphology images. As shown in Figure 5, this enables the identification of potential protein-protein interactions. In this context, the generalizability pertains to the conditional cell morphology images. Since both the HPA and OpenCell datasets contain approximately 100 cells for each protein, the model can learn diverse cellular morphologies, achieving strong generalizability with respect to different cell morphologies.\n\nAnother application is **sequence-to-image prediction**, which involves generating images for unseen proteins. In cell biology, rather than predicting images for entirely artificial sequences, the focus is often on generating images for biologically relevant variants, such as protein truncations or point mutations. Consequently, the effective protein sequence space in practical applications is much smaller than in tasks like de novo protein design or unconditional sequence generation.\n\nFor specific biological problems, such as predicting protein phase separation or identifying disease-related mutations, task-specific datasets can be collected. These datasets are often easier to obtain compared to large-scale, general-purpose datasets. By fine-tuning CELL-Diff on task-specific datasets, the model's generalizability can be improved, making it adaptable to new biological contexts.\n\nTo further validate the generalizability of CELL-Diff, we expanded the NLS screening experiment to the INSP yeast NLS dataset[1], which contains 50 valid NLSs from yeast. We generated new proteins by fusing the yeast NLS sequences to the C-terminus of Green Fluorescent Protein (GFP), which is not included in the HPA or OpenCell datasets, and tested CELL-Diff's ability to predict the function of these yeast NLSs.\n\nFor the generated images, we quantified the median fluorescence intensity inside and outside the nucleus. If the median intensity inside the nucleus was higher than outside, we considered the model to have made a correct prediction. The total number of correct predictions is denoted as $N_{hit}$, and we computed the identification rate as $N_{hit}/N$, where $N$ is the total number of NLS-tagged proteins.\n\nTo further evaluate the model's biological reasoning, we tested different fusion patterns by increasing the number of NLS tags. In practice, proteins fused with multiple NLS sequences have a higher efficiency of entering the nucleus. Thus, we evaluated three configurations: GFP + NLS, GFP + NLS + NLS, and GFP + NLS + NLS + NLS, which correspond to the increasing effectiveness of nuclear localization.\n\nWe compared CELL-Diff's identification rate against CELL-E2, and the results are presented below.\n\n| **Test Protein**         | **CELL-Diff** | **CELL-E2** |\n|-------------------------|---------------|-------------|\n| GFP + NLS               | 0.82          | 0.54        |\n| GFP + NLS + NLS         | 0.86          | 0.78        |\n| GFP + NLS + NLS + NLS   | 0.90          | 0.88        |\n\nFrom the table, we observe that both CELL-Diff and CELL-E2 can effectively identify NLS-tagged proteins, with CELL-Diff demonstrating higher identification rates, especially for single NLS fusions. Additionally, the identification rate increases as more NLS tags are added, which aligns with real-world biological behavior. This experiment suggests that CELL-Diff captures some underlying biological logic rather than merely memorizing dataset-specific patterns.\n\n### 2. Technical details and visual aids.\nWe will include more technical details and visual illustrations of our method in the revision.\n\n### 3. Computational cost and scalability.\nThe CELL-Diff model contains approximately 1 billion parameters, while the VAE used for latent diffusion has 42 million parameters. The model was trained on 2 NVIDIA H200 GPUs for approximately 10 days. CELL-Diff is scalable due to its latent diffusion framework, which reduces computational costs by operating in a lower-dimensional latent space. This enables the efficient generation of high-resolution images. Furthermore, CELL-Diff can be scaled by increasing the model size and utilizing larger computational clusters, making it adaptable for more complex biological datasets and tasks.\n\n[1] Guo, Yun, et al. \"Discovering nuclear targeting signal sequence through protein language learning and multivariate analysis.\"", "claims_and_evidence": "Yes\n1.Claim: CELL-Diff facilitates bidirectional generation between protein sequences and images.    \n\n1.Evidence: The paper provides Figure 1 and discusses the model's ability to generate protein images from sequences and vice versa.  The methodology section details how the model is trained to handle both types of generation.  Experimental results in Section 5.3 and Appendix B also visually support this claim.    \n\nThe claim is supported by the presented evidence.\n\n2.Claim: CELL-Diff outperforms existing methods in generating high-fidelity protein images.    \n\n2.Evidence: The paper compares CELL-Diff with CELL-E2 and uses metrics like MSF-resolvability, IoU, and FID to demonstrate superior performance.  Visual comparisons in Figure 4 and Appendix B also highlight the improved image quality.    \n\nThe claim is well-supported by quantitative and qualitative evidence.\n\n3.Claim: CELL-Diff can be applied for virtual screening of protein localization signals, virtual staining, and protein localization signal generation.    \n\n3.Evidence: Section 6.2 details these potential applications and provides supporting figures (Figure 5, 6, and 9) and generated sequence tables.    \n\nThe applications are well-described, and the results seem promising, but further experimental validation would strengthen these claims.", "methods_and_evaluation_criteria": "Methods: The proposed CELL-Diff method combines continuous and discrete diffusion models within a unified framework.  It employs a transformer-based U-Net architecture with cross-attention mechanisms.  The training objective function includes noise prediction loss for the continuous diffusion model and masked value prediction loss for the discrete diffusion model.  A latent diffusion model is used to reduce computational costs. \n\nThe methods are clearly described and seem appropriate for the problem. The combination of continuous and discrete diffusion, along with the transformer-based architecture, is a reasonable approach.\n\nEvaluation Criteria: The paper uses metrics:\n\n1.MSF-resolvability: This metric measures the capability to discern fine structural details in microscopy images.    \n\n2.IoU (Intersection over Union): This metric measures the similarity between two masks, used here to compare predicted and real protein image masks.    \n\n3.FID (Fréchet Inception Distance): This metric evaluates the similarity between the real and generated images regarding their feature distributions.\n\nThese evaluation criteria are appropriate for assessing the quality and accuracy of generated microscopy images. MSF-resolvability is a particularly relevant metric for this task.", "theoretical_claims": "The paper includes theoretical claims related to the diffusion models and the derivation of the training objective. \n\n1.The forward process of the continuous diffusion model is defined in Equation 1, and the subsequent derivation of  q(I_t∣I_0) is standard.    \n\n2.The reverse process is defined in Equation 2, which is also a standard formulation.    \n\n3.The ELBO (Equation 3) and its simplified form (Equation 4) are correctly presented.    \n\n4.The derivation of the loss function for OA-ARDM (Equation 7) appears to be correct.    \n\n5.The conditional ELBO for continuous and discrete diffusion models (Equations 8 and 9) are derived logically from the previous equations.    \n\n6.The combined loss function (Equation 12) is a straightforward combination of the individual losses.", "experimental_designs_or_analyses": "The experiments are designed to evaluate the protein image generation performance of CELL-Diff and to demonstrate its potential applications.    \n\nThe model is trained on the HPA dataset and fine-tuned on the OpenCell dataset.    \n\nThe performance is compared with CELL-E2 using MSF-resolvability, IoU, and FID metrics.    \n\nAblation studies are conducted to evaluate the effectiveness of the cross-attention mechanism.    \n\nPotential applications are demonstrated through virtual staining, virtual screening of protein localization signals, and localization signal generation.", "relation_to_broader_scientific_literature": "The paper builds upon previous work in the field of predicting protein properties using learning-based methods. It cites examples such as predicting protein structure, interaction partners, and subcellular localization.    \n\nIt is related to the development of generative models for designing functional proteins and drug-like molecules.    \n\nThe work focuses on the relationship between protein sequences and their cellular functions, as characterized by microscopy images, particularly fluorescence microscopy.    \n\nIt is specifically related to recent work that proposed CELL-E, a text-to-image transformer that predicts fluorescence protein images from sequence input and cell morphology condition, and its enhancement CELL-E2.", "essential_references_not_discussed": "Some reference of Stable Diffusion", "other_strengths_and_weaknesses": "Strengths:\n1.The paper addresses an important problem: understanding the relationship between protein sequences and their cellular functions.    \n2.The proposed model has the potential to be a valuable tool for investigating subcellular protein localization and interactions, with potential applications in drug discovery and disease research.    \n3.The paper demonstrates the model's potential through several applications, including virtual staining, virtual screening of protein localization signals, and localization signal generation.\n\nWeakness:\n1.While generally clear, some parts of the paper, especially the technical details of the model and the training process, could be challenging for readers without a strong background in machine learning and diffusion models.\n2.The datasets used in the experiments, while relevant, might be considered limited in size and diversity, which could affect the generalizability of the model.", "other_comments_or_suggestions": "In the methodology section, providing more visual aids or diagrams to illustrate the diffusion processes and the network architecture could further improve clarity.\n\nIt would be beneficial to discuss the computational cost and scalability of CELL-Diff in more detail, as this is an important consideration for practical applications.", "questions_for_authors": "The authors mentioned that \"Some parts of the paper, especially the technical details of the model and the training process, could be challenging for readers without a strong background in machine learning and diffusion models.\" Could the authors provide more clarification or additional explanations on these technical details to make the paper more accessible to a broader audience?\n\nThe authors used the HPA and OpenCell datasets for their experiments. Given the limited size and diversity of these datasets, could the authors discuss the potential impact of this limitation on the generalizability of the model and how future work could address this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "CcQN28MkK4", "reviewer": "Reviewer_nmZP", "summary": "This paper introduces CELL-Diff, a diffusion-based model capable of bidirectionally generating microscopy images from protein sequences and protein sequences from microscopy images. Using conditional morphology reference images (nucleus, ER, microtubules), the model combines continuous diffusion for images and discrete diffusion for sequences. CELL-Diff significantly improves image quality compared to previous methods, evaluated on the HPA and OpenCell datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1:** The claim of accurate image-to-sequence transformation lacks rigorous quantitative evidence. Evidence provided for sequence generation is limited to qualitative motif analyses.\n\n**Response:** To quantitatively evaluate the accuracy of the generated sequences, we used DeepLoc 2.1[1] to assess whether the generated sequences contain recognizable Nuclear Localization Signals (NLS) motifs. DeepLoc 2.1 is a deep learning-based classification model for predicting protein subcellular localization based on discrete annotations.\n\nSpecifically, we fused the generated sequences to the C-terminus of Green Fluorescent Protein (GFP) and input the fused proteins into DeepLoc 2.1. The model then predicted whether the fusion proteins contained NLS.\n\nWe compared CELL-Diff's performance with CELL-E2. For CELL-Diff, we used the 100 generated proteins listed in Table 3, while for CELL-E2, we evaluated the first 100 sequences from their supplementary materials, ranked by their generation scores.\n\nAccording to DeepLoc 2.1, 78 out of 100 CELL-Diff-generated sequences were predicted to contain NLS motifs, while only 46 out of 100 CELL-E2-generated sequences were recognized as NLS motifs. This indicates that CELL-Diff has a higher probability of generating NLS-containing sequences compared to CELL-E2.\n\n---\n\n**Comment 2:** How deterministic or diverse are the sequences generated from a given microscopy image?\n\n**Response:** To assess the diversity of sequences generated from a given microscopy image, we performed a diversity analysis on 100 generated NLS sequences. We used the following three metrics: Levenshtein Distance, Sequence Entropy, and Tanimoto Diversity. The results are shown below:\n\n| **Model**    | **Levenshtein Distance** | **Sequence Entropy** | **Tanimoto Diversity** |\n|--------------|---------------------------|-----------------------|------------------------|\n| CELL-E2      | 14.25                     | 3.83                  | 0.99                   |\n| CELL-Diff    | 9.91                      | 2.87                  | 0.83                   |\n\nFrom the results, we observe that CELL-E2 produces more diverse NLS sequences compared to CELL-Diff. However, the lower diversity in CELL-Diff reflects its stronger conditioning on the input image, which leads to more biologically meaningful and consistent sequence patterns. As demonstrated in response to Comment 1, CELL-Diff generates NLS sequences more likely to be valid, as confirmed by DeepLoc 2.1 analysis.\n\n---\n\n**Comment 3:** Is there any cycle consistency between the two directions? Say for a protein sequence, generate an image, and then feed that generated image back into the model to generate a sequence, do you get back a similar sequence?\n\n**Response:** In general, the space of protein sequences is much larger than the functional space of images. For instance, proteins with different NLSs can produce visually indistinguishable images, as they all localize to the nucleus. Therefore, if we start with one NLS sequence, generate an image showing nuclear localization, and then use that image to generate a new sequence, the resulting sequence will likely contain a different NLS but still direct the protein to the nucleus. Hence, sequence-to-image-to-sequence cycle consistency is not a suitable measure of the model's performance, as it does not guarantee the recovery of the same sequence.\n\nHowever, image-to-sequence-to-image cycle consistency is more meaningful, as it evaluates whether the generated sequence preserves the same localization or pattern as the original input image. To assess this, we conducted a cycle consistency validation experiment using NLS localization.\n\nWe started with a protein image showing nuclear localization, similar to Figure 9 (left). We then used the model to generate 10 sequences fused to the C-terminus of GFP with different sequence lengths. Next, we fed the generated sequences back into the model to generate corresponding protein images.\n\nTo quantify the consistency, we measured the IoU similarity between the original and regenerated images. We compared CELL-Diff with CELL-E2, and the results are presented below:\n\n| **Number of Amino Acids** | **CELL-E2** | **CELL-Diff** |\n|----------------------------|-------------|----------------|\n| 10                         | 0.575       | 0.763          |\n| 20                         | 0.566       | 0.753          |\n| 30                         | 0.579       | 0.764          |\n| 40                         | 0.574       | 0.751          |\n\nFrom the results, CELL-Diff demonstrates better cycle consistency than CELL-E2, as indicated by the consistently higher IoU scores. This suggests that CELL-Diff generates sequences that better preserve the localization and pattern information of the original input image.\n\n---\n\n[1]: Ødum, Marius Thrane, et al. \"DeepLoc 2.1: multi-label membrane protein type prediction using protein language models.\" *Nucleic Acids Research* 52.W1 (2024): W215-W220.", "claims_and_evidence": "The authors claim CELL-Diff provides high-fidelity microscopy image generation and accurate sequence-to-image and image-to-sequence transformations. While the image generation performance is strongly supported through quantitative (FID, IoU, MSF metrics) and qualitative evaluations, the claim of accurate image-to-sequence transformation lacks rigorous quantitative evidence. Evidence provided for sequence generation is limited to qualitative motif analyses.", "methods_and_evaluation_criteria": "- The methodological choice (combining continuous diffusion for images and discrete diffusion for sequences in a unified transformer-based U-Net) is interesting and sound. \n- However, the authors' evaluation of image-to-sequence generation is weak, relying primarily on qualitative assessments rather than robust quantitative analyses. The paper could benefit from a larger-scale motif analysis / cluster overlap with localization etc.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design for image generation is solid with clear metrics and appropriate baselines.", "relation_to_broader_scientific_literature": "The paper adequately discusses relevant works in the field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Can you please share your thoughts on previous sections?\n- How deterministic or diverse are the sequences generated from a given microscopy image?\n- Is there any cycle consistency between the two directions? Say for a protein sequence, generate an image, and then feed that generated image back into the model to generate a sequence, do you get back similar sequence?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces CELL-Diff, a unified diffusion model for bidirectional generation between protein sequences and fluorescence microscopy images. It combines continuous diffusion for image synthesis and discrete diffusion for sequence prediction, using a transformer-based architecture with cross-attention to integrate multimodal information. The model was trained on the Human Protein Atlas and fine-tuned on the OpenCell dataset, and it significantly improves image fidelity and protein localization. While many reviewers appreciate its potential practical values, one reviewer raised concerns on the small size of datasets. In particular, the reviewer commented that the protein localization, which depends on cell type, environment, and post-translational modifications can be very complex, and the sequence-to-image mapping is inherently one-to-many. In such a case, there is a risk of overfitting and memorizing the training data, and there is no strong evidence of good performance on the generalization of the model to new samples. The authors emphasized that CELL-Diff doesn't aim to model all sequence-to-image mappings but focuses on "virtual experiments" involving sequence variants in well-characterized cell lines. They also explained that HPA covers over 60% of human proteins and that pretrained protein embeddings help mitigate limited sample size, and GFP + yeast NLS sequences were specifically selected to avoid homology with the training data, and BLAST was used to confirm this. These explanations addressed some of the questions raised by the reviewer. Therefore, combining the positive feedbacks from other reviewers, I recommend a weak accept.
Decision: Accept (poster)