CELLE-2: Translating Proteins to Pictures and Back with a Bidirectional Text-to-Image Transformer
Authors: Emaad Khwaja
Yun S. Song
Aaron Agarunov
Bo Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: text-to-image
protein localization
protein engineering
transformers
TL;DR: A bidirectional text-to-image transformer trained on a large corpus of amino acid and fluorescent image data. We demonstrated a novel image-based protein engineering method.

Abstract: We present CELL-E 2, a novel bidirectional transformer that can generate images depicting protein subcellular localization from the amino acid sequences (and vice versa). Protein localization is a challenging problem that requires integrating sequence and image information, which most existing methods ignore. CELL-E 2 extends the work of CELL-E, not only capturing the spatial complexity of protein localization and produce probability estimates of localization atop a nucleus image, but also being able to generate sequences from images, enabling de novo protein design. We train and finetune CELL-E 2 on two large-scale datasets of human proteins. We also demonstrate how to use CELL-E 2 to create hundreds of novel nuclear localization signals (NLS). Results and interactive demos are featured at https://bohuanglab.github.io/CELL-E_2/.

CELL-E 2: Translating Proteins to Pictures and Back with a Bidirectional Text-to-Image Transformer
Emaad Khwaja"1 emaadOberkeley. edu
Yun S. Song yss@berkeley.o edu
Aaron Agarunoy agarunov aaron@mmail.ccomm
Bo Huang bo. www.guuuss.....m edu
Abstract
We present CELL-E 2, a novel bidirectional transformer that can generate images depicting protein subcellular localization from the amino acid sequences (and vice versa). Protein localization quence and image information, which most existing methods ignore. CELL-E 2 extends the work of CELL-E, not only capturing the spatial complexity pro- tein localization and produce probability estimates of localization atop nucleus image, but also being able to generate sequences from images, enabling de novo protein design. We train and finetune CELL-E 2 on two large-scale datasets of human proteins. We also demonstrate how to use CELL-E 2 to create hundreds of novel nuclear localization signals (NLS). Results and interactive demos fea- tured at http://ouuuungaaaggiguubbo///CCo
challenging problem that requires integrating se-
Introduction
Subcelllular protein localization is vital aspect of molecular biology as it helps in understanding the functioning of cells and organisms [1]. The correct localization of protein is critical for its proper functioning, and mislocalization can lead to various diseases
[2]. Protein localization prediction models have typically relied on protein sequence data [3, 4] or fluorescent microscopy images [5, 6] as input to predict which subcelleular organelles protein would localize to, designated as discrete class labels [7. 8). The CELL-E model was markedly different in that utilized an autoregressive -xtatt--mmae framework to predict subcellular localization as images [9]. thereby overcoming bias from discrete class labels derived from manual annotation [10]. Furthermore, CELL-E was capable of producing 2D probability density function throughout the dataset, yielding more a far more interpretable output for the end user.
image based on localization data seen
Although novel, CELL-E was inherently restricted by the following limitations:
Autoregressive Generation. Alongside other autoregressive models [11-14], CELL-E was lim- ited by slow generation times and unidirectionality. When provided with input, CELL-E required separate step for each image patch (256 for the output image composed tokens of size 16 16).
"CC Berkeley UCSF Joint Bioengincering Graduate Program
Department of Statistics, UC Berkeley, CA 94720
Computer Science Division, UC Berkeley, 94720
Department of Pathology, Memorial Sloan Kettering Cancer Center, 10065
Department Pharmaceutical Chemistry, UCSF, San Francisco, CA 94143
|| Department of Biochemistry and Biophysies, UCSF, San Francisco, CA 94143
Chan Zuckerberg Biohub- San Francisco, San Francisco, CA 94158
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

This slow image generation severely limits the ability of CELL-E to perform thghtthroughput mutagenesis screening.
Unidirectional Prediction. The unidirectional nature of CELL-E allowed for predictions to be made in response predictions of sequence given a localization pattern. Such capability would be advantageous for those working in a protein engineering domain [15, 16]. One could imagine a researcher wanting to know the optimal localization sequence to append to a protein on either the N or C terminus [17] while maintaining essential function within an active site region, well reducing the chance of off-target trafficking.
amino acid sequence, however it may be of interest to biologists to make
Limited Dataset. CELL-E utilized the OpenCell dataset [18], relatively small dataset. Vision transformers often require large amounts of data to make robust predictions [19], however a small dataset was utilized in the original model. This led to degree of overfitting and prediction bias based on the limited diversity in localization patterns of the original dataset.
Present Work. As in CELL-E, our method CELL-E 2 is able to generate accurate protein local- ization image prediction as illustrated Fig. I, but differs from CELL-E by employing a non- autoregressive (NAR) paradigm which improves the speed of generation. Similar to CELL-E, retrieve embeddings from a pre-trained protein language model and concatenate these with learned embeddings corresponding to image patch indices coming from nucleus (a subcellular organelle containing DNA [20]) image and protein threshold image encoders (Fig. 2). We then apply masking to both the amino acid sequence as well the threshold image in unsupervised fashion, and reconstructed tokens are predicted in parallel, allowing for generation fewer steps. This also al- lows for bidirectional prediction, (sequence protein threshold image or protein threshold image to sequence). Additionally, to improve the predictive performance we utilize a larger corpus of data, the Human Protein Atlas (HPA) [21] in pre-training to expose the model ization diversity, and finetune on the OpenCell dataset [18], which better represents natural protein localization because acquired from live instead of fixed cells. We explore multiple strategies towards finetuning which serves to generally inform task-specific refinement text-to-image models in Section 5.3.
higher degree of local-
2
Related Work
2.1 Protein Language Models
Embeddings from unsupervised protein language models can be used to predict and analyze the properties of proteins, such as their structure, function, and interactions [22]. By exploring the hid- den patterns and relationships within these sequences, protein language models can help to advance our understanding of the complex world of proteins and their roles various biological processes. Masked language modelling has been particularly successful. Ankh [23], ProtT5 [24], ProGen [25], ESM-2 [26], and OmegaFold [27] are examples of recent models which use masked langauge ap- proaches. Embeddings from ESM-2 and Omegafold in particular have been used for structural prediction, indicating hierarchies of information beyond the primary sequence contained in the em- beddings [28].
2.2 Protein Localization Prediction
Protein localization prediction via machine learning is an emerging field that uses computational algorithms and statistical models predict the subcellular spatial distribution of proteins [29]. This is an essential task in biology, as the subcellular localization of a protein plays a crucial role in determining function and interactions with other proteins [30, 31] The prediction of protein lo- calization is performed by analyzing protein sequences, amino acid composition. and other features that can provide clues about their subcellular location. Machine learning algorithms are trained on large datasets of labeled proteins to recognize patterns and make predictions about the subcellular location of a protein. This field has the potential to improve our understanding of cellular processes, drug discovery, and disease diagnosis.

---

Thresholded Label (Ground Trutk)
Predicted Distribution
Original Image
Generated Image
T-complex protein subuniic thota
Chromosome: asso KIPIA
Phoeppaate heapppaauyyyyeeeeeee tnoeitol -binding clathrin ausembly
Protensome assembly
Figure 1: Localization predictions from CELL-E 2 (HPA Finetuned (Finetuned HPA VQGAN) 4.480) on randomly chosen validation set proteins from the OpenCell dataset. All images feature the Hoescht-stained nucleus image base. The "Original Image" column shows the fluo- rescently labelled protein from the dataset. The "Thresholded Label" shows pixels greater than the median value. This serves the ground truth for the model during training. "Generated Image" is the image specifically predicted by CELL-E and is compared against the thresholded ground truth image. "Predicted Distribution" is the latent space interpolation of the binary threshold image tokens which uses which utilizes the output logits of CELL-E 2. See Fig. SI for colorbars corresponding to all plots in this work.
Recently, attention-based methods have demonstrated the ability to predict localization from se- quence [32], enabling the use of long context information when compared to convolutional-base counterparts [33-35]. These methods, however, predict localization as discrete classes rather than as an image. CELL-E, on the contrary, does not utilize existing annotation and provides a heatmap of the expected spatial distribution per-pixel basis [9]. This approach enables learning at scale by eliminating the bottleneck of manual annotation while also circumventing label bias.
2.3 Text-to-Image Synthesis
Recent gains in the text-to-image field have have largely been made by autoregressive models [11, 13), which correlate text embeddings with image patch embeddings, as well as diffusion models, [14, 36-39], which condition on sentence embeddings to gradually synthesize images from random noise.

---

145RASOL BIIGH PAGDTENSSR
Enoss-y
Alicinur RHHNA
Protein Language Model Tokenizur
Nuclous VOGAN Tokenizaar
Reconstructed Tokens
Imputs
Threshold VODAN Tokenzeer
Aftentics
Input
Tokens
Transformer
Figure 2: Depiction of training paradigm for CELL-E Gray squares indicate masked tokens. Loss is only calculated on masked tokens in the sequence and protein threshold image.
Other works implement non-autoregressive models (NAR), which take advantage of masked re- construction procedure, similar to BERT [40], where the model is tasked with predicted randomly masked portions of an input image. These types of models are partieulalry advantageous because they enable parallel decoding, allowing images to be synthesized relatively view steps when com- pared to autoregressive models. Furthermore, NAR models are not bound to a particular direction of synthesis like autoregressive models, which only perform nex(-token prediction. Viewiew2 [41] utilizes modified transformer architecture where attention on masked tokens is eliminated. MUSE [42] builds on MaskGIT [43] by concatenating resentation of corresponding image. It uses vanilla transformer architecture [44] and yielded state-of-ther image synthesis performance in terms of FID and human evaluation.
pre-trained text embedding to token masked rep-
3 Datasets
We pretrained our model on protein images from the Human Protein Atlas (HPA) [45], which cov- ers various cell types and imaging conditions using inmunoftuoreseecce microscopy 1. We then finetuned on the OpenCell dataset [18]. which has consistent modality using live-cell confocal microscopy of endogenously tagged proteins. To ensure generalization to new data, we followed the homology partitioning method of Almagro Armenteros al. [35]. We used PSI-CD-HIT [46] to cluster HPA proteins at (2 50%) sequence similarity and randomly split the clusters into 80/20 train/validation sets. We applied the same clustering and splitting the OpenCell proteins, match- ing the train/valin hili/validtiinn labels from HPA. For proteins present in OpenCell but not HPA (= 176).
assigned the protein based the other labels the cluster. Any remaining unassigned proteins (= 1) were assigned to the training set. See Appendix for more details about the datasets and preprocessing.
4 Methods
CELL-E 2 (Fig. 2) a masked NAR transformer model, which upgrades the capabilities of CELL- E, an autoregressive decoder-only model [47]. Due to the NAR nature of the model, CELL-E capable of both image generation (sequence to image), as well as sequence prediction (image to sequence).

---

4.1 Amino Acid Sequence Embeddings
Proteins are biological molecules which are comprised of individual amino acids. CELL-E 2 utilizes embeddings from ESM-2 [26]. where amino acid molecules are denoted with individual letter codes (e.g. for alanine) [48]. We opt use frozen amino acid embeddings for the prediction task, which has been demonstrated to yield superior reconstruction performance in text-to-image models [9, 37, 42]. The embeddings obtained from a protein language model contain valuable information about amino acid residues, biochemical interactions, structural features, positional arrangements, as well as other characteristics like size and complexity [22]. We train models of varying size based on the released ESM-2 checkpoints (See Section 5). The output of the final embedding layer per respective model is used as the amino acid sequence embedding.
4.2 Image Tokenization
Just as in Khwaja et al. [9], we utilize a nucleus image, which serves as a spatial reference with which a binarized protein threshold image is associated. We chose this in order to parallel the type of images which are typicall acquired in a wet lab scenario.
We also utilize VQGAN autoencoders [49] trained both the HPA and OpenCell datasets, respec- tively. VQGAN surpasses other quantized autoencoders by incorporating a learned discriminator derived from GAN architectures [50]. Specifically, the Nucleus Image Encoder employs VQGAN to represent 256 x 256 nucleus reference images as 16 x 16 image patches, with a codebook size of (n 512) image patches. To enable transfer learning, we explore finetuning strategies these VQGANs in Section 4.5.
The protein threshold image encoder acquires compressed representation of a discrete probability density function (PDF) that maps per-pixel protein positions, presented as an image. We binarize the image based on the median pixel value of the image (see Appendix A.4). We utilize VQGAN architecture identical to the Nucleus VQGAN to estimate the entire set of binarized image patches to denote local protein distributions. These VQGANs trained until convergence, and the discrete codebook indices are used for the CELL-E 2 transformer. Hyperparameters (Table S1, Table S2, Table S3) and training details can be found in Appendix B.2.
4.3 Input Masking Strategy
We adopt a cosine-scheduling technique for masking image tokens, which has been used by other works. The probability of an image patch being masked is determined by a cosine function, favoring high masking rates with an expected masking rate of 64% [42, 43]. This technique provides various levels of masking during the training process, exposing the model to spatial context for masked language tokens.
Of similar interest as image prediction, sequence in-filling with respect to a localization pattern has potential to significantly augemnt biologist's workflow. Typically, protein localization sequences are found through sequence similarity searches with proteins that have known localization in par- ticular organelles [51-53] or via experimentation [54, 55]. CELL-E 2's bidirectionality enables the model to make predictions for image with respect to amino acid sequences, well as sequence predictions conditioned on images. This enables an entirely new paradigm in protein engineering which relies image information. To achieve this, we also mask the language tokens along with the protein threshold image tokens. We experimented with using the same cosine function for image masking but found it led to numerical instability and vanishing gradients. Therefore, linearly scaled the cosine function to ensure the maximum masking rate matched 15% masking rate used to train ESM-2.
4.4 Base Transformer
The base transformer is an NAR model in which the embedding dimension set to the embedding size of the pretrained language model used. We utilized two types of masking tokens. For masking the amino acid sequence, we leveraged the mask token which already exists within the ESM-2 dictionary, designated as <MASK_ _EQ> The VQGAN does not contain masking token within its codebook, so to represent it, we add an additional entry in the image token embedding space of CELL-E 2 (with n + (512+1 +1= 513), where is the number of tokens in the VQGAN codebook),
5

---

and designate the final token as <MASK_ IM>. We additionally create embedding space of length 1 for the <SEP> token which is appended the end of the amino acid sequence. Training details can be found in Appendix B.2.
We sample from this transformer by strategically masking positions in the image or sequence (see Appendix B. The logit values for the image prediction are used as weights for the threshold image patches to produce a predicted distribution (Fig. I, Fig. S5).
4.5 Finetuning
We sought to leverage useful information from both HPA and OpenCell. HPA contains many pro- teins (17,268), but is potentially subject to inaccuracies, fundamentally because of the immunohis- tochemistry used for staining requires several rounds of fixation and washing [21]. This means the proteins are not observed in live cell; are subject to signal loss, artifacts, and/or relocalization events; and therefore may not necessarily represent the true nature of protein expression and dis- tribution within cell [56]. The OpenCell dataset, while comparatively smaller, overcomes these issues by using a split- fltt-fluorescent protein fusion system allows for tagging endogenous genomic proteins, maintaining local genomic context, and the preservation of native expression regulation for live cell imaging [18, 57]. We therefore utilize the HPA dataset for pretraining, and then finetuned on OpenCell.
Optimally finetuning withinin the text-to-image domain remains an open question. The use of mul- tiple models makes it difficult to pin down the correct strategy. Contemporary efforts utilize pre- trained checkpoints to fine-tune on domain specific data [58-60]. Chambon et al. [61] reported improved synthesized image fidelity when fine-tuning the U-net of text-to-image diffusion model, but similar fine-tuning strategies have not been explored for patch-based methods. We report our findings in Section 5.3.
Results
Similar to CELL-E, we cast the embedding spaces for the image tokens at the same size as the ones used by the pre-trained language model. The size of the embedding vectors ("Hidden Size") for each model was chosen based on the publicly available ESM-2 checkpoints. For instance. a CELL-E 2 model with hidden size 480 uses esm2_ t12 1_255M______ UR50D, which corresponds model with 12 attention layers. Khwaja et al. [9] demonstrated a positive relationship between the number of attention layers (designated "Depth") in the base transformer and the image prediction performance. The maximum depth was set based on our available GPU memory capacity. We refer to models using the name format "Training Set Set_Hidden Size".
35M parameter
5.1 Protein Localization Image Prediction Accuracy
To predict the protein localization image, we provide CELL-E 2 with the protein sequence and nucleus image, and fill the protein image token positions with <MASK_ IM> tokens (Fig. S3).
We evaluated the models on several image metrics (see Appendix C.1) that measure the quality and diversity of the generated protein images (Table 1). Additionally, we assessed the model's generalization capabilities by testing them on other dataset (HPA-trained model on OpenCell and vice versu) (Table S4). We report the results for each model on its respective dataset. We observed a significant positive effect of depth on performance across all metrics and datasets. The models with hidden sizes of 480 and 640 achieved the highest scores, with significant difference between them. However, on the HPA dataset, HPA_640 surpassed the HPA_ HPA_480 480 model in more categories. On the OpenCell dataset, OpenCell_42B 480 performed better than the OpenCell 640.
We conducted visual examination of the generated protein images as depicted in Figures Fig. S6 and Fig. S7. Among the models, OpenCell demonstrated higher visual resemblance and consis- tency with its respective ground truth labels, although they exhibited low entropy in predicted distribution. This indicates that while these models accurately identified the correct tokens with high probability, they struggled to account for the uncertainty and variety inherent in other valid choices, possibly due to tendency for rapid overfitting which hindered their generalizability.

---

Table 1: Validation Set Image Prediction Accuracy. MAPE: mean absolute percentage error, MAE: mean absolute error, SSIM: structural similarity index measure, FID: Fréchet inception distance, IS: inception score.
Dataset
Hickenn Size
Depth
Nucleus Proportion MAPE
Image MAB
PDF MAB
SSIM
FDD
1S
480
68
0,0257 0.0250
0.3340 -: 0.0788
0.1846 1: 0.0985
0.2633 a: 0.1781
12.0332
2.2900 4: 0.0410
HPA
610
55
0.0294 1: 0.0278
0.3283
0.0805
0.2842 1: 0.0991
0.2826
0.1827
21.7942
2.2618
4: 0.0361
1280
25
0.0370 1: 0,0360
0.3622
= 0.0799
0.2967
4: 0.0985
0.2645
4 0.1857
1.5161
2.5440
=: 0,0490
2560
0.0794
0.3516
0.0792
0.3101
0.0904
0.2558
0.1619
23.7977
2.1578
4801
68
0.0161 1.0148
0.4953
0.0064
0.3620 + 0,1168
0.1220
0.1188
1,5844
2.6069
0.1175
OpenCell
640
55
0,0159 0.0136
0.4995
0.0006
0.3785
0.1008
0.1011 + 0.1012
2.6966
2,0974 + 0.0981
1280
25
0.0272 0.0223
0.4996
0.0010
0.1359
1: 0,0700
0.0694
0.0472
8,9102
1.3712 0.0432
2560
0.0684 + 0.0511
0.4996 = 0.0005
0.4145
+ 0.0889
0.0890 0.0667
9.5116
1,4176 0.0329
Table 2: Validation Set Masked Sequence In-Filling
Dataset
Hlikken
Depth
Sequence MAE
Cosine Simillaitty
480
0,8628
0.0951
0,9004 0.0237
640
0.7917 0:1245
09577 0.0216
1280
25
0,6512 = 0.1794
0,9708 = 0.0163
2560
0,5759 = 0.2332
0,9722
0/0200
480
68
0.7507 + D.1709
0,9533
0.1255
OpenCaall
640
0.6641 + 0.1764
0.0272
1280
0.5698 0.2016
0.9709
0.0220
2560
0.4950 5
= 0.2456
0,9711 = 0.9271
We also observed models had stronger performance with respect to the dataset on which they were trained. Notably, the model trained on the HPA dataset outperformed the OpenCell-trained model on the OpenCell dataset, showcasing lower PDF MAE values across all categories. This HPA model also displayed lower FID on the OpenCell validation set, underscoring the advantage of hav- ing a more extensive dataset even under differing imaging conditions. The OpenCell, __880 model achieved the best scores in half of the evaluated metries: MAPE, MAE, SSIM, and IS.
5.2 Masked Sequence In-Filling
To test each model's sequence learning, we used masked in-filling task similar to the training task. Similar to Section 5.1, we provide CELL-E 2 with randomly masked (15%) sequence, an unmasked nucleus image, and an unmasked protein threshold image. To select the sequence predic- tion, we perform a weighted random sampling operation from the 3 amino acids with the highest predicted probabilities. We measured the accuracy as the percentage of correct predictions (noted as "Sequence MAE", see Appendix C.2). We then embedded each reconstructed sequence with esm2_ $tt__3___Be___33_ 3B__RODD the largest model we could fit in memory, with 3B parameters, 36 layers and an embedding dimension of 2560. We computed the mean cosine similarity between the embed- dings of the original and reconstructed sequences at masked positions. We show validation results in (Table 2) and all results in (Table S6).
Most models had low performance on this task in terms of reconstruction. This is understandable because the models learned to generate amino acids that were common or frequent in the dataset, but not necessarily correct for the specific sequence. On the other hand, we observed values close to 1 for the cosine similarity, indicating that the predicted amino acids had similar embedding values to the original ones at the masked positions. This could be because the models learned to capture some semantic or structural features of the amino acids, such as polarity or charge, that were reflected in the embedding space and contributed to the biological function of the sequence. Models that used the embedding model with an embedding dimension of 2560 had the best performance. For example, OpenCEll 2560 had the best performance on both metrics, with MAE of .4950 and cosine similarity of .9711. When compared to randomly selected amino acids for each position (Table S7), we note significantly higher Sequence MAE and Cosine Similarity.
We also note that the reconstruction ability does not improve the performance of the original lan- guage models (Table S5). This may be result of the combined image/sequence loss used during training or smaller corpus of data compared to datasets used for the training the original language model. Evaluation results across both datasets can be found in (Table S6).
7

---

Table 3: OpenCell Validation Set Image Prediction Accuracy after Finetuning
Fine-Tuned
Threshold Image Eneader
Nacleus Proportion MAPE
Image MAE
PDF MAE
SSIM
FID
15
No
HPA
0,0181 0,0168
0/415 1 0.0594
0.3887 1. 0.1270
0.1250 1 0.1149
3,9509
2,1739 1 0.52
OpenCell
0.0161 0.0148
0.4953 0.0064
0.3620 0.1168
0.1220
0.1188
1.5844
2,6069
0.1175
HPA
0.0151
0,3776 0,0834
0.1268
0.1869
0.1503
17.4075
2,9113
0.1199
Yes
OpenCall
0.0159 0.0156
0.4996 1: 0.0006
0,3506
0.1208
0.1574
1: 0.1372
2,5026
2.7168
0.1137
Yes
HPA Finetuned
0,0170 0.0160
0.3449 + 0.1305
0,3488
0.1340
0,1881 0.1541
19,2683
3,6083
0.2013
5.3 Finetuning
While the HPA dataset contains information about wide variety of proteins, the model does not innately perform as well on the OpenCell data. We considered the potential of utilizing an HPA- trained model and finetuning on the OpenCell data, thereby introducing wider protein context than what is found in the OpenCell data alone while adapting to the imaging conditions and cell type found within the new dataset. We experimented with different finetuning strategies for CELL-E 2 on the OpenCell dataset. We used the pre-trained HPA checkpoint as the starting point for all finetuned models, continuing training on the OpenCell train set. We also evaluated the pre-trained HPA and OpenCell checkpoints without any finetuning as baselines. The finetuned models differed in how they updated the image encoders:
HPA Finetuned (HPA VQGAN): we kept the original VQGAN image encoders from the HPA checkpoint.
HPA Finetuned (OpenCell VQGAN): we replaced the image encoders with VQGANs trained only on OpenCell data.
HPA Finetuned (Finetuned HPA VQGAN): we finetuned the HPA image encoders while keeping the rest of the model frozen, then froze the image encoders and update the transformer weights.
Fig. S8 shows image predictions 480. Surprisingly, the pretrained HPA model already achieved strong performance on the OpenCell dataset without any finetuning (see Table S8).
OpenCell validation protein for models with hidden size
The best results were obtained by utilizing a pre-trained HPA checkpoint. We first finetuned both VQGAN image encoders while freezing the rest of the model. We then froze the VQGAN weights and allowed the base transformer to update (see Table 3). We attribute the 1.81% improvement in MAE, along with the improvements in FIDD and IS, to the finetuning of both the VQGANs, as it improved the consistency of image patch tokens. This provided the checkpoint with more reliable image patches to generate from. However, swapping the HPA VQGAN with an OpenCell one led to a similar losses of distribution information seen in Fig. S7. This could be because the model overfits before being able learn probabilities across tokens. The learning obstacle comes from the possibility that images patches within the finetuned OpenCell VQGAN have sufficient (or even more) pixel consistency with the images, but the patch positional indices are misaligned with those of the HPA VQGAN. These findings are consistent with those found in analogous text-to-iman to-image works utilizing diffusion models. We did not find that finetuning improved the model's 's sequence reconstruction ability (see Table S9).
6 Discussion
6.1 CELL-E Comparison
In Table S10 and Table S11, we compare the performance of image localization prediction from scratch for CELL-E 2 and CELL-E. On the OpenCell validation set, CELL-E merer-perfomms CELL- 2 both before and after finetuning with regards to Nucleus Proportion MAPE. CELL-E 2 achieves worse Image and PDF MAE metrics before finetuning, however after finetuning CELL-E 2 achieves a 2.2% improvement for Image MAE and 1.7% for PDF MAE. On the contrary, CELL-E performs better with respect to image fidelity metrics SSIM and FID.
With respect to generation time, we found that the CELL-E 2 with hidden size of 480 was able to generate a prediction 65x faster than the CELL-E model. This is result of CELLE- 2's capability
8

---

to generate prediction in a single step (.2784 seconds), which enables the advent of large-scale in silico mutagenesis studies.
6.2 De nove NLS Design
CELL-E 2's bidirectional integration of sequence and image information allows for an entirely novel image-based approach to de novo protein design. We applied CELL-EE 2 to generate NLSS, which is short amino acid sequence motif that can relocate target protein into the cell nucleus when append to the target protein. In this case, our choice of the target protein the Green Fluorescent Protein (GFP), a common protein engineering target [62-64] that is non-native to the human proteome and absent in the datasets. NLSs are usually identified by experimental mutagenesis studies or in silico screens that search for frequent sequences in nuclear proteins [51, 65]. However, these methods may yield candidates that are highly similar to known ones or not specific to the target protein. A more recent approach uses machine learning statistical priors [17], but it is limited by the distribution of training samples due to the scarcity of experimentally verified NLSs. CELL-E 2 overcomes these limitations because it does not rely on explicit labels, and can therefore leverage significantly more unlabelled image data.
sequence identity to augment featurization and
We generated a list of 255 novel NLS sequences for GFP using the procedure described in Ap- pendix D.2. Briefly, we insert mask tokens of set length in a GFP sequence and task model with best sequence in-filling performance (OpenCell_ 2560) to fill in the masked amino acids, conditioned on a threshold image generated from the nucleus image (via Cellpose segmentation [66]). To verify the accuracy of the prediction, we pass the predicted sequence through the best performing image model (HPA Finteuned (Finetuned HPA VQGAN) _480), and quantify the proportion of signal intensity within the nucleus of the predicted threshold image (Fig. S9). The NLS sequences were then ranked based on sequence and embedding similarity with known NLSs (see Appendix D.2). The list of candidates can be found in Appendix D.3. We found several NLS candidates with high predicted signal in the nucleus, but which were fairly dissimilar from any protein found within NLSdb [65].
Classical NLSs are characterized by having regions of basic, positively charged amino acids arginine (R) and lysine (K) [67, 68], and are categorized as "monopartite" or "bipartite", either having single cluster of basic amino acids or two clusters separated by linker, respectively [69]. We observed a postive correlation between percentage of R and K residues in our predicted NLSs and sequence homology with known NLSs (Table S12). The number of clusters per sequence followed a similar trend, with sequences with relatively low sequence homology (Max ID % I 33) having at most 2 clusters in 88% of predictions (Fig. S10). The remaining predictions, if correct, represent non- classical NLSs.
To further verify our predicted sequences, we passed the predicted NLS appended to GFP through Deeploc 2.0 [32], a leading sequence-to-class protein localization model, which predicted 89% of generated sequences were nuclear localizing and 91 % contained potential nuclear localizing signal.
Similar to CELL-E, we observed high attention weights on documented localization sequences cor- related with positive protein signal within the threshold image (Fig. S11). For sequences with high predicted nucleus proportion intensities, we observed high activation across the entire sequence (novel NLS and GFP residues), with some NLS weights being an order of magnitude higher than others across the GFP sequences (Fig. 3). On the contrary, predicted sequences with comparatively less predicted intensity within the nucleus had low activation across the sequence, with little to none in the proposed NLS. We observed similar amounts of attention placed on the nucleus image patches, which largely corresponded to the location of the predicted threshold patches.
7 Conclusion & Future Work
In this paper, we have presented CELL-E 2, a novel bidirectional NAR model for protein design and engineering. CELL-E 2 can generate both image and sequence predictions, handle multimodal inputs and outputs, and run significantly faster than the state-of-the-art. By pre-training on a large HPA dataset and fine-tuning on OpenCell, CELL-E 2 can achieve competitive or superior perfor- mance on image and sequence reconstruction tasks. However, one limitation of CELL-E 2 is its output resolution, which currently (256 256). This resolution may not capture the fine details of microscopy images, which may limit applications in real-world use where Megapixel images are ac-

---

Relative Attention Weights for Image Prediction
AQAKIRRIGFIE Puykt
Valdon Iaag Alenker Wesall
Nolien Veaa Adtedion Bingla
Madaa Iridde Alindie Righh
Gesrrddd Thnee
Nedia Fate Abudio Wriple
Ginerited Houg
Vednn loog Aliemkes Wichlt
Ororatal lawgy
Nelies Aterice Begplo
Gromand
Figure 3: Attention weights associated with positive signal within the predicted image. Tokens with higher attention weight associated with background patches (low signal) not highlighted. See Appendix D.3 for more information about the visualization process. We show sequences with the highest (left column) and lowest (right column, not included in Table S13) predicted nucleus proportion intensity. The NLS+GFP sequences are shown with the designed NLS boxed in red.
quired. Increasing the output resolution CELL-E 2 is one direction for future work. Furthermore, the sequence prediction struggles with the prediction of large stretches of amino acids opposed to singular masked positions. Within this work, we encountered trade-off between sequence pre- diction quality and prediction speed which may be overcome by reformulating the masking strategy. Similar findings were seen when compared with CELL-E, where we found accuracy measurements improve with CELL-E 2 the detriment of image quality metrics. The in-order prediction se- quence we utilized in this paper may serve as a bottleneck for protein engineering applications despite the speed advantages gained from using NAR architecture.
Another direction for future work is to incorporate structural information into the sequence embed- dings. CELL-E 2 can generate novel NLS sequences with similar properties to GFP but low ho- mology to existing sequences. However, the current sequence embeddings are based on a language model that may not capture all the structural features of the proteins. These features may affect the image appearance and vice versa.
We believe that CELL-E 2 is promising model for protein design and engineering. We hope that our work will inspire more research on bidirectional NAR models for this domain and other domains that involve multimodal data.
10

---

8 Acknowledgments.
This research is supported in part by NIH grants R01GM131641 (BH) and R35-GM134922 (YSS). B.H. is a Chan Zuckerberg Biohub Investigator. A.A. contributed to visualizations and demos. E.K. responsible for exploration and code.
References
[1] Nicholas C. Bauer, Paul W. Doetsch, and Anita Corbett. Mechanisms Regulating Pro- tein Localization. Traffic, 16(10):003--061, 2015. ISSN 1600-0854. doi: 10.1111/
tra. 12310. URL httpps://onwneelliibbtee wwtparryyylllee wiley. commcooc///oooco. /oo///bs//e /tt////.../////raaaa.a.a.a. www./tra...a...eee 12310.
_eprin:: https//ontintiiiaaaysseesssoooop
1/tra. 12310.
[2] Mien-Chie Hung and Wolfgang Link. Protein localization in disease and therapy. Journal of Cell Science, 124(Pt 20):3381-3392 October 2011. ISSN 1477-9137. doi: 10.1242/jes.
089110.
[3] Yuexu Jiang, Duolin Wang. Yifu Yao, Holger Eubel, Patrick Künzler, lan Max Møller, and Dong Xu. MULocDeep: A deep-learning framework for protein subcellular and suborganellar localization prediction with residue-level interpretation. Computational and Structural Biotechnology Journal, 19:4825-4839, January 2021. ISSN 2001-0370, doi:
10. 10.1016/j.csbj 2021 08.007.. URL https: /www. sciencedirect. com sctience/ee/eeccel/// pii p11/$2000133002103555..
[4] Wen-Yun Yang. Bao-Liang Lu, and Yang Yang. A Comparative Study on Feature Extraction from Protein Sequences for Subcellular Localization Prediction. In 2006 IEEE Symposium on Computational Intelligence and Bioinformatics and Computational Biology, pages 1-8, Toronto, ON, Canada, September 2006. IEEE, ISBN 978-1-4244-0623-4 978-1-4244-0624- 1. doi: 1199CIBBBB.2066 330991. URL http://iseexploree ieee. ptrg/document//
4133173/.
[5] Tanel Pärnama and Leopold Parts.
Accurate Classification of Protein Subcellular Localization from High-Throughput Microscopy Images Using Deep Learning. GenerlGenomeslGeenttes 7(5):1385-1392 May 2017. ISSN 2160-1836. doi: 10.1534/g3.
116.033654. URL https://doi 10rg//...134//33 116. 033654.
[6] Sonam Aggarwal, Sheifali Gupta, and Rakesh Ahuja. A Review Protein Subcellular Localization Prediction using Microscopic Images. In 2021 6th International Conference on Signal Processing, Computing and Control (ISPCC), pages 72-77, October 2021. doi:
10.1 0011009/SPCC33331.22011 212999999..211999433...... ISSN: 2643-8615.
.9699437.
[7] Yuexu Jiang, Duolin Wang, Weiwei Wang, and Dong Xu. Computational methods for pro- tein localization prediction. Computational and Structural Biotechnology Journal, 19:5834- 5844, January 2021. ISSN 2001-0370. doi: 00.1016f..sbb.22011100..
URL https:
/wwp.scceenediree.... com
[8] Gaofeng Pan, Chao Sun, Zijun Liao, and Jijun Tang. Machine and Deep LearningDeep learning (DL) for Prediction of Subcellular Localization. In Daniela Cecconi, editor, Pro- teomics Data Analysis, pages 249-261. Springer US New York, NY, 2021. ISBN 978-1- 07-161641-3. doi: 10. 1-0716 778-1-0716-1641-3.11
URL http:/:///oo..o/1.....11
[9] Emaad Khwaja, Yun Song, and Bo Huang. CELL-E: Biological Zero-Shot Text-to-lmage Synthesis for Protein Localization Prediction, May 2022. URL httpst//:/// /www. biorxiv org/ content/10. 11t1//0112222/ 05 27. 493774v1. sults.
Pages: 2022.05.27.49311
Section: New Re-
[10] Alexandra M. Schnoes, David Ream, Alexander W. Thorman, Patricia C. Babbitt, and Iddo Friedberg. Biases in the Experimental Amnotations of Protein Function and Their Effect on Our Understanding Protein Function Space. PLOS Computational Biology, 150ee033 May 2013. ISSN 1553-7358. doi: 10. 10.1137/jourmal, 1003063. URL https: : // //journals plos. pcbii 1003063. Publisher: Public
org/pl httpppppw...tttteeeee..0..0///// Library of Science.

---

[11] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πya Sutskever. Zero-Sho Shot Text-to-Imane Image Generation. arXiv:2102. 12092 [cs], Febru- ary 2021. URL http:http:p/aeee //axivv httpp://///aiioooo//aa////////// 22022 12092. arXiv: 2102.12092.
[12] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. CogView: Mastering Text-to-Image Gen- eration via Transformers. In Advances in Neural Information Processing Systems, volume 34, pages 19822-19835, Curran Associates, Inc., 2021. URL U wtw.reeeeennnnnnnnniieeeeee neurips. html.
[13] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors, March 2022. URL httt::tttt//itit org/ haagg//b///// abs/2203.... 13131. arXiv:2203.13131 [es].
[14] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Va- sudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Mod- els for Content-Riee
Lext-to-Image Generation, June 2022. URL http://arxiv, org/ abs// 2206. 10789. arXiv:2206.1000 10789 [cs].
[15] Gaurav Bhardwaj. Jacob OConnor, Stephen Rettie, Yen-Hur Huang, Theresa A. Ramelot, Vikram Khipple Mulligan, Gizem Gokce Alpkilic, Jonathan Palmer, Asim K. Bera, Matthew J. Bick, Maddalena Piazza, Xinting Li, Parisa Hosseinzadeh, Timothy W. Craven, Roberto Tejero, Anna Lauko, Ryan Choi, Calina Glynn, Linlin Dong, Robert Griffin, Wesley C. van Voorhis, Jose Rodriguez. Lance Stewart, Gactano Montelione, David Craik, and David Baker. Accurate de novo design of membrane-traverse
macrocycles. Cell, 185(19):3500- 3532.e26, September 2022. ISSN 00928674, doi: (el.l///..ell.2222222... //linkinghub. elsevier /
URL https:
[16] Jing Yang (John) Wang, Alena Khmelinskaia, William Sheffler, Marcos C. Miranda, Aleksan- dar Antanasijevic, Andrew J. Borst, Susana V. Torres, Chelsea Shu, Yang Hsia, Una Natter- mann, Daniel Ellis, Carl Walkey, Maggie Ahlrichs, Sidney Chan, Alex Kang, Hannah Nguyen, Claire Sydeman, Banumathi Sankaran, Mengyu Wu, Asim K. Bera, Lauren Carter, Brooke Fiala, Michael Murphy, David Baker, Andrew B. Ward, and Neil P. King. Improving the secre- tion of designed protein assemblies through negative design of cryptic transmembrane domains. Proceedings of the National Academy of Sciences, 120( 2222124556611 20, March 2023. doi: 10. 1073/pmas. 00173ppas.221455666 2212222124 URL https: /www. pnas, prgg/dii/10 wt/.////...///// 10700007aa 2214556120. Publisher: Proceedings the National Academy of Sciences.
[17] Yun Guo, Yang Yang, Yan Huang, and Hong-Bit -in Shen. Discovering nuclear targeting signal sequence through protein language learning and multivariate analysis. Analytical Biochemistry, 191:1, February 2020. ISSN 0003-2697, doi: 10. 10.10166jiab.219..113.
URL https:
//www. aciencedirect..c..cc.c....cci... com/ wttssccinccceeeeeeeeee/eeeecccee
[18] Nathan H. Cho, Keith C. Cheveralls, -nvie Brunner, Kibeom Kim, André C. Michaelis, Preethi Raghavan, Hirofumi
Hirofuum Kobayashi, Laura Savy, Jason Y. Li, Hera Canaj, James Y. S. Kim, Edna M. Stewart, Christian Gnann, Frank McCarthy, Joana P. Cabrera, Rachel M. Brunetti, Bryant B. Chhun, Greg Dingle, Marco Y. Hein, Bo Huang, Shalin B. Mehta, Jonathan S. Weissman, Rafael Gömez-Sjöberg, Daniel N. Itzhak, Loic A. Royer, Matthias Mann, and Manuel D. Leonetti. OpenCell: proteome-scale endogenous tagging enables the cartography of human cellular organization. Technical report, March 2021. URI. https: / //www. wtwp:.birree... biorxiv. org/ www//ennnnnno.c//oo/oo 10t.tten/10.1101//0000 1101/2021 00. 29. 433440v1. Company: Cold Spring
Harbor Laboratory Distributor: Cold Spring Harbor Laboratory Label: Cold Spring Harbor Laboratory Section: New Results Type: article.
[19] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Trans- formers. pages 12104-12113, 2022. URL https://openaccess thecvf. com/ content/ CVPR2022/htm1 /haii Scaling............... Vision_Treeiieeeeeeeeee Transformers SttetieerrssccR___ @UPP__002__P______2______ _aapl__ppe.... html.
[20] Anthony T. Annunziato.
DNA Packaging: Nucleosomes and Chromatin DNA Packaging: Nucelosomes and Chro-
Learn Science at Scitable. matin, 1(1):26, 2008.
URL http://wwee nww.re. htt.tuure.ooo/cccttleee//t///o/e
12

---

dna maa-packaging maccaagingg--loossssseee and chromatin- cleosomes and Chromatin Eg_level: MED topic: DNA Packaging: Nucleosomes and Chromatin.
Cg cat: DNA Packaging: Nu- Cg_cat:
[21] Peter J. Thul and Cecilia Lindskog. The human protein atlas: A spatial map of the human proteome. Protein Science: Publication of the Protein Sociery, (7(1):233-244 January 2018. ISSN 1469-896X. doi: 10. 10.1002/pto.3307.
[22] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S. Song. Evaluating Protein Transfer Learning with TAPE. Advances in Neural Information Processing Systems, 32:9689 December 2019. ISSN 1049-5258.
[23] Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy. Charlotte Rochereau, and Burkhard Rost. Ankh: Optimized Protein Language Model Unlocks General-Purpon Modelling, January 2023. URL http://arxiv, org/abs hts/abb/23011. 2301 03801.6568.. 06568. arXiv:2301.0663 1.65668..6888 [es, q-bio].
[24] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs. Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Toward Understanding the Language of Life Through Self- Supervised Learning. IEEE transactions on pattern analysis and machine intelligence, 44(10): 7112-7127, October 2022. ISSN 1939-3539. doi: 10. 109/T00MMI 2021.309538..
[25] Ali Madani, Ben Krause, Eric R. Greene, Subu Subramanian, Benjamin P. Mohr, James M. Holton, Jose Luis Olmos, Caiming Xiong. Zachary Z. Sun, Richard Socher, James S. Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, pages 1-8, January 2023. ISSN 1546- URL https: hopps://wwrr nature. com/articles/ 01187-022-0111220 2. Publisher: Nature Publishing Group.
1696. doi: 10. 00088/415577022-0161111
[26] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie. Zhongkai Zhu, Wenting Lu. Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmucli, Allan dos Santos Costa, Maryam Pazel-Zarandi Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic level protein structure with a language model, December 2022. URL https / /www. biorxiv. arg/content/10. 102/2022..2.2222222 07 20. 500902v3. Pages: 2022.07.222 20.2077.00.0090 Section: New Results.
[27] Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, Jianzhu Ma, and Jian Peng. High-resolution <em>de novo</em> structure prediction from primary sequence, January 2022. URL http: 000/22/002/222 /2//22222222222 07 $1..90099..... abstract.
[28] Robert Verkuil, Ori Kabeli, Yilun Du, Basile I. M. Wicky, Lukas F. Milles, Justas Dauparas, David Baker, Sergey Ovchinnikov, Tom Sereu, and Alexander Rives. Language models gener- alize beyond natural proteins, December 2022. URL https: / /tt////ww.// biorxiv. .rgg ctt///on/////////////
10. 1001/2222.2.... 12.21 521521v1.
Pages: 2022. 12.21. 2521521 Section: New Results.
[29] Henrik Nielsen, Konstantinos D. Tsirigos, Søren Brunak, and Gunnar von Heijne. Brief History of Protein Sorting Prediction. The Protein Journal, 38(3):200-216, June 2019. ISSN 1875-8355.
doi: 10.10077/110930-099-0883
URL
110930-019-09838
[30] Katelyn C Cook and lleana M Cristea. Location is everything: protein translocations as a viral infection strategy. Current Opinion in Chemical Biology, 48:34 February 2019. ISSN 1367-5931. doi: 10. 10.1016/j.cbpa.2011 20t.cpppp..0..0.0021.... URL. https: //ww:///wwwww//// www.sssciinneeeec....m com/
[31] Josie Christopher, Charlotte Stadler, Claire E. Martin, Marcel Morgenstern, Yanbo Pan, Cora N. Betsinger, David G. Rattray, Diana Mahdessian, Anne-Claude Gingras, Bettina Warscheid, Janne Lehtiö, Ileana M. Cristea, Leonard Foster, Andrew Emili, and Kathryn Lilley. Subcellular proteomics. Nature Reviews Methods Primers, 1(1):1-24, April 2021. ISSN 2662-8449. doi: 000000///44580000000000000
URL https: //wwww/www///. nature. com/ articles/ seceles//44380 0eee///3386022250 00029-00099000000 -y.. Number: 1 Publisher: Nature Publishing Group.
13

---

[32] Vineet Thumuluri, José Juan AmmagrraAAmeneross.. AlexanderaiRosebberg Johansen, Henrik Nielsen, and Ole Winther. DeepLoc 2.0: multi-label subcellular localization prediction using protein language models. Nucleic Acids Research, 50(W1): W223, July 2022. ISSN 1305-1048... doi: 10. 10.00//a//gaa.278.. URL wtw..tssss//bi...gg/.o.o.ooo 10w.///////////////////// gkac278.
[33] Leyi Wei, Yijie Ding, Ran Su, Jijun Tang, and Quan Zou. Prediction of human protein subcellular localization using deep learning. Journal of Parallel and Distributed Comput- ing. 117:212-217, July 2018. ISSN 0743-7315. doi: 10.1016/j.pdcc.0077.08 001.00000........00000... URL https://wwe sciencedirect. www.eeedieeetccooccceeeeeeeeeeee http:///wwaeeettcccc////////////
[34] Bin Yu, Wenying Qiu, Cheng Chen, Anjun Ma, Jing Jiang, Hongyan Zhou, and Qin Ma. SubMito-XGBoost: predicting protein submitochondrial localization by fusing multiple fea- ture information and eXtreme gradient boosting. Bioinformatics, 36(4):1044-108,, February 2020. ISSN 1367-4803. doi: 1001099/hoiffommatss/b///// URL https://doi. org/1.. 1093/bioinforo/ee www.ooiiofrmmtics/tt////2/2.
[35] José Juan Almagro Armenteros, Casper Kaae Sønderby, Søren Kaae Sønderby, Henrik Nielsen, and Ole Winther. DeepLoc: prediction of protein subcellular localization using deep learn- ing. Bioinformatics, (3(21):3387-3399 November 2017. ISSN 1367-4803. doi: 10.1093/
bioinformaties/btx411.. URL https: /ttp::///oo....o...//o
[36] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical TextiConditional Image Generation with CLIP Latents, April 2022. URL http://arxiv. 06125. arXiv:2204.06122 [cs].
[37] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photoreal- Ad-
istic Text-to-Image Diffusion Models with Deep Language Understanding. vances in Neural Information Processing Systems, 35:36479- -3444,, December 2022. URL https://proceet wwwriingnn.nuuur..eeeeeeerrrinee wttpurrpss.cpppppppeee lemerr_flles files wtllll/ppperr/eee paaprr/022/haall c wtw.://oooeeeottttrt----ooooooee Confrecctt-oofeeeceee html.
[38] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc- Grew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models, March 2022. URL /ttt:://arxis argg/bs// 10741. 1eXiv:2112.100 [cs].
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- Resolution Image Synthesis with Latent Diffusion Models, April 2022. URL http://arxxi org/abs/2112 117221111........ arXiv:2112.1075: [cs].
[40] Jacob Devlin, Ming-We Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019. URL http: w.Xiv:1810.04800 [cs].
//arxiv. org/ a.0.1110..4805..
[41] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. CogView2: Faster and Better Text-to- Image Generation via Hierarchical Transformers, May 2022. URL http: org/ abs/ 2204. 14217. arXiv:2204.1 14217 [cs].
[42] Huiwen Chang, Han Zhang, Jarred Barber, A. J. Maschinot, Jose Lezama, Lu Jiang, Ming- Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text- Io-aee-l Generation via Masked Generative Transformers, January 2023. URL http: //apxiv.///ii///// org/ /b3/ 00704. arXiv:2301.0070. [cs].
[43] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked Generative Image Transformer, February 2022. htlhhtp://arxx org/abs/2204 2220b..200..00000. 04200. arXiv:2202.04200 [cs].
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need, December 2017. http: www.oos/gtteeeeoeeeoooooeee ab/////b/// 03762. arXiv:1706.03747 [cs].
// //arxiv.
14

---

[45] Andreas Digre and Cecilia Lindskog. The Human Protein AtlasSpatial localization of the human proteome in health and disease. Protein Science, 30(1):218 2333. 2021. ISSN 1469- 896X. doi: 10. 100000000.099/3 URL. httph:tt///lllnenlliiiteee 1002/pro. 3987. _eprint: http://oollinnbooaaaieeeeeeeiiii http:///wwwiiiiddoo..0///0000990 1002/pro.3987.
wiley. com/doi/an /atp//as///000/
[46] Ying Huang, Beifang Niu, Ying Gao, Limin Fu, and Weizhong Li. CD-HIT Suite: a web server for clustering and comparing biological sequences. Bioinformatics, 26(5):680 682, March 2010. ISSN 1367-4803. doi: 10.109//boonfoomaiii////gg// URL https://dol org/10. 1093/bioiie www.a/fttaaatttsttsaaseeeee
[47] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford. Gretchen Krueger. Jong Wook Kim, Sarah Kreps, Miles McCain, Alex New- house, Jason Blazakis, Kris McGuffie, and Jasmine Wang. Release Strategies and the Social Impacts of Language Models, November 2019. URL UtLpttp://arxt org/ has//bbs/19...../ 09203. MxXiv:1908.09200 [es].
[48] IUPAC-IUB Commission on Biochemical Nomenclature A One-Letter Notation for Amino Acid Sequences13. Journal of Biological Chemistry, 243((1);3557-3559, July 1968, ISSN 0021-9258. doi: 10. 11A1////002112588119934488
URL https://dot otttp:///do..rrg/1.00006
90021-9258 34176-6. 6. Publisher: Elsevier.
[49] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for High-Resolution Image Synthesis. pages 12873-12883, 2021. URL stttps://openaccess thecvf com/ Cattent/OVPP2222//tm///eeBsse http://ttmmm/eeeeeraaaiannnnn Transformers for Reiih-RRssoltiin, Image_ SetpeesssscvRRPPe_
html http///eeettttse/g/tttuooaaa
com.
[50] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL https: https:// http://ppoeeenn...///////o seroceedings.neurips.. www.ees..ccppapr/////..... cc/paperr201//hahh 1ca29911261118880099ccccbaaaaaae
Abstract. .tmll.
[51] M. Cokol, R. Nair, and B. Rost. Finding nuclear localization signals. EMBO reports, 1(5): 411-415, November 2000. ISSN 1469-221X.
doi: 10. 10.1033/embo--epporss/vvdd/
[52] Paul Horton, Keun-Joon Park, Takeshi Obayashi. Naoya Fujita, Hajime Harada. C.J. Adams- Collier, and Kenta Nakai. WoLF PSORT: protein localization predictor. Nucleic Acids Re- search, 35(supp 2):5555-5587 July 2007. ISSN 0305-1048. doi: (0.1993/na//kkm255,
http: org 10. www11/////////// /05//////.0///////mm/25.....
[53] Michelle Scott, François-Michel Boisvert, Mark D. McDowall, Angus I. Lamond, and Geoffrey J. Barton. Characterization and prediction of protein nucleolar localization se- quences. Nucleic Acids Research, 38(21):7388-77 -3999, November 2010. ISSN 0305-1048. doi: 10.1093/ URL battps://dol org/ 10. 1093/n0r/// 2a00///aa/gg/65.... gkq653.
[54] Yin- Yuan Mo, Chengyi Wang, and William T. Beck. A Novel Nuclear Localization Sig- nal in Human DNA Topoisomerase 1 Journal of Biological Chemistry, 275(52):41107 107- 41113, December 2000. ISSN 0021-9258. doi: 10.10774/bcc.0003135000. URL https: / www..sciencedirec.. com/ wttssccinceeeeeeeeeeeeeeeeeeeees wwwpeeeeeeeaeeeeeeeeeeeeeeeeeeee arr.iiceeppi//002122551555555551
[55] Allison Lange, Ryan Mills, Christopher J. Lange, Murray Stewart, Scott E. Devine, and Anita H. Corbett. Classical Nuclear Localization Signals: Definition, Function, and Interac- tion with Importin * Journal of Biological Chemistry, 282(8)55101-510) February 2007, ISSN 0021-9258, 1083-351X. doi: 10. article/so021-900 (20) www00-///0attrreeeeeee
URL https: :/wp:///wwwwww/ jbc. org/ Publisher: Elsevier.
[56] Ulrike Schnell, Freark Dijk, Klaas A. Sjollema, and Ben Giepmans. Immunolabeling artifacts and the need for live-cell imaging. Nature Methods, 922):152-158, February 2012. ISSN 1548-7105. doi: 10. 1038/me/m.., 1855. URL https: :// nature. com/articles/ nmeth. 1855. Number: Publisher: Nature Publishing Group.
15

---

[57] Christian von Mering, Roland Krause, Berend Snel, Michael Cornell, Stephen G. Oliver. Stan- ley Fields, and Peer Bork. Comparative assessment of large-scale data sets proteinprotein in- teractions. Nature, 41 417(6887) 403, May 2002. ISSN 1476-4687. doi: 10. 10.1038/nature750
URL https:::::// //www. nature. www.//aaticlessattuuuueeeee naturee00. Number: 6887 Publisher: Nature Publishing Group.
[58] Justin Pinkney. How to fine tune stable diffusion: how made the text-to-pokemon model at Lambda, September 2022. URL
com/blog/
[59] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An Image is Worth One Word: Personalizing Text-to-lmage Gener- ation using Textual Inversion, August 2022. URL http://aaxixii www.oogg////////ooooo/ooooeeee/e abs. arXiv:2208.011 016618 [cs].
[60] Natamiel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. DreamBooth: Fine Tuning Text-to-lmage Diffusion Models for Subject-Driven Genera- tion, March 2023. URL httt:://aaxxaae www./ogg///////o//o//..o/..e abs/////..////// 12242. arXiv:2208. 12242 [es].
[61] Pierre Chambon, Christian Bluethgen, Curtis P. Langlotz, and Akshay Chaudhari. Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains, October 2022. [cs].
URL http: /ttp/v/wvvvv/v////////e.///aaa httpg//bbs/221110 04133.
[62] R. H. Köhler, W. R. Zipfel, W. W. Webb, and M. R. Hanson. The green fluorescent protein as marker to visualize plant mitochondria in vivo. The Plant Journal: For Cell and Molecu- lar Biology, 113):613-621, March 1997. ISSN 0960-7412. 11030613.x.
doi: 10. 13466).136--111x11199
[63] Nicole Maria Seibel, Jihane Eljouni, Marcus Michael Nalaskowski, and Wolfgang Hampe. Nu- clear localization of enhanced green fluorescent protein homomultimers. Analytical Biochem- istry, 368(1):))))) 11::95999, September 2007. ISSN 0003-2697. doi: 10. 20.7.05555.02 URL https: //wps: www. sciencedirect. wtt.cccoccccc.c wwwps/ii/w../////ttttcct..tttcce http:///ecc.cceeeeeeeeaea/aee//a
[64] Akira Kitamura, Yusaku Nakayama, and Masataka Kinjo. Efficient and dynamic nuclear local- ization of green fluorescent protein via RNA binding, Biochemical and Biophysical Research Communications, 463(3):401-400) July 2015. ISSN 1090-2104. doi: 10.1016/..brr..015..oo 084.
[65] Michael Bernhofer, Tatyana Goldberg, Silvana Wolf, Mohamed Ahmed, Julian Zaugg. Mikael Boden, and Burkhard Rost. NLSdb-major update for database of nuclear localization signals and nuclear export signals. Nucleic Acids Research, 46(D1): January 2018. ISSN 1362-4962. doi: 10. 1..109/////gxx102/
[66] Marius Pachitariu and Carsen Stringer. Cellpose 2.0: how to train your own
model. Nature Methods, 19(12):1634-144) December 2022. ISSN 1548-7105 doi: 00088/441592-022-11666
URL https: //www. nature. com/articles/ Number: 12 Publisher: Nature Publishing Group.
111592-022-01663-11
[67] Alex N. Nguyen Ba, Anastassia Pogoutse, Nicholas Provart, and Alan M. Moses. NL- Stradamus: a simple Hidden Markov Model for nuclear localization signal prediction. BMC Bioinformatics, 10(1):202, June 2009. ISSN 1471-2105. doi: 10.1186/1771-2105-00-222 URL https: org/ 10. 1ww.1111//11111111 211111141122011-002000
[68] Juane Lu, Tao Wu, Biao Zhang, Suke Liu, Wenjun Song. Jianjun Qiao, and Haihua Ruan. Types of nuclear localization signals and mechanisms of protein import into the nucleus. Cell Communication and Signaling, 19(1):60, May 2021. ISSN 1478-811X. doi: 10.1186/ s $12964-021-007441- URL htwph:ttt///eeett //ooi. org/ 11///11.11..11111 111111111//6444 -22-2664-222-0071---
[69] J. Bradley, M. R. Bowl, S. E. Williams, B. N. Ahmad, C. J. Partridge, A. L. Patmanidi, A. M. Kennedy, N. Y. Loh, and R. Thakker. Parafibromin is a nuclear protein with functional monopartite nuclear localization signal. Oncogene, 26(8):1213-122) February 2007. ISSN 0950-9232, doi: 10. 1038////.o.c... 1208993.
16


Reviews:
[{"review_id": "q57z8Rd1aj", "reviewer": "Reviewer_ikCN", "summary": "The authors propose a new architecture, CellBERT-E, for producing flexible embeddings that encode combinations of protein amino acid sequences and protein localization images. It can be used to generate localization images given a sequence and vice versa. Compared to its predecessor, it has many favorable characteristics; it was trained on slightly more data, is faster, and is bidirectional. It performs well on benchmarks.", "questions": "- Why do you clip logit values instead of softmaxing them (Supplement section B.1)?\n- Varying depth along with width in Table 1 is a little awkward — it would be good to control for the effect of depth vs width. How do the parameter counts of the different models compare?\n- Why are the HPA FID scores so erratic? E.g. it seems like HPA_640 achieves good scores in almost all categories but severely underperforms the other models in terms of FID.\n-     “We also visually inspected some of the generated protein images (Fig. S6, Fig. S7). The output images from the OpenCell models appeared realistic and consistent with the ground truth labels, but they had low entropy in the predicted distribution. This suggests that the models learned to assign high probability to correct tokens, but failed to capture the uncertainty and variability of other valid selections. This could be attributed to the rapid overfitting of the OpenCell models, which limited their generalization ability.”\n\n    Do the datasets contain the kind of diversity you allude to here (e.g. multiple protein images per sequence)? Is there any reason to expect the models to learn wide distributions over tokens?\n-     \"Most models had low performance on this task in terms of reconstruction. This is understandable because the models learned to generate amino acids that were common or frequent in the dataset, but not necessarily correct for the specific sequence.\"\n\n    This is surprising to me, especially in light of Table S4. Why would ESM + image be so much worse than ESM alone? What exactly was the experimental setup for Table S4.\n\n", "limitations": "The authors include a thorough discussion of limitations. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is exceptionally well-written and the method clearly improves on its predecessor, CELL-E, in multiple ways. Evaluation is thorough and carefully analyzed. The architecture is well-suited to the task and thoughtfully constructed.", "weaknesses": "My only real criticism is that the subject matter is quite niche (even to the point where some of the significance of this is lost on me), and I suspect that it will not interest most NeurIPS readers per se. That being said, I think the three-way multimodal architecture is well-executed and potentially worth acceptance as a nice case study in its own right.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">My only real criticism is that the subject matter is quite niche (even to the point where some of the significance of this is lost on me), and I suspect that it will not interest most NeurIPS readers per se. That being said, I think the three-way multimodal architecture is well-executed and potentially worth acceptance as a nice case study in its own right.\n\nWe appreciate the recognition of our three-way multimodal architecture as a nice case study. We also acknowledge that the subject matter may seem niche to some NeurIPS readers, but we would like to emphasize its relevance and significance to the broader machine learning community. \n\nFirst, our work aligns with the NeurIPS goal of fostering submissions for “Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences)”, which we believe is an important and growing area of interest. Second, our work has a high potential impact on the field of biology, especially for studying biological pathways and protein engineering. By enabling the in silico replication of experiments, our method can drastically reduce the experimental time and cost, as well as increase the number of possible targets to screen. Third, our work showcases a novel application of multimodal learning that integrates different types of data in a principled way. We believe that this approach can inspire other researchers to explore similar problems that involve multiple modalities and complex relationships. \n\n> Why do you clip logit values instead of softmaxing them (Supplement section B.1)? \n\nWe would like to clarify that we do apply softmax to the logit values in our model, as described in Supplement section B.1. The clipping operation that we mention in the same section is not applied to the logits, but to the pixel values in the output heatmap. This is a post-processing step that we use to handle out-of-range values that may occur from the latent space interpolation within the VQGAN. We apologize for any confusion that this may have caused, and we will make this more clear in the manuscript. \n\n>Varying depth along with width in Table 1 is a little awkward — it would be good to control for the effect of depth vs width. How do the parameter counts of the different models compare? \n\nCELL-E [1] demonstrated that transformer depth and predictive performance have a strong positive correlation. The embedding size is pre-determined by the embedding dimensions used in the ESM-2 language model. We will include the rationale in Section B.1.  To accommodate our available compute hardware, we maximized the depth which could be fit in memory during training. We will include also include a table of parameter counts in the supplemental.\n\n>Why are the HPA FID scores so erratic? E.g. it seems like HPA_640 achieves good scores in almost all categories but severely underperforms the other models in terms of FID. \n\nThe HPA dataset contains significant diversity in terms of cell type, image resolution, and antibody staining, which may cause inconsistencies in the model learnings. We have reported FID because it is a standard measure in text-to-image studies for natural images, but further investigation is needed to understand the behavior of FID in response to data in this domain. There is already ongoing debate on whether FID and IS truly correlate with image quality, as FID is not a stable metric. Recent work [2] has show that changing the jpeg quality from 100 to 75 can cause a difference of up to 20 FID points. \n\n> Do the datasets contain the kind of diversity you allude to here (e.g. multiple protein images per sequence)? Is there any reason to expect the models to learn wide distributions over tokens? \n\n> ... \n\n>  This is surprising to me, especially in light of Table S4. Why would ESM + image be so much worse than ESM alone? What exactly was the experimental setup for Table S4. \n\nThe training objective required the model to reconstruct both image and text tokens. However, the HPA dataset, which is the largest available protein image dataset, containing just 17,268 proteins. This number pales in comparison to the 617M used in training ESM. We expect that performance would improve if model loss were only calculated on the masked amino acids. \n\n\n\n[1] Emaad Khwaja, Yun S. Song, and Bo Huang. CELL-E: Biological Zero-Shot Text-to-Image Synthesis for Protein Localization Prediction, May 2022. Pages: 2022.05.27.493774 Section: New Results. \n\n[2] Gaurav Parmar, Richard Zhang, Jun-Yan Zhu; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 11410-11420\n\n"}, {"review_id": "paWZc2SR5v", "reviewer": "Reviewer_9W6q", "summary": "This paper proposes a novel bidirectional transformer named CellBERT-E to generate accurate protein localization image prediction from the amino acid sequences. To solve the ignorance of the integration of sequence and image information in existing methods, CellBERT-E adopts a BERT-like architecture so that the model can generate both image and sequence predictions in a non-autoregressive (NAR) paradigm at a fast speed. Therefore, the model allows for bidirectional prediction, making the model a possible candidate for de novo protein design. The model is trained by reconstructing the masked tokens in both the amino acid sequences and images in an unsupervised manner. Benefiting from the pretraining on a large HPA dataset and finetuning on the OpenCell dataset, CellBERT-E achieves competitive or superior performance compared with SOTA methods, which is shown by extensive experiment results. \n", "questions": "1. What's the function of nucleus images over here? According to the authors'  explanation, the nucleus images are passed to the encoder but their tokens are not masked and thus not reconstructed during bidirectional prediction. Can the authors specify what's the role of nucleus images in this case (if there are any biological backgrounds related please clarify briefly)?\n\n2. In Section 7, the sentence \"By pre-training on a large 310 HPA dataset and fine-tuning on CELL-E, ...\" should be changed by replacing CELL-E with OpenCell. \n\n3. Although it is easier to parallel Transformer encoder-based models during inference, NAR Transformer decoders do exist [1, 2] and can significantly accelerate the decoding speed. Considering the recent success of GPT-based models, it would be great if the authors can discuss a little bit on whether the encoder-only CellBERT-E can be modified to a decoder-based one.\n\n4. The goal of this paper is to train a model for sequence-to-image and image-to-sequence generation, so I'm wondering whether it's possible to train an encoder-decoder model (and cross attention might be more helpful in explaining how the amino acid sequences and threshold images are related), and maybe the authors can explain a little bit how they determine the model architecture.\n\n5. With respect to the finetuning stage, I'm wondering what's the task here, is it the same as the training procedure described in Fig. 2? Why not mask all the tokens in the threshold figure and unmask the sequences (similar to the illustration in Fig. S3), which is closer to what the model is trained for: generate images depicting protein subcellular localization from the amino acid sequences? Probably, such a gap will make the model performs worse in the real application settings.\n\n6. For the different finetuning strategies mentioned in section 5.3, I'm wondering whether the authors have tried finetuning the whole model simultaneously and how the performance is compared with other finetuned models.\n\n[1] Gu, J., Bradbury, J., Xiong, C., Li, V. O., & Socher, R. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281.\n\n[2] Huang, F., Tao, T., Zhou, H., Li, L., & Huang, M. On the learning of non-autoregressive transformers. In International Conference on Machine Learning (2022).", "limitations": "The authors have partially addressed the limitations of their work, though there is space for improvement (see Strengths, Weaknesses, and Questions).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThe paper proposes bidirectional transformer for text-to-image translation, and explores how this model could be used for protein design. Therefore, the paper uniquely contributes to the field by making fast and accurate predictions for protein sequence or image generation in the non-autoregressive manner.\n\nQuality:\nThe paper carefully designs the experiments to support the idea and make clear visualizations.\n\nClarity:\nThe paper effectively communicates its ideas and findings with clarity. The paper is well-written, and the logic is coherent. The authors clearly illustrate the details in each section and make the ideas transparent to readers. \n\nSignificance:\nThe paper focuses on the bidirectional prediction of amino acid sequences and protein localization images with the advantage of faster prediction speed and possibly better protein de novo design than models of auto-regressive manner. And the proposed model does outperform baseline models. Therefore, this work is a promising model for protein design and engineering and could inspire research on bidirectional NAR models for this domain. ", "weaknesses": "1. More background knowledge on biological terms mentioned in the paper is required (or explained more clearly), e.g., what are nucleus images. Besides, the protein images shown in the manuscript and supporting materials are nice, but it's not easy for a non-expert in biology to interpret something useful from the figures. \n\n2. Although the authors' presentation is quite clear in general, the details of the finetuning task provided in the paper are not enough. Besides, it would be better if the authors can provide any intuition on why the task is designed in this way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">More background knowledge on biological terms mentioned in the paper is required (or explained more clearly), e.g., what are nucleus images. \n\nThank you for the feedback, we have updated the language in the introduction and methods to more to clearly explain those terms. \n\n> Although the authors' presentation is quite clear in general, the details of the finetuning task provided in the paper are not enough. \n\nWe adopt a selective finetuning strategy that involves different parts of the model (e.g. the image representation via the VQGAN encoders) at different stages. This is motivated by the idea of domain adaptation, which aims to improve the performance of a model on a new domain by leveraging the knowledge learned from a different domain. We will update the manuscript to make this more clear.\n\n>Besides, it would be better if the authors can provide any intuition on why the task is designed in this way. \n\nWe design the task to parallel the cell-imaging workflow, while also offering additional insights. In experiments, biologists stain the protein of interest as well as nucleus as a spatial reference. Therefore, we use the nucleus as a fixed input. Multiple acquisitions are often needed over time to understand protein dynamics. We demonstrate how our model can generate images quickly *in silico*, thus increasing the potential throughput and efficiency of such studies. \n\n> Can the authors specify what's the role of nucleus images in this case (if there are any biological backgrounds related please clarify briefly)? \n\nThe nucleus image serves as a conditional image with which the model makes a prediction of protein localization with respect to. A predicted localization image holds little informational value without a spatial reference with which to compare to. Nucleus images are obtained for reference in wet-lab imaging workflows as standard practice. We have updated the manuscript to include this information in Section 4.2. \n\n>In Section 7, the sentence \"By pre-training on a large 310 HPA dataset and fine-tuning on CELL-E, ...\" should be changed by replacing CELL-E with OpenCell.  \n\nThank you, this will be changed. \n\n> Although it is easier to parallel Transformer encoder-based models during inference, NAR Transformer decoders do exist [1, 2] and can significantly accelerate the decoding speed. Considering the recent success of GPT-based models, it would be great if the authors can discuss a little bit on whether the encoder-only CellBERT-E can be modified to a decoder-based one. ...\n\nWe have not tried the decoder-based model, but it is an interesting research direction. We use an encoder-only model to leverage the parallelism and independence of the patches, which are crucial for protein image synthesis. A decoder-based model would introduce dependencies and sequentiality among the patches, which may affect the heatmap quality. We are not aware of any work that uses NAR Transformer decoders for image generation.\n\n> ..I'm wondering whether it's possible to train an encoder-decoder model ... and maybe the authors can explain a little bit how they determine the model architecture. \n\nOur goal is to enable in silico screening of protein candidates, which requires a fast and scalable model (enabled by an encoder-based NAR model [1]) that can generate high-quality protein images from sequences and vice versa. An encoder-decoder model with cross-attention would introduce additional computational complexity and latency, which may hinder the practical application of our method. It is an interesting path to explore though.\n\n> With respect to the finetuning stage, I'm wondering what's the task here, is it the same as the training procedure described in Fig. 2? \n\nWe pre-train our model on HPA data, which is diverse and heterogeneous. We finetune it on OpenCell data, which is more realistic and consistent for protein image synthesis, but cointains a specific human cell line. This allows the model to adpt to the domain.\n\n> Why not mask all the tokens in the threshold figure and unmask the sequences (similar to the illustration in Fig. S3), which is closer to what the model is trained for: generate images depicting protein subcellular localization from the amino acid sequences?...\n\nWe do not mask all the tokens in the threshold image and unmask the sequences, because we want our model to be able to generate both modalities from each other. Our goal is not only to produce protein images from sequences, but also to produce sequences from images. This is useful for applications such as protein annotation, identification, and analysis. Therefore, we mask both image and text tokens during finetuning, as we did during pre-training, to train a bidirectional model that can handle both sequence-to-image and image-to-sequence generation. \n\n> ... I'm wondering whether the authors have tried finetuning the whole model simultaneously and how the performance is compared with other finetuned models. \n\nWe have not tried finetuning the whole model simultaneously, because we follow the common practice of using a fixed image codebook in text-to-image models [1-3]. A fixed image codebook allows us to leverage the pre-trained image features and reduce the computational cost of finetuning. We can train on consumer-grade hardware because freezing parts of the model requires less VRAM, which is limited if our model is fully finetuned.\n\n[1] Huiwen Chang, Han Zhang, Jarred Barber, A. J. Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-To-Image Generation via Masked Generative Transformers, January 5 2023. \n\n[2] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers, May 2022. \n\n[3] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors, March 2022. \n\n "}, {"review_id": "yvcj9KGyHL", "reviewer": "Reviewer_pNFc", "summary": "This work proposes an image-sequence multimodal encoder to model the interdependencies between cellular image and protein sequence. The pre-trained ESM-2 protein language model is employed to extract protein sequence embeddings, and the pre-trained VQGAN is used to extract cellular image patch embeddings. A Transformer encoder is trained upon these two kinds of embeddings to model the interaction between image patches and amino acid residues. The Validation set performance of image prediction and sequence infilling are analyzed to demonstrate model design choices. The application on de novo NLS design shows the effectiveness of the proposed model. ", "questions": "1. The proposed method learns image-enhanced representations of protein sequences. Such representation could be superior over pure protein sequence representations learned by ESM-2. The subcellular localization prediction benchmarks proposed by DeepLoc [a] and DeepLoc 2.0 [b] could be a good test field for such a hypothesis, where the proposed CellBERT-E can compare with various protein language models.\n2. In the image prediction and sequence infilling experiments, authors report the performance on validation set. However, from a standard machine learning perspective, the validation set should be used for model selection, and another hold out test set serves for evaluation. Authors are strongly encouraged to align such a standard. \n\n[a] Almagro Armenteros, José Juan, et al. \"DeepLoc: prediction of protein subcellular localization using deep learning.\" Bioinformatics 33.21 (2017): 3387-3395.\n\n[b] Thumuluri, Vineet, et al. \"DeepLoc 2.0: multi-label subcellular localization prediction using protein language models.\" Nucleic acids research 50.W1 (2022): W228-W234.\n", "limitations": "In the conclusion section, authors have not sufficiently discussed the limitations of their current method. They are encouraged to discuss potential limitations in terms of effectiveness, efficiency and the scope of applications. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The proposed multimodality learning framework of cellular images and protein sequences is technically sound, and such a multimodality learning setting is novel to my best knowledge.\n+ The results on de novo NLS design shows the model could helpful in real-world applications. \n", "weaknesses": "- Important downstream applications and baseline methods are not investigated in the experiment section. \n\n- The evaluation protocol of image prediction and sequence infilling is not that standard from the machine learning perspective. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Important downstream applications and baseline methods are not investigated in the experiment section.   \n  \nWe included a highly tangible downstream application in the discussion. There, we demonstrated the generation of new NLS sequences using in-filling. In the protein engineering space, identifying candidate sequences is an important first step. Our demonstration showed the capability for CellBERT-E to generate new sequences driving the engineered protein to the desired subcellular localization. We have additionally passed our generated NLS candidate sequences (appended to GFP) to DeepLoc 2.0 [1], which is an independent model predicting multiclass annotations of protein localizations from their sequences. DeepLoc 2.0 predicted **89%** of the generated NLS candidates with nuclear localization and **91%** having potential nuclear localizing signals, clearly validating our model. We hope to experimentally validate these sequences.  \n  \nWith respect to baseline methods, we note that this is a new field of study lacking baseline methods to compare against. For image prediction, the only available comparison is CELL-E. We will add an explicit, direct comparison in the revised paper. For sequence infilling, our model is the first of its kind with this capability. Therefore, we hope that our CellBERT-E itself will serve as the baseline for future developments in this field.   \n  \n> The evaluation protocol of image prediction and sequence infilling is not that standard from the machine learning perspective.  \n  \nWe have tried our best to match the standard practices. However, the lack of established baselines and the absence of suitable image-based benchmarks complicates the having a comprehensive assessment. Therefore, we designed our evaluation protocol considering practical limitations and needs within the protein engineering domain.   \n  \n> ...The subcellular localization prediction benchmarks proposed by DeepLoc [a] and DeepLoc 2.0 [b] could be a good test field for such a hypothesis, where the proposed CellBERT-E can compare with various protein language models.  \n  \nCellBERT-E was not intended to be a potentially superior protein sequence representation. Instead, it was developed as a tool to translate between the amino acid sequence of a protein and its functional properties, akin in spirit to AlphaFold and other protein structure prediction models. Sequence embedding is just a part of CellBERT-E model.   \n  \nWe thank the reviewer for bringing up the idea that CellBERT-E could lead to a better protein sequence representation that is “image-enhanced”. If so, the protein representation in CellBERT-E could indeed outperform purely sequence-trained ones for certain tasks, such as localization prediction in discrete categories like in DeepLoc [2] and DeepLoc 2.0 [1]. For this paper, we focus on the image prediction and sequence infilling capabilities of CellBERT-E, which has important downstream applications in protein engineering. We hope that future studies will elucidate the potential of CellBERT-E protein representation in other applications.   \n  \nAdditionally, for benchmarking the image prediction of CellBERT-E, we previously considered comparing the indicated localizations in the predicted images to the multiclass predictions of DeepLoc 2.0 [2]. However, such a comparison requires conversion between two distinct modalities, i.e., annotate the protein localization in the predicted images using a separate image segmentation model. Therefore, we were afraid that this comparison could be inconclusive and would not serve as a proper benchmark. Still, if the reviewer feels that such comparison is necessary, we will include it in the revision.    \n  \n>  ... the validation set should be used for model selection, and another hold out test set serves for evaluation... \n  \nWe agree that a train-validation-test split is a standard practice in machine learning, but we argue that it is not feasible or necessary for our task. Protein datasets are very scarce and expensive to obtain, as they involve complex and costly wet-lab experiments. We use the HPA dataset, which is the largest available dataset of protein images, but still contains only 17,268 proteins. (Small compared to the 617M proteins used in training the sequence model ESM-2). Splitting the HPA dataset into three subsets would reduce the amount of data for training and validation, and it may not reflect the true performance of our model on unseen data.  \n  \nMoreover, we use another dataset, the OpenCell dataset, for finetuning and evaluation. The OpenCell dataset contains live imaging data of a specific human cell line, which is more realistic and consistent than the HPA dataset. The OpenCell dataset contains only ~1000 proteins. Splitting this dataset into three subsets would further limit the data availability and diversity for our task. Therefore, we follow the precedent of a recent work [3] that uses the same OpenCell dataset and adopts a two-way split. We believe that this is a reasonable and practical choice for our task, given the data constraints and challenges.  \n  \n> In the conclusion section, authors have not sufficiently discussed the limitations of their current method....\n  \nIn Section 7, we will be including a longer discussion on limitations, including real-world implications of the limited resolution and potential ways to address them in future works. We also include a discussion on the trade-off between prediction quality and speed in the sequence prediction stage.  \n  \n[1] Thumuluri, Vineet, et al. \"DeepLoc 2.0: multi-label subcellular localization prediction using protein language models.\" Nucleic acids research. (2022): \n  \n[2] Almagro Armenteros, José Juan, et al. \"DeepLoc: prediction of protein subcellular localization using deep learning.\" Bioinformatics (2017):\n  \n[3] Kobayashi, H., et al. Self-supervised deep learning encodes high-resolution features of protein subcellular localization. Nat Methods (2022).  "}, {"review_id": "g9i8ybbgGm", "reviewer": "Reviewer_DiCT", "summary": "In this work, the authors present CellBERT-E, a transformer-based model to generate protein localization images. The model takes in the nucleus and threshold images as well as amino acid (AA) sequences as input. The images are tokenized via VQGAN tokenizer and AA sequences are tokenized via pretrained protein language model. The model is pretrained via mask modeling for both AA and image tokens. Experiments show that CellBERT-E achieves reasonable results in protein localization prediction. The authors also include experiments concerning generating AA motifs from image input. Besides, CellBERT-E generates images in a non-autoregressive manner, which is more efficient than previous CELL-E model.", "questions": "Major questions:\n1. Major results in Table 1&2 only compare different settings of proposed CellBERT-E and miss the performance of other baseline models. Though this work focuses on a relatively new application that few works have investigated. The authors can at least include the comparison to CELL-E which the proposed method is based on. \n2. The choice of hyperparameters needs more validation. In Table 1&2, the deeper the model is, the smaller the hidden size is. Why do the authors choose such settings in architecture exploration? Besides, it's hard to conclude the effect of hidden size and depth on the performance as they are changing together. \n3. Table 1 reports several image metrics to show the performance of CellBERT-E. Are there other domain metrics that are important in protein localization prediction? Will these image metrics fail to capture such important details?\n4. The authors mention the potential rapid overfitting on OpenCell containing limited data. Have the authors considered any data augmentation strategies to enlarge the training set?\n5. In Table 2, all models achieve high cosine similarities. Is this because only 15% of AA sequence is masked? The authors can add the results of random sampling to better validate the effectiveness of proposed model.\n6. In Table 3, models that perform well on FID are likely to perform poorly on the other 5 metrics. What could be the reason?\n7. In Section 6.2, the authors use the predicted signal in nucleus from the image model to validate the generated NLS. I find it unconvincing since the image model trained on limited data can fail when new data is fed in. \n\nMinor questions:\n1. Typo in line 88, I suppose it should be \"allowing images to be synthesized in relatively few steps\".\n", "limitations": "Besides the structural information of proteins mentioned by the author. The limited training data can be another limitation of the work. Even HPA, the larger dataset in this work, contains only ~17,000 data which is noisy. The authors bring up this in Introduction. However, how the proposed method addresses the concern should be further clarified. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Application of the transformer-based generative model to protein localization prediction, a relatively new domain. \n2. Leveraging non-autoregressive generation which is more efficient than previous work.\n3. Application of the proposed model to generate short AA sequences from nucleus images. Such a setting is tested on NLS design which can be an interesting attempt. ", "weaknesses": "1. Lack of baseline models. The authors only compare different settings of the proposed CellBERT-E but miss other baselines. \n2. Experiments may not support the conclusion of the proposed method. It can be hard to tell the effect of hyperparameters based on current experimental settings. \n3. Evaluation metrics may not fully reflect the performance on the application. Authors apply common metrics for image/sequence generation to evaluate the protein localization prediction performance. Domain details may be missed when only applying these metrics. \n \nPlease see \"Questions\" for more details. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback, we will address the concerns below: \n\n> ... The authors can at least include the comparison to CELL-E which the proposed method is based on.  \n\nAs the reviewer agrees, this is an incredibly new field of study. As such, there are no proper baseline models with which to compare. For image prediction, the only comparison would be to CELL-E. Our evaluations of different training data (CELL-E uses OpenCell only whereas CellBERT-E combines HPA with OpenCell) and language models (CELL-E uses the BERT model from [1] whereas CellBERT-E uses ESM-2) are implicitly comparing these two models and showing the improvements of CellBERT-E. However, we will include an explicit comparison with CELL-E in the final revision.\n\n> The choice of hyperparameters needs more validation. In Table 1&2, the deeper the model is, the smaller the hidden size is. Why do the authors choose such settings in architecture exploration? Besides, it's hard to conclude the effect of hidden size and depth on the performance as they are changing together.  \n\nHidden size in predetermined by the ESM-2 language model that we use as a backbone. Depth is based on the findings of the CELL-E paper, showing a correlation between depth and token prediction ability. We maximized the depth that based on the available VRAM in training. We will be including this rationale in Section B.2. \n\n> ...Are there other domain metrics that are important in protein localization prediction? Will these image metrics fail to capture such important details? \n\n“Nucleus Proportion MAPE” is included as a domain-specific metric which calculates the proportion of intensity within the cell nucleus. This is the most relevant metric to the problem. We include other image metrics for the sake of thoroughness and evaluation for future models. \n\n> The authors mention the potential rapid overfitting on OpenCell containing limited data. Have the authors considered any data augmentation strategies to enlarge the training set? \n\nWe already utilize standard data augmentation techniques which involved randomly cropping the amino acid sequence (A.3) and randomly cropping and flipping on the image (A.4). HPA much larger and more diverse than OpenCell, and pretraining helps our model learn generalizable features for localization prediction.\n\n> In Table 2, all models achieve high cosine similarities. Is this because only 15% of AA sequence is masked? The authors can add the results of random sampling to better validate the effectiveness of proposed model. \n\nMasking 15% of the amino acids is a standard in unsupervised protein sequence learning. Could you clarify what “random sampling” means in this context, as the chosen positions to mask in Table 2 are selected at random. \n\n> In Table 3, models that perform well on FID are likely to perform poorly on the other 5 metrics. What could be the reason? \n\nFID measures the similarity between the synthesized images and the ground truth images in terms of the feature space of Inception v3. Models that performed well on FID have only seen one type of image during training (Table 3), and they may have learned to generate images that resemble that type. This may result in high FID scores, but low scores on the other metrics, which are more sensitive to the correct localization of the protein within the cell. FID is also not a stable metric, and can be affected by factors such as JPEG quality [2]. \n\n> In Section 6.2, the authors use the predicted signal in nucleus from the image model to validate the generated NLS. I find it unconvincing since the image model trained on limited data can fail when new data is fed in. \n\nOur goal was to create a list of candidate NLS sequences for future experimental validation. The image prediction is used to filter out unlikely candidates and not for validation. It is part of the process to generate the NLS candidate list. A two-step process, which combines the reverse model of sequence infilling and forward model of image prediction, should increase the success rate. \n\nOur current validation of the generated NLS candidates is based on the biological knowledge about NLS, such as the enrichment of certain types of amino acids and sequence features (domain knowledge not provided to the model). As shown in Table S9 and Figure S10, our generated NLS candidates have a strong potential to be functional.  \n\nTo address the reviewer’s concern, we have additionally passed our generated NLS candidate sequences (appended to GFP) to DeepLoc 2.0 [3], an independent model predicting multiclass annotations of protein localizations. DeepLoc 2.0 predicted **89%** of the generated NLS candidates with nuclear localization and **91%** having potential nuclear localizing signals, clearly validating our model.   \n\n> Typo in line 88, I suppose it should be \"allowing images to be synthesized in relatively few steps\". \n\nThank you, we have updated the manuscript. \n\n> Besides the structural information of proteins mentioned by the author. The limited training data can be another limitation of the work. Even HPA, the larger dataset in this work, contains only ~17,000 data which is noisy. The authors bring up this in Introduction. However, how the proposed method addresses the concern should be further clarified. \n\nWe agree that the limited and noisy training data is a challenge. We utilize a frozen pre-trained language model, ESM-2, which was trained on millions of protein sequences, threreby providing a rich source of prior knowledge from a large corpus. We have updated Section 4.2 to explain this rationale in more detail.\n\n[1] Rao, et. al. Evaluating Protein Transfer Learning with TAPE. Advances in Neural Information Processing Systems 2019\n\n[2] Parmar, et. al. On Aliased Resizing and Surprising Subtleties in GAN Evaluation. CVPR, 2022\n\n[3], Vineet, et. al. DeepLoc 2.0: multi-label subcellular localization prediction using protein language models. Nucleic acids research. 2022"}]

Author rebuttal:

Meta-review: All reviewers agree that this paper should be accepted. This paper borrows the recent work in text-to-image generation that tokenized the image information using vector quantized approach and performs the text-to-image generation task using non-autoregressive token generation, which is currently one of the two mainstream methods in image generation. 
This paper formalizes the protein location problem as the text-to-image generation, i.e., from protein sequence to a Nucleus image, which is quite promising and works well in the experiments. The main concerns of this paper lie in:1) the background of this task being less introduced, making it hard to understand the literature. 2) The task is very new, and should include more baseline for comparisons.
Decision: Accept (poster)