A foundation model for multi-task cross-distribution restoration of fluorescence microscopy images

Domain: biology
Published: 2026-03-10
Editorial outcome: minor_revision

Abstract
Deep learning has demonstrated remarkable abilities in restoring fluorescence microscopy images degraded by noise, blur, or undersampling. However, most existing models are task-specific and trained on limited, homogeneously distributed data, which restricts their generalizability and practicality. Here, we present FluoResFM, a foundation model for multi-task and cross-distribution fluorescence microscopy image restoration in a unified model. FluoResFM leverages textual prior information to adapt to specific tasks and data distributions. Trained on datasets across three tasks (image denoising, deconvolution, and super-resolution) and over 20 biological structures, FluoResFM demonstrates superior restoration performance and enhanced generalization across datasets with varied biological structures and imaging conditions. Through fine-tuning with only a single sample, FluoResFM can further improve its performance on unseen data, achieving results comparable to conventional models trained on hundreds of samples, and be easily adapted to additional tasks, including 3D image restoration, surface projection, isotropic reconstruction, and super-resolution with various scale factors. Moreover, the performance of existing cell/organelle segmentation models can be enhanced using the high-quality images restored by FluoResFM.

Introduction
Fluorescence microscopy is an essential tool for observing the morphology and dynamics of biological structures1. However, fluorescence microscopy images are often compromised by noise, blur, or undersampling due to limitations in light exposure, imaging speed, and spatial/temporal resolution2,3. Recently, deep learning-based algorithms have shown remarkable promise in fluorescence microscopy image restoration, including image denoising4,5, deconvolution4,6,7, and super-resolution5,8,9, enabling more accurate observation and analysis of biological structures and dynamics.

Current deep learning-based fluorescence microscopy image restoration methods are typically task-specific and trained on limited, homogeneously distributed data, which restricts their generalizability and practicality in biological research. Separate models are generally required for each restoration task due to the significant differences in task objectives4. Moreover, imaging objects (e.g., clathrin-coated pits (CCPs) and microtubules (MTs)) and imaging conditions2,10 (e.g., light exposure and type of microscopy) vary widely across research contexts, leading to heterogeneous data distributions. This distribution shift poses a challenge to the generalizability of deep networks. When a deep network trained on a specific data distribution is applied to data with a different characteristic, it may hallucinate details and produce artifacts11. Furthermore, due to the complexity of sample preparation and image acquisition, task-specific homogeneous datasets are often limited, increasing the risk of overfitting11. The inefficiency and impracticality of deploying separate models for each task and dataset highlight the need for a unified model capable of handling diverse tasks and leveraging large, heterogeneous datasets.

Foundation models have gained significant attention in image analysis12–17 and processing18–20 for their effectiveness and strong generalizability, which stems from their pre-training on large-scale, diverse datasets. For image restoration problems, foundation models can handle diverse degradations simultaneously18,19, adapt to unseen tasks or datasets through fine-tuning on limited data20, or serve as effective preprocessing tools for enhancing downstream image analysis tasks18. In fluorescence microscopy, UniFMIR19 utilized task-specific heads and tails along with shared Swin transformer layers to address different restoration tasks in a unified framework. While UniFMIR demonstrated commendable image restoration precision and generalization across different tasks, it did not fully account for the significant data heterogeneity introduced by variations in imaging objects and conditions, potentially leading to confusion between different data distributions.

In this study, we present FluoResFM, a generalizable foundation model for multi-task, cross-distribution restoration of fluorescence microscopy images. FluoResFM enhances the quality of images from diverse biological structures and imaging conditions through denoising, deconvolution, and super-resolution in a unified model. It leverages textual prior information, including task type, imaging object, and imaging condition, to guide adaptation to specific task and data distribution. We trained and evaluated FluoResFM on an extensive collection of real-world datasets (over 4.3 million paired patches for training), spanning three restoration tasks and more than 20 types of biological structures from varying imaging scenarios. Comprehensive evaluations demonstrated its superior restoration performance and generalizability on diverse internal and unseen external datasets across different tasks and data distributions. The textual prior information enabled effective control over image contents and patterns, providing a clear indication of the underlying task objective and data distribution. As a foundation model with strong generalization capability, FluoResFM can achieve improved restoration performance on unseen data through fine-tuning with as few as a single sample, obtaining results comparable to those of conventional deep networks trained on hundreds of samples. FluoResFM can also be readily adapted to other tasks through fine-tuning on a single sample, including three-dimensional (3D) image restoration, surface projection from 3D volume, isotropic reconstruction, and super resolution with various scale factors. Moreover, the cell and organelle segmentation performance of existing automatic segmentation models can be further improved when applied to the high-quality images restored by FluoResFM.

To make FluoResFM readily available to the biological research community, we developed a user-friendly napari plugin that allows straightforward integration into imaging workflows.

Results
Inspiration and development of FluoResFM
Training a deep learning model on homogeneous data from a specific task, imaging object, and imaging condition enables the model to learn a strong prior that effectively constrains the solution space of an inverse problem. However, while this specificity mitigates the inherent ambiguity of ill-posed problems, it concurrently limits model generalizability and introduces bias when applied to out-of-distribution data. In fluorescence microscopy, a model trained on data with a specific type of imaging structure often exhibits unexpected bias when applied to structures with different patterns due to distribution shift (Supplementary Fig. 1). For example, the content-aware image restoration (CARE) model trained on data with CCP structures (CARE:CCP) produced incorrect CCP-like structures when restoring MT images (Supplementary Fig. 1a). Only when training and testing data shared the same distribution could the model generate expected structures (Supplementary Fig. 1a) and achieve optimal quantitative performance (Supplementary Fig. 1b). Although training on more heterogeneous data (such as a mixture of CCP, MT, and endoplasmic reticulum (ER) structures) alleviated the output bias problem (Supplementary Fig. 1a), it expanded the feasible solution space, increased output uncertainty, and yielded sub-optimal results (Supplementary Fig. 1). Additionally, task conflicts arising from differing objectives across different tasks further complicate image restoration within a unified model. Recent studies have demonstrated that prior information related to tasks21 or image contents22 can serve as a powerful condition for ill-posed inverse problem and guide deep learning models to adapt to different data distributions, which is a promising strategy for improving the generalization ability of models. Motivated by this insight, we designed the architecture of FluoResFM to effectively fuse image features with textual prior information, leveraging the latter to direct the model in adapting to fluorescence microscopy images from different tasks and data distributions (Fig. 1a). FluoResFM was based on the widely-used U-Net architecture23(Fig. 1b). To incorporate textual guidance, the textual prior information (including task type, structure type, and imaging condition) was encoded into a feature embedding via a text embedder. The embedding was subsequently fused with spatial image features at each scale of the U-Net through cross-attention operation24,25 (Supplementary Fig. 2). We adopted the text encoder from BiomedCLIP26 as the text embedder, which had been pre-trained on diverse biomedical datasets, including generic biomedical illustrations, radiography, digital pathology, and microscopy.

FluoResFM was trained on more than 4.3 million paired low and high-quality patches (Fig. 1a) across three tasks (denoising, deconvolution, and super-resolution (×2 upscaling)) and over 20 types of biological structures (Fig. 1c). To comprehensively evaluate the restoration performance of FluoResFM, we curated 302 internal datasets and 51 external unseen datasets. Each dataset was internally homogeneous, while the collection as a whole spanned three tasks and a wide range of structure types and imaging conditions (Fig. 1d).

Performance of FluoResFM on internal datasets
We compared the performance of FluoResFM with its variant trained without textual prior information, and UniFMIR19, a state-of-the-art multi-task fluorescence microscopy image restoration foundation model, on the internal datasets. Across the majority of internal datasets, FluoResFM attained higher average peak signal-to-noise ratio (PSNR), multi-scale structural similarity index measure (MS-SSIM), zero-normalized cross-correlation (ZNCC), resolution-scaled Pearson coefficient (RSP) values and lower resolution-scaled error (RSE) values (Fig. 2a, Supplementary Fig. 3a, and Supplementary Data 1). The resolution of images restored by FluoResFM were more consistent with that of ground-truth (GT) images (Supplementary Fig. 3b). Overall, FluoResFM demonstrated superior performance compared to FluoResFM without text guidance and UniFMIR across all three restoration tasks (Fig. 2c). The FluoResFM model effectively suppressed the noise, removed the blur, and enhanced the spatial resolution of the images, producing outputs closer to the high-quality GT images (Fig. 2b). FluoResFM better resolved closely spaced structures (e.g., super-resolution of nuclear pore complexes (NPCs) and MTs) (Fig. 2b, red arrows) and reconstructed more continuous and sharper MTs and ER networks (e.g., deconvolution of MTs and ERs) (Fig. 2b). In contrast, UniFMIR merged adjacent objects or produced fragmented reconstructions of MTs and ERs (Fig. 2b). FluoResFM without textual information guidance tended to deblur all the input images and could not fulfil the denoising task as expected (e.g., denoising of CCPs and F-actin filaments) (Fig. 2b), resulting in noticeably lower PSNR, MS-SSIM, ZNCC, and RSP values and higher RSE on certain datasets for the denoising task (Fig. 2a and Supplementary Fig. 3a). These results suggest that simply unifying multiple tasks in a single universal model without any task-related prior information can result in uncontrollable and undesired predictions. Furthermore, in both the deconvolution and super-resolution tasks, FluoResFM without textual guidance was not only quantitatively inferior (Fig. 2a,c and Supplementary Fig. 3a) but also produced visibly blurrier reconstructions compared to FluoResFM (Fig. 2b). Benefiting from the superior restoration performance of FluoResFM, the CCP diameter measured in the restored images was more consistent with that of GT images (Supplementary Fig. 4a,d). Long filaments in the MT network could be tracked more easily (Supplementary Fig. 4b), and more complete skeleton and more nodes of the ER network were detected (Supplementary Fig. 4c) in the restored images. Consequently, the measured median filament length (Supplementary Fig. 4e), average node degree (Supplementary Fig. 4f), and node count (Supplementary Fig. 4g) were closer to those of GT images compared to the raw input.

Performance of FluoResFM on external unseen data
To demonstrate the generalizability of FluoResFM without model retraining or parameter adjustment, we evaluated its image restoration performance on 51 external unseen datasets. FluoResFM achieved higher PSNR, MS-SSIM, ZNCC, RSP values and lower RSE values across the majority of datasets (Fig. 3a, Supplementary Fig. 3c, and Supplementary Data 2). The resolution of images restored by FluoResFM were more consistent with that of GT images (Supplementary Fig. 3d). Overall, FluoResFM exhibited significantly superior quantitative performance across all three tasks (Fig. 3c). FluoResFM without textual guidance still struggled with the denoising task and tended to reconstruct a deblurred image instead of removing noise (e.g., denoising of CCPs and F-actin) (Fig. 3b), which was consistent with the observations on internal datasets (Fig. 2b). Despite not being trained on the external data, FluoResFM successfully restored accurate structural details that closely resembled the GT references, such as the hollow, ring-shaped CCPs, spherical lysosomes, as well as filamentous F-actin and MTs (Fig. 3b).

Influence of textual prior information on FluoResFM restoration
Textual prior information can serve as an effective condition to guide the image reconstruction process and control the generated image content, which has shown promising results in natural image synthesis24,27, restoration28, and enhancement29. When combined with user-specified restoration task type, such prior knowledge can provide valuable cues for inferring image content patterns and alleviating the inherently ill-posed nature of image restoration. In fluorescence microscopy, imaging objects and conditions are typically known, and this knowledge can serve as a prior for the restoration.

FluoResFM can effectively control the reconstructed patterns of structures by altering the input text prompt (Fig. 4a). For instance, with the “denoising” prompt, FluoResFM only removed the noise from the ER image while preserving its intrinsic blur and low resolution (Fig. 4a, blue). The “deconvolution” prompt deconvolution” prompt deconvolution” prompt deblurred the input image (Fig. 4a, green), and the “super-resolution” prompt further up-sampled the image and restored finer structural details (Fig. 4a, red). Only with the correct structure type prompt, such as “ER” in Fig. 4a, can FluoResFM produce the most accurate restoration. In contrast, when using the “CCP” prompt as input, the ER was mistakenly reconstructed as CCP-like structures, while the “MT” prompt yielded structures resembling evenly thick, crisscrossing tubules rather than the correct reticular structure with uneven thickness (Fig. 4a, red arrows). Furthermore, with the correct structure type prompt, FluoResFM achieved more accurate morphology-aware measurements, such as CCP diameter, filament length in the MT network, and count and degree of nodes detected in the ER network, compared to the raw input (Fig. 4d and Supplementary Fig. 5). In contrast, incorrect structure type prompts may degrade the accuracy of these measurements. For example, when “MT” or “ER” prompts were used to restore CCP images, the CCP diameter was overestimated (Supplementary Fig. 5a). Using the mismatched “CCP” prompt led to severe reconstruction distortions in both the MT and ER images (Fig. 4d), resulting in shorter extracted filaments in the MT network on average (Supplementary Fig. 5b), and fewer detected nodes in the ER network (Supplementary Fig. 5c). Due to the structural similarity between MT and ER, the confusion between the “MT” and “ER” prompts did not lead to significant degradation in filament length or node count/degree measurements (Supplementary Fig. 5b-d), although detailed structural features were distorted (Fig. 4a). These results highlight the strong controllability offered by textual priors in guiding the image restoration process. Given the critical role of structure type prompt in restoration, we employed an image retrieval-based classifier to help general users recognize the structure type of the input image, thereby preventing image-prompt mismatch (Supplementary Note 1 and Supplementary Fig. 7).

Additionally, the deficiency of prior information decreased the performance of FluoResFM (Fig. 4b,c). Compared to FluoResFM using only the task information (FluoResFM-T), only the task and structure information (FluoResFM-TS), or only the task, structure, and input metadata (i.e., microscope type and parameters used for acquiring input image) (FluoResFM-TS-in) as input, FluoResFM with full textual prior information achieved significantly higher PSNR, MS-SSIM, and ZNCC values on both internal (Fig. 4b) and external unseen datasets (Fig. 4c). The quality of the restoration increased with the addition of textual information (Fig, 4b,c). When detailed target imaging parameters were unavailable, FluoResFM using only the task, structure, input metadata, and target microscope type information (FluoResFM-TS-in-tm) still achieved acceptable restorations, showing relatively good results on internal datasets (Fig. 4b) and no significant differences in PSNR and ZNCC on external datasets (Fig. 4c). Removing the target metadata during both training and inference stages led to a modest performance decrease across most metrics on the internal (Supplementary Fig. 6a) and external datasets (Fig. 4e). Compared to using structured categorical embeddings (Methods), FluoResFM showed slightly degraded performance on internal datasets (Supplementary Fig. 6b) but demonstrated improved generalization to external datasets (Fig. 4f).

Transfer FluoResFM to unseen data via fine-tuning
The availability of training data is a critical factor influencing the performance of deep learning models. Insufficient training data often leads to suboptimal restoration performance11. Transfer learning offers a solution by leveraging prior knowledge learned from large-scale datasets and transferring it to limited, task-specific, homogeneous unseen data that would be inadequate on its own19,30,31. To evaluate the efficiency of FluoResFM in transferring knowledge to unseen data, we fine-tuned the model separately on 12 external datasets with different biological structures. For each dataset, only a single image was used for fine-tuning. For comparison, conventional deep learning-based image restoration models (the CARE network4 and the deep Fourier channel attention network (DFCAN)8) were trained from scratch using the same sample. CARE yielded the lowest PSNR, MS-SSIM, and ZNCC values, even worse than the low-quality input images (Fig. 5a). Although DFCAN achieved better results than CARE, its performance remained substantially inferior than FluoResFM (Fig. 5a). The poor performance of them was primarily attributed to overfitting, resulting from the insufficiency of training data (Fig. 5c). While the overfitting issues of CARE and DFCAN could be mitigated with more training data, they required hundreds of images to reach the performance achieved by FluoResFM fine-tuned on just a single image (Fig. 5d). In contrast, both UniFMIR and FluoResFM exhibited stronger resistance to overfitting (Fig. 5c) and obtained better performance after fine-tuning (Fig. 5a). The fine-tuned FluoResFM model attained significantly higher PSNR, MS-SSIM, and ZNCC values compared to CARE and DFCAN trained from scratch, UniFMIR and FluoResFM without fine-tuning, as well as the fine-tuned UniFMIR model (Fig. 5a). To test its applicability in dynamic imaging, we fine-tuned FluoResFM on the image acquired at 0 seconds in live imaging of CCPs and lysosomes separately (Fig. 5b). The fine-tuned models were then applied to restore images at subsequent time points. FluoResFM markedly enhanced the signal-to-noise ratio (SNR) and resolution of raw images, allowing for clearer differentiation of tiny structures and more accurate tracking of their dynamics (Fig. 5b, red dashed boxes). Benefiting from these, the diameter of CCPs (Supplementary Fig. 8a,c) and the count of lysosomes (Supplementary Fig. 8b,d) measured in the restored images were more consistent with those measured from GT images across all the time points and showed reduced spurious frame-to-frame fluctuations (Supplementary Fig. 8c).

FluoResFM can be further adapted to other restoration tasks by fine-tuning, such as 3D image restoration, surface projection, isotropic reconstruction, and super-resolution with higher up-sampling factors. For 3D image restoration, the low-quality 3D volume input can either be restored slice-by-slice using FluoResFM or treated as a multi-channel 2D input19 such that all slices can be restored via FluoResFM simultaneously (Fig. 6a). For surface projection from 3D volume, to obtain a high-SNR 2D surface image, the low-SNR 3D volume input can first be projected to a low-SNR 2D surface image via maximum intensity projection and then denoised using FluoResFM. Alternatively, the 3D volume can also be treated as a multi-channel 2D input and directly projected to a 2D surface image through FluoResFM (Fig. 6b). For each task, only a single sample was used for fine-tuning.

For the 3D image restoration task, compared with slice-by-slice restoration (FluoResFMS-S2S), FluoResFM with multi-channel input and output (FluoResFM-M2M) could leverage information from neighboring slices and produced more continuous structures in the axial direction (white arrow in Fig. 6c). Quantitatively, FluoResFM-M2M achieved higher PSNR, MS-SSIM, and ZNCC values than FluoResFM-S2S (Supplementary Fig. 9a). For the surface projection task, compared with denoising the maximum projection image (FluoResFM (dn)), directly projecting the low-SNR 3D volume to a 2D surface image via FluoResFM (FluoResFM (proj)) preserved more detailed structures (white arrow in Fig. 6d) and obtained higher PSNR, MS-SSIM, and ZNCC values (Supplementary Fig. 9b). In the isotropic reconstruction task, FluoResFM effectively improved the axial (z-axis) resolution of the image, enabling better discrimination of adjacent nuclei (Fig, 6e,f). The Fourier spectrum analysis of the images before and after restoration demonstrated that the frequencies along the axial dimension were well restored, and the aliasing artifacts in the frequency domain caused by insufficient axial sampling were reduced (Fig. 6g). For the super-resolution tasks with scale factors of 3×, 4×, and 8×, FluoResFM effectively enhanced the quality of input images, producing outputs close to the high-quality references (Fig. 6h-j). Quantitatively, the restored images achieved significantly higher PSNR, MS-SSIM, and ZNCC values compared to the low-quality inputs (Supplementary Fig. 9c-e).

Boost automated cellular and organelle segmentation
Accurate segmentation of cell bodies, membranes, nuclei, and other organelles is crucial for many quantitative cell biological applications32,33. Existing automated cellular and organelle segmentation methods, such as Cellpose-SAM34 and Nellie35, have demonstrated impressive versatility across diverse cellular and intracellular structures, yet they show limited robustness to image degradation. Their segmentation performance can be significantly improved when applied to images restored by FluoResFM. We used 72 datasets encompassing a variety of cellular and organelle structures to evaluate the effectiveness of FluoResFM for cellular and organelle segmentation. The low-quality image was first restored using FluoResFM, and then segmented using either Cellpose-SAM or Nellie to obtain object masks (Fig. 7a). Specifically, Cellpose-SAM was used for nuclear and membrane segmentation, while Nellie was employed for intracellular organelle segmentation. Segmentation masks derived from high-quality GT images served as the reference.

Directly applying Cellpose-SAM or Nellie on degraded images often results in inaccurate segmentation, producing missing (CCP and nuclei in Fig. 7b), connected (lysosome and nuclei in Fig. 7b), fragmented (mitochondria (Mito), ER, and MT in Fig. 7b) organelle or nuclei masks. In contrast, segmentation on FluoResFM-restored images was markedly more accurate (Fig. 7b). Quantitatively, both the average precision (AP) and Intersection over Union (IoU) values were improved on most evaluated datasets (Fig. 7c), highlighting the robustness and effectiveness of FluoResFM across diverse biological structures and imaging conditions.

Methods
Data collection and preprocessing
We collected numerous publicly accessible fluorescence microscopy image datasets4,5,19,44–51 covering diverse imaging objects and conditions across denoising, deconvolution, and super-resolution tasks for model training and evaluation (Supplementary Table 1-4 and Supplementary Data 3). No cell culture experiments were performed by the authors. Imaging object and condition information was manually curated for each dataset. Both low-quality input and high-quality GT images were normalized to a common intensity range using percentile-based normalization with the 3rd percentile as the minimum and the 99.5th percentile as the maximum value. For the super-resolution task, low-quality images were up-sampled using nearest-neighbor interpolation to match the dimension of high-quality GT images. For internal datasets (Supplementary Table 1 and Supplementary Data 3), we manually divided the datasets lacking pre-defined splits into training and test sets using an approximately 80:20 ratio. To enable simultaneous training across different datasets, we cropped the training images into patches with the same spatial size (64×64 pixels) (Fig. 1a). Patches containing only noise were excluded from the training set. The final training data relatively uniformly distributed across the three tasks, with more data for super-resolution given the complexity of this problem (Fig. 1c). The training data encompassed over 20 types of structures (Fig. 1c), including CCP, ER, F-actin, MT, myosin-IIA, nucleus, histone, nuclear-envelope, membrane, granule, tubulin, nucleoid, MreB, Mito, lysosome, survivin, neural crest cell (NCC), NPC, chromosome, and Golgi, imaged with wide-field microscopy, structured illumination microscopy, instant structured illumination microscopy, confocal microscopy, multi-photon microscopy, light-sheet microscopy, two-photon microscopy, and stimulated emission depletion microscopy. The textual prior information consisted of task type, sample type, structure type, fluorescence indicator, and microscopy techniques with their parameter settings used for both the low-quality input and high-quality GT image acquisition, which were concatenated sequentially as a single text input (Supplementary Table 7). The detailed information of internal and external datasets for restoration performance evaluation, as well as datasets for segmentation evaluation were shown in Supplementary Data 4. The information of datasets used for the tasks of 3D image restoration, surface projection, isotropic reconstruction, and super-resolution with 3×, 4×, and 8× up-sampling factors were shown in Supplementary Table 3.

Network architecture and training loss
The backbone of FluoResFM was realized as a text-conditional U-Net23,24 (Fig. 1b, Supplementary Fig. 2, and Supplementary Table 6) comprising an encoder, a decoder, and skip connections. The encoder started with a convolution layer, while the decoder ended with a convolution layer to generate restored images. Each scale of the encoder and decoder contained a residual block and a text-image fusion block (Supplementary Fig. 2). The text-image fusion block utilized cross-attention mechanism25 to integrate textual prior information with image features. The textual prior information was first projected into a text embedding 𝐳∈ℝ𝑙×𝑑 (where 𝑙 is the context length and is set to 160, and 𝑑 is the size of embedding vector) through the pre-trained text encoder in BiomedCLIP26 and then injected into the cross-attention layer24 in the text-image fusion block. For the FluoResFM (SCE) model, we replaced the unstructured textual prior embeddings with structured categorical embeddings for task, structure, and microscope type. Specifically, a unique ID was assigned to each task type, structure type, and microscope type and mapped to a learnable embedding. The structured categorical embeddings can be expressed as:
𝐳 =embed((𝑥 ,𝑥 ,𝑥 ,𝑥 )) (1)
sce task structure micro−in micro−target
where 𝐳 ∈ℝ4×𝑑, 𝑑 is the size of embedding vector, embed(⋅) denotes a lookup table that stores learnable embeddings for each task type, structure type, and microscope type, and 𝑥 , 𝑥 , 𝑥 , and 𝑥 are the IDs corresponding to the task type, structure type, and the input and target microscope types, respectively.

We used a mean absolute error loss 𝐿 to calculate the errors between restored images and corresponding high-quality GT images:
1
𝑁
1
𝐿 = ∑|𝐈 (𝑖)−𝐈 (𝑖)|, (2)
1 𝑁 e gt
𝑖=1
where 𝑁 is the total number of pixels in image 𝐈, 𝑖 is the index of pixel, 𝐈 is the restored image, and 𝐈 is the high-quality GT image.
e gt

Adapting FluoResFM to process 3D data
For the 3D image restoration task, to enable multi-channel input and output, the number of input channels of the convolution layer at the head of FluoResFM and the number of output channels of the convolution layer at the tail were set to match the number of slices of the input volume. For the surface projection task, only the number of input channels of the convolution layer at the head of network was adjusted to match the number of slices of the input volume.

Network training and fine-tuning
The FluoResFM model was implemented using the PyTorch deep learning framework (Pytorch 2.6.0) and optimized by adaptive moment estimation with decoupled weight decay (AdamW)52 optimizer. The batch size was set to 16, and the model was trained for 700,000 iterations. The initial learning rate was set to 1×10-5 and halved every 100,000 iterations. To maintain the consistency of training conditions across different networks and conduct a fair comparison, the comparison methods followed similar training settings (Supplementary Table 5). During fine-tuning, only parameters in the first and last convolution layers of the U-Net backbone were updated, while all other parameters were frozen (Supplementary Fig. 2). The training settings remained unchanged, except for the total number of iterations (Supplementary Table 5). All experiments were conducted on a computer equipped with an Intel Core i9-14900K processor and a NVIDIA GeForce RTX 4090D graphic processing card (24 GB of RAM).

Evaluation metrics
We adopted five image quality assessment metrics (PSNR, MS-SSIM, ZNCC, RSE, and RSP) to quantitatively evaluate the image restoration performance of different methods. PSNR was used to measure the pixel-wise difference between restored and GT images and implemented using the scikit-image53 package (version 0.22.0). MS-SSIM54 was employed to quantify the structure similarity between restored and GT images across multiple spatial scales, which was implemented using the pytorch-MS-SSIM package (version 1.0.0). Unlike single-scale SSIM55, MS-SSIM incorporates the structures of different scales into assessment. ZNCC was used for evaluating the cross-correlation between restored and GT images and was defined as56
𝑁
1 (𝐈 −𝜇 )(𝐈 −𝜇 )
e e gt gt
ZNCC= ∑ , (3)
𝑁 𝜎 𝜎
e gt
𝑖=1
where 𝑁 is the total number of image pixels, 𝑖 is the index of pixel, 𝐈 is the restored image, 𝐈 is the high-quality GT image. 𝜇 and 𝜇 are the mean pixel intensity in 𝐈 and 𝐈 , respectively. 𝜎 and 𝜎 are the standard deviation of the pixel intensities in 𝐈 and 𝐈 , respectively. SQUIRREL analysis19,57,58 was used to calculate the RSE and RSP metrics, and this analysis was implemented using the NanoPyx59 framework in Python. RSE measures the root-mean-square error between the reference and resolution-scaled image, and RSP computes the Pearson correlation coefficient between the reference and resolution-scaled image57. Image resolution was estimated using decorrelation analysis60, which has demonstrated consistent robustness across different structures and imaging modalities61.

Before calculating these metrics, we performed percentile-based normalization for each restored image and its corresponding GT image to match the dynamic range between them:
𝐈−percentile(𝐈,𝑝 )
Norm(𝐈,𝑝
low
,𝑝
high
)= 
percentile(𝐈,𝑝 )−perce
l
n
ow
tile(𝐈,𝑝 )
, (4)
high low
where 𝐈 is the restored or GT image, percentile(𝐈,𝑝) computes the 𝑝-th percentile of the pixel values in image 𝐈, and 𝑝 and 𝑝
low high
are set to 0.03 and 0.995, respectively.

We quantified the segmentation performance using the IoU and AP62 between the predicted and ground truth masks. The AP is defined as:
TP
AP= , (5)
TP+FP+FN
where true positive (TP) is the number of matches with IoU above a threshold of 0.5, false positive (FP) is the number of predicted masks without matches, and false negative (FN) is the number of missed ground truth masks. All the metrics mentioned above were implemented with Python. Additionally, we measured the CCP diameter using in-house MATLAB (R2023b) code, quantified the length of filaments in MT network using the open-source software package SIFNE63, and detected the nodes in ER network using Nellie35 (version 0.4.1).

Statistics and reproducibility
All neural networks were trained once for each set of hyperparameters. Random seeds were fixed for network parameter initialization. The training and testing images were randomly split for each dataset. The images within each dataset were acquired from different samples under the same imaging condition. For comparisons between two groups, a one-sided Wilcoxon signed-rank test was used. A p value < 0.05 was considered statistically significant. All statistical analyses were performed using Python (version 3.11.9) and SciPy package (version 1.12.0). No statistical method was used to predetermine sample size. No data were excluded from the analyses.

Image visualization
The subplots were generated using Python and assembled into composite figures with Adobe Illustrator. For the image patches in Fig. 1a and b, we used the “hot” colormap to illustrate the signal intensity. Prior to visualization, images in Fig. 1d, Fig. 2-6, Supplementary Fig. 4, and Supplementary Fig. 1 were normalized and then converted to RGB format. For the radar plots of Fig. 2a, Fig. 3a, and Supplementary Fig. 3a,c, the metric values of each dataset were individually normalized according to the minimum and maximum values:
𝐌−𝑀
𝐌 = l , (6)
norm 𝑀 −𝑀
h l
where 𝐌 is the vector of metric values across different methods, 𝑀 and 𝑀 are the lower and upper limits, respectively. These limits were automatically determined such that the minimum and maximum data points correspond to the positions at 20% and 95% of the axis range, which are defined as:
(𝑎−𝑏)+𝑏⋅𝑚−𝑎⋅𝑛
𝑀 =⌈ ⌉⋅𝑝, (7)
h (𝑚−𝑛)⋅𝑝
𝑏⋅𝑚−𝑎⋅𝑛
𝑀 =⌊ ⌋⋅𝑝, (8)
l (𝑚−𝑛)⋅𝑝
where 𝑎 and 𝑏 are the maximum and minimum metric values, respectively. 𝑚=0.95, 𝑛=0.2, and the precision 𝑝 is set as 0.1, 0.00, and 0.001 for PSNR, MS-SSIM, and ZNCC, respectively. The images in Fig. 7a and b were plotted using the “gray” colormap.

Data availability
All the datasets used in this study are derived from existing literature and are publicly accessible through the corresponding links provided in Supplementary Table 1-4. Example data for training and testing are available on Zenodo repository (https://doi.org/10.5281/zenodo.18382702). Source data are provided with this paper. Other raw data (e.g., image raw data underlying the figures and intermediate data for deep learning) are available from the corresponding author upon request due to their large file size.

Code availability
The source codes are available via GitHub repository (https://github.com/qiqi-lu/fluoresfm) and Zenodo repository64 (https://doi.org/10.5281/zenodo.18383925). The napari plugin code is publicly accessible via GitHub at https://github.com/qiqi-lu/napari-fluoresfm. The guidance video on using the napari plugin is available at https://www.bilibili.com/video/BV16JeFzuEof. The pre-trained text and image encoder from BiomedCLIP and the pre-trained FluoResFM model are publicly accessible on Zenodo repository (https://doi.org/10.5281/zenodo.18382702).

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The paper introduces FluoResFM, a text‑conditioned U‑Net that fuses image features with textual priors (task type, imaging object, and imaging conditions) via cross‑attention, using a BiomedCLIP text encoder. The goal is a single model for denoising, deconvolution, and 2x super‑resolution across diverse structures and acquisition settings. The authors train on >4.3M paired patches from >20 structure types and evaluate on 302 internal and 51 external datasets. They report superior PSNR/MS‑SSIM/ZNCC versus UniFMIR and an ablation without text guidance, show one‑image fine‑tuning that outperforms training CARE/DFCAN from scratch, and demonstrate that restoring images with FluoResFM improves downstream segmentation by Cellpose‑SAM and Nellie. A Napari plugin is provided for inference, preprocessing, and finetuning.\n\nThe paper has many strengths. The motivation and design of the model is clear, the architecture (U‑Net + cross‑attention with textual priors) is well motivated by the need to disambiguate multi‑task, multi‑distribution restoration. Training spans many paired patches across three tasks and a variety of structures, with heterogeneous microscopy modalities and conditions and the external evaluation covers a large number of datasets. It seems to generalize well, the authors present compelling cross‑distribution results on both internal and external sets and ablations on textual prior completeness. On downstream tasks, they show demonstrable gains in instance/semantic segmentation across a variety of datasets when segmenting restored images. The software seems quite user friendly, with source code available alongside a point-and-click Napari plugin.\n\nThere are a few points that should be addressed before accepting this manuscript, mainly revolving around the text-conditioning and textual priors:\n1. Methods state: \"The textual prior information consisted of... with their parameter settings used for both the low‑quality input and high‑quality GT image acquisition, concatenated as a single text input.\" (line 341). At inference the GT acquisition parameters are unknown, and the user may not know what the target should be, so including \"target\" conditions at train time risks a distribution gap. Please retrain/evaluate using only input‑side metadata (task + input structure + input conditions), and report the performance delta. The partial ablation (T, TS vs full in Fig. 4b/c) suggests sensitivity to the amount of prior, but it is crucial to explicitly exclude GT‑side tokens at inference and in training inputs that simulate inference.\n2. Fig. 4a shows that wrong prompts produce wrong morphologies (e.g., ER reconstructed as CCP‑like rings). This is a major scientific risk if users mis‑specify the prompt. Please:\na. Quantify failure under prompt mismatch by sweeping structure prompts with a confusion matrix of morphology‑aware metrics (e.g., filamentness, tubule thickness distributions, ring diameters for NPCs).\nb. It would also likely be very beneficial to the general audience to add prompt‑validation safeguards (e.g., a small classifier that flags likely mismatches between input image and text prior) and show reduced error rates with such guardrails.\nc. Provide user‑facing warnings in the plugin and a model card describing these risks.\n3. Relatedly, it would be helpful to compare to structured categorical embeddings (one‑hot / learned tokens) for task, structure, and microscopy. This would test whether domain‑specific categorical conditioning suffices and might be more robust than unstructured text.\n4. Figures 2a and 3a plot normalized metrics (radar plots), and text notes axis scaling. Normalization plus radar visualization can obscure absolute gains. Please include tables of raw PSNR/MS‑SSIM/ZNCC (means +/- st. dev) per dataset, and effect sizes (e.g., median paired improvement with 95% CI). Keep normalization for visualization if desired, but report raw values in the Supplement.\n5. PSNR/MS‑SSIM/ZNCC against SIM/STED ground truth do not fully capture biological correctness and may inherit reconstruction biases from the ground truth modality. Please add:\na. Task‑specific biological validations, e.g., density/length distributions for MTs, ring diameters for NPCs, organelle counts and size distributions, compared to GT.\nb. Temporal consistency metrics for live‑cell sequences (beyond the qualitative Fig. 5b).\nOther minor points:\n1. Supplementary Table 5 is poorly formatted and cannot be read in full, which should be addressed.\n2. Fig. 1c shows per‑structure patch counts. Please also report per‑dataset image counts, not only patches, and the train/val/test splits (Supp. Table 1 lists sources but not split sizes)\n3. The authors note the 3D limitation. Even without 3D training data, a 2.5D variant (slice‑wise with axial context) could be a pragmatic step. Consider reporting preliminary results.\n4. \"The authors declare non competing interest.\" (line 540) should be \"no competing interests.\"\n5. Supp. Table 4 is helpful. Also consider adding examples of exact prompts.\nAssuming everything still holds up well after these revisions, I would be strongly supportive of acceptance.\nBest,\nAustin E. Y. T. Lefebvre", "reviewer_2": "his paper presents FluoResFM, a unified foundation model for restoring fluorescence microscopy images. It performs three tasks—denoising, deconvolution, and super-resolution—using a single model trained on a diverse dataset of over 4.3 million image patches from more than 20 biological structures. A key feature of the model is its use of textual priors (e.g., task type, imaging object, conditions) to guide image restoration. This text information, encoded via a BiomedCLIP embedder and integrated with image features through cross-attention, allows the model to adapt to different data distributions and tasks.\n\nIn evaluations, FluoResFM outperformed a state-of-the-art multi-task model and a variant of itself lacking text guidance. The results were validated quantitatively (PSNR, MS-SSIM, ZNCC) and qualitatively across 302 internal and 51 external datasets, demonstrating strong generalization. The model also exhibits effective transfer learning, achieving high-quality results on new data after fine-tuning with a single sample. Furthermore, the authors show that images restored by FluoResFM improve the performance of downstream segmentation models like Cellpose-SAM and Nellie. To facilitate adoption, the authors provide a napari plugin for training, fine-tuning, and prediction. The code is open-source, and the methods are detailed for reproducibility.\n\nMajor Comments\n(1) The text-based model is heavily depends on the text prompt which need human tuning. It could limit the application when the some condition is unknown or out-of-domain. The ablation and robustness of the text prompt need to be further explained.\n(2) Since the image captured of biological samples are huge in pixel sizes, the complexity of the foundational model can be a obstacle in high-throughput processing. It is a conventional practice to test the computational cost of each module in the model for multiple resolutions. However, the time and memory are not shown nor compared with previous works.\n(3) The current model is focused on only three common restoration tasks: denoising, deconvolution, and 2x super-resolution. It does not address other relevant tasks like isotropic reconstruction or super-resolution with higher up-sampling factors.\n(4) Biological samples are inherit 3D and 3D operations like projection and volumetric reconstruction are common for analysis. While the authors state it could theoretically be extended to 3D by replacing 2D convolutions with 3D ones, the effectiveness of text-prompt in the situation is unknown. It is a crucial ability to process 3D data for a foundational model.\n(5) The evaluation relied on several metrics, such as PSNR, SSIM, and ZNCC. While common in computer vision, these metrics may not thoroughly represent the quality for fluorescence microscopy. Other metrics, such as FRC, Decorrelation analysis or SQUIRREL analysis, may provide more biologically meaningful assessments.\n\nMinor Comments\n(1) Figure 5d: The caption mentions that FluoResFM was fine-tuned on a single sample, but the plot for \"FluoResFM (ft)\" is a horizontal line across all \"number of samples.\" This is confusing. It should be clarified in the caption or plot that the dashed line represents the performance of FluoResFM fine-tuned on one sample, shown for reference against CARE and DFCAN performance as their training sample number increases.\n(2) Figure 6c: The x-axis labels are very dense and difficult to read. While this is challenging with 72 samples, the authors might consider grouping them by structure type (e.g., all \"nuclei\" together, all \"Mito\" together) and perhaps using alternating background colors to visually separate these groups, which would improve readability.\n(3) Extended Data Fig. 2: While the structure of the network is shown, the detailed hyper-parameters of each layer is unknown, which is crucial for reproduction. It coude be better to include a full table of network layers."}, "author_response": "Response to Reviewer Comments\nDear Editors and Reviewers,\nWe are grateful for the opportunity to revise our manuscript in response to the careful and insightful comments provided.\nThe primary concerns raised by the reviewers include further exploration on the impact of the text prompt, adaptation of FluoResFM to 3D volume data, and the incorporation of more biologically meaningful metrics for evaluation. In response, we have conducted additional experiments, which mainly include:\n1. Further exploration of the influence of text prompt on our method under various conditions, including partial text prompt missing, structure-type prompt mismatch, and structured categorical embedding.\n2. Adaptation of FluoResFM to process 3D data, including 3D image restoration and surface projection from 3D volumes. Besides, through fine-tuning, FluoResFM has been adapted to perform isotropic reconstruction and super-resolution with higher scale factors.\n3. Additional evaluation of the restored images using more biologically meaningful metrics, such as image resolution-related metrics, CCP diameter estimation, filament length measurement in MT network, and node detection in ER network.\nAdditionally, we have addressed all the constructive suggestions from the reviewers and revised the manuscript accordingly. As requested in the previous email, we have updated the link to the example data for training and testing, the pre-trained text and image encoder, and the pretrained FluoResFM model checkpoint in “Code availability” section for easier accessibility.\nSpecific modifications are highlighted in dark red in the revised manuscript. The responses to the reviewer’s comments are presented in blue. We have also formatted the manuscript according to the Nature Communications formatting instructions. The modifications are also highlighted in dark red.\nThe raw data are provided in the “Source Data” file, with each figure’s data placed in a separate sheet. The name of sheet corresponds to the index of the respective figure.\nWe believe these revisions have enhanced the quality of the manuscript, and we hope it now meets the criteria for publication in Nature Communications.\nThank you for reconsidering our revised manuscript.\nSincerely,\nShenghua Cheng, PhD (on behalf of all the authors)\nProfessor, School of Biomedical Engineering, Southern Medical University, China\ne-mail: chengsh2023@smu.edu.cn\nREVIEWER COMMENTS\nReviewer #1 (Remarks to the Author):\n[... full response text ...]"}, {"round": 2, "reviewer_comments": {"reviewer_1": "The authors' rebuttal and revisions fully address my concerns and strengthen the manuscript. They provided clarifications, and added new experiments, analyses, and practical safeguards that improve both scientific rigor and usability.\nSpecifically, they retrained/evaluated variants that exclude target-side metadata during training and inference and reported the performance impact and practical guidance for cases where target metadata is unknown; quantified failure modes under structure-prompt mismatch using morphology-aware measurements (e.g., CCP diameter, MT filament length, ER node statistics); implemented a practical prompt validation guardrail via an image retrieval-based structure type predictor (clever!), demonstrating strong accuracy and clear benefit for non-expert users; compared unstructured text conditioning to structured categorical embeddings and articulated the tradeoff between in-distribution performance and out-of-distribution generalization; added raw metric tables and effect-size summaries; and added biologically meaningful validation metrics and temporal consistency quantification for live-cell sequences. Finally, they also provided a concrete extension toward 3D use cases which adds impact and relevance to the paper.\nOverall, the revisions are thorough, technically sound, and above and beyond what I requested, and the paper now reads as both a strong methodological contribution and a practical tool.\nI have no remaining concerns and strongly support acceptance.\nBest,\nAustin E. Y. T. Lefebvre", "reviewer_2": "I would like to thank the authors for their thorough and comprehensive response to the review comments. The authors have addressed each of my major and minor concerns with additional experiments, new quantitative evaluations, and text clarifications. The revisions have significantly strengthened the manuscript, particularly regarding the model's extensibility to 3D data and additional restoration tasks, as well as the transparency regarding computational costs.\nMajor Comments\n1.Reliance on Text Prompts: The authors have adequately addressed the concern regarding the robustness of text prompts. The new experiments comparing `FluoResFM-TS-in` and `FluoResFM-TS-in-tm` (Fig. 4b, c in the revised manuscript) clearly quantify the performance drop when target metadata is missing, while demonstrating that the model still yields acceptable results. Furthermore, the inclusion of the image retrieval-based classifier to assist users in identifying structure types is a practical solution to the \"unknown\" condition issue. The detailed analysis of structure type mismatch in Fig. 4d provides necessary transparency on the risks of incorrect prompting.\n2.Computational Complexity: Thank the authors for providing the detailed comparison in Supplementary Table 8 (Table R1 in the rebuttal). While FluoResFM is indeed computationally heavier than task-specific models like CARE or UniFMIR, the data provided allows users to make informed trade-offs. I appreciate the honest discussion added to the manuscript acknowledging this limitation and pointing toward future architectural improvements.\n3.Limited Task Scope (Isotropic/High-SR): The additional experiments demonstrating the transferability of FluoResFM to isotropic reconstruction and super-resolution with higher scale factors (3x, 4x, 8x) are excellent. The results in Fig. 6e-j and Supplementary Fig. 7 effectively demonstrate that the model can be adapted to these relevant tasks via fine-tuning, substantiating the claim of it being a \"foundation\" model.\n4.3D Data Processing: The implementation of \"slice-by-slice\" and \"multi-to-multi\" strategies for 3D restoration and surface projection addresses the critical need for volumetric analysis. While a fully 3D network remains a future direction, the 2.5D strategy (multi-channel 2D input) presented in Fig. 6a-d shows clear improvements over simple projection methods and proves the model's utility for biological 3D data.\n5.Evaluation Metrics: The incorporation of Decorrelation analysis for resolution estimation and SQUIRREL analysis (RSE and RSP) significantly improves the biological meaningfulness of the evaluation. The updated results in Supplementary Fig. 1 (Fig. R16) provide a more robust validation of the method beyond standard computer vision metrics.\nMinor Comments\n1.Fig. 5d Caption: The clarification regarding the dashed line representing the single-sample fine-tuned baseline has been added.\n2.Fig. 6c (now 7c) Readability: The grouping of samples by structure type and the use of colors in the x-axis labels have improved the readability of the figure.\n3.Network Hyper-parameters: The inclusion of Supplementary Table 6 with detailed layer-wise hyper-parameters satisfies the reproducibility requirement."}, "author_response": ""}]

Decision letter