Orochi: Versatile Biomedical Image Processor
Authors: Gaole Dai
Chenghao Zhou
Yu Zhou
Rongyu Zhang
Yuan Zhang
Chengkai Hou
Tiejun Huang
Jianxu Chen
Shanghang Zhang
Venue: NeurIPS 2025 spotlight ()
Keywords: life science
biology
microscopy
foundation model
Primary area: machine_learning_for_sciences

Abstract: Deep learning has emerged as a pivotal tool for accelerating research in the life sciences, with the low-level processing of biomedical images (e.g., registration, fusion, restoration, super-resolution) being one of its most critical applications. Platforms such as ImageJ (Fiji) and napari have enabled the development of customized plugins for various models. However, these plugins are typically based on models that are limited to specific tasks and datasets, making them less practical for biologists. To address this challenge, we introduce **Orochi**, the first application-oriented, efficient, and versatile image processor designed to overcome these limitations. Orochi is pre-trained on patches/volumes extracted from the raw data of over 100 publicly available studies using our Random Multi-scale Sampling strategy. We further propose Task-related Joint-embedding Pre-Training (TJP), which employs biomedical task-related degradation for self-supervision rather than relying on Masked Image Modelling (MIM), which performs poorly in downstream tasks such as registration. To ensure computational efficiency, we leverage Mamba's linear computational complexity and construct Multi-head Hierarchy Mamba. Additionally, we provide a three-tier fine-tuning framework (Full, Normal, and Light) and demonstrate that Orochi achieves comparable or superior performance to current state-of-the-art specialist models, even with lightweight parameter-efficient options. We hope that our study contributes to the development of an all-in-one workflow, thereby relieving biologists from the overwhelming task of selecting among numerous models. Our pre-trained weights and code will be released.

Orochi: Versatile Biomedical Image Processor
Gaole Dai
Chenghao Zhou 1.2
Yu Zhou
Rongyu Zhang!
Yuan Zhang
Chengkai Hou
Tiejun Huang
Jianxu Ca jianxu isas.de
Shanghang Zhangle
shanghang@ @ pku.edu..cn
State Key Laboratory of Multimedia Information Processing. School of Computer Science. Peking University
2 Academy for Advanced Interdisciplinary Studies, Peking University Leibniz-Instiuu für Analytische Wissenschaften ISAS e. V.
Abstract
Deep learning has emerged as pivotal tool for accelerating research in the life sci- ences, with the low-level processing of biomedical images (e.g., registration, fusion, restoration, super-resolution) being one its most critical applications. Platforms such as Imagel (Fiji) and napari have enabled the development of customized plug- ins for various models. However, these plugins are typically based on models that are limited to specific tasks and datasets, making them less practical for biologists. To address this challenge. we introduce Orochi, the first application-oriente. effi- cient, and versatile image processor designed overcome these limitations. Orochi is pre-trained on patches/volumes extracted from the raw data of over 100 publicly available studies using our Random Multi-scale Sampling strategy. We further pro- pose Task-related Joint-embedding Pre-Traine Training (TJP), which employs biomedical task-related degradation for supervsuervn rather than relying on Masked Image Modelling (MIM), which performs poorly in downstream tasks such as registration. To ensure computational efficiency, we leverage Mamba's 's linear computational complexity and construct Multi-head Hierarchy Mamba. Additionally, a three-tier fine-tuning framework (Full, Normal, and Light) and demonstrate that Orochi achieves comparable or superior performance to current state-of-the-art specialist models, even with lightweight parameter-effie efficciet options. We hope that our study contributes the development of an all-in-one workflow, thereby relieving biologists from the overwhelming task of selecting among numerous models. Our pre-trained weights and code will be released.
provide
Introduction
With the rapid advancement of deep learning, modern neural networks have demonstrated remarkable scalability and spawned wide array of downstream applications in AI for Life Science M| 2 3 Among these, biomedical image processing is pivotal topic. Its significance arises from the inherent
Equul Contribution, scCorresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Registration
Restoration
Super- Ressl-Rettttiinn
tata from Study
Raaddr
Natisodle
Dataset
OASIS
CARE
HBAA
Coopallier
ConvexAdam (TMI 2014)
Dice BL.33
X
BSAFusion /AAAI
Diic
QUA
X
Mamba
UnIFMIR (Noture Methods 2024)
27/40
26.08
Ja mebeing Fre trae
VCM (MACV 2025)
x
PArK
21.53
(Light)
73.63
0.24
20:30
33.50
Three Tire-Tering
Orochi (Best)
Diee
0.41
Dut
PSAR
PORR
d Train Frees
Figure 1: Trend of Versatile Biomedical Image Precessor. We listed the recent advancements in biomedical image processing, where matched row-to-column colour coding highlights the main task of each model. Stickers display the reported scores from the respective papers. Orochi extends the versatile bandwidth and exhibits exceptional performance across tasks and tuning modes.
constraints in acquiring biomedical images compared to natural images, which often compromise source image quality. Specifically, the most common limitations stem from imaging device operational trade-offs.. For instance, in optical microscopy, excessive laser intensity can damage target tissues, while insufficient laser power introduces low signal-to-noise ratios [4]. Similarly, in computed tomography (CT), thinner slice scans subject patients prolonged high radiation exposure, posing health risks, whereas sparse slicing results in low-resolution data [5]. These challenges drive the demand for biomedical image restoration 6 7 g g and super-resolution (10] [1 [3 [
tasks. Another class of limitations originates from the intrinsic shortcomings of imaging modalities. For example, CT imaging efficient and provides clear hierarchical information but suffers from poor soft-tissue contrast, in contrast, magnetic resonance imaging (MRI) excels in soft-tissue resolution but requires longer acquisition times and susceptible to motion artifacts. Such modality-speeiff weaknesses necessitate biomedical image fusion tasks [14] 15 16.
17
18
19
20
21
[2] 23
24
25 Furthermore, acquiring synchronous multi-modal data imposes demands on equipment and environments, making asynchronous data more prevalent. However, misalignment exists between
tissues or cells due to temporal/specimen variability, motivating image registration 26 27 28 29
30
31
32 tasks to align asynchronous or even heterogeneous datasets of the same specimen.
33
With the emergence of long-range dependency models [34 35 36 and self-supervised pre-training methods 37 38 39 40). models designed for the aforementioned issues have advanced rapidly, giving rise powerful specialist models (see Figure D. However, we argue that practical applications, these specialist models neglect three critical factors: (1) Task Perspective: Real- world biomedical imaging tasks often require multiple sequential steps (e.g., registration followed by fusion, as discussed carlier). (2) Degradation Perspective: Since the underlying causes of degradation share similarities, these degradations are interrelated for example, both low signal-to-noise ratio and low resolution result in information loss. (3) Data Perspective: Due to their characteristics of being multi-channel, large-scale scale, and high-throughput, biomedical images are considerably larger than natural images, making the training and inference of multiple specialist models highly inefficient. From both efficiency and effectiveness standpoints, these issues collectively motivate the development of universal foundational model. We aim for such generalist model to optimize the aforementioned challenges by: (1) handling diverse low-level tasks within a unified framework, thereby avoiding the difficulties of selecting and integrating several specialist models; (2) capturing more generalized and robust features via cross-task learning during the pre-training phase; and (3) addresses real-world biomedical data processing costs to reduce redundant training and inference.
Therefore, we introduce Orochi (named after the legendary multi- melti- serpent). To fulfill the envisioned goals, our design emphasizes four aspects (see Figure (2): (1) Dataset Level: We extensively employ unlabeled raw data from over 100 publicly available studies (see Appendix (A.2) and perform our Random Multi-scale Sampling, which considers the different scales of Region- ontenettsst (ROI). (2) Pre-training Level: Inspired by Joint-embedding Prediction Architecture (JEPA) [40], where different degradations serve as context for each others. Our Task-related Joint-

---

Row Sata Sampleed
12 Cambo
Macked
Delormed
Halty
Low-daz
NISC Single cell Image Dadanet
HIP-CT
Public Regesitory of imane Balikett
Humman Organ
Adaas
Lccl
JAN JITTS
B2
Raw naw-level Dataset, 5D dimensionality, MB-TE level file size
Multi modal, Multi dimensionality Dataset with Randoms Nulti scale Sampling (RMS)
Errceder
Beccoder
Helgy
Rustoree:
Pioe
Master
Ricennd
Suparcnsshitien
Tasle Regioraation
Task-related Joint- boint-ambedding Pre: ree-training (TJP) on Multi Multi-head Hierarchy Mamba2 (MHH)
Figure 2: Overview of the Construction of Orochi. The upper panel illustrates the data conversion pipeline, taking into account patches/volumes at multiple scales. The lower panel presents the self- supervised strategies utilized during pre-training. Additionally, we provide supplementary images (e.g., Visible A+B, Error Map) to facilitate the comparison between inputs and outputs.
embedding Pre-training (TJP) applies various forms of task-specific degradation, and the model learns from reconstructing them jointly. (3) Model Level: On one hand, we employ Mamba [1]
as the building blocks to leverages its linear complexity 42 41
the other hand, the
overall structure draws inspiration from the hierarchical design of the Swin-Transformer (36) by incorporating patch merging to enhance model efficiency further. (4) Post-training Level: We propose three-tier fine-tuning framework to reduce the tuning cost. Ranging from full fine-tuning (Full), to fine-tuning only the replaced dense convolution head (Normal), and finally to the most lightweight variant using depth-wise separable convolution 45 (Light), thereby achieving Parameter- Efficient Fine- Fine-Tuning Tuning (PEFT) [46]. To this end, we hope that Orochi will distinguish itself as an exceptional tool among the extensive array of plugins available platforms such as ImageJ (Fiji) [47 and napari [88] further advancing towards a user-friendly workflow with unified functionalities.
In summary, our main contributions are follows:
1. We systematically review the significance of low-level biomedical image processing and highlight that even this era of powerful foundational models, the paradigm centred on spe- cialist models still exhibits inherent deficiencies. Limiting both effectiveness and efficiency from the perspectives of task, degradation, and data. To the best our knowledge, Orochi is the first versatile foundational model addressing these issues.
2. We curated raw-level datasets from over 100 studies 49
covering wide range of imaging modalities from 2-5D - with a total data size over 100 terabytes. During training, we introduce Random Multi-scale Sampling to achieve unitedly training patches/volumes These converted data are used for both local and stream training, alleviating the challenges with the transmission and storage of extremely large datasets.
50
51
data conversion into
3. We propose Teskere Joint-embedding Pre-training (TJP), which directly learns the inter- relations among various task-specific degradations rather than relying common Masked Image Modelling (MIM). For the model architecture, we leverage the linear complexity of Mamba and design multi-head hierarchical structure minimize the costs of training and inference. Finally, for post-training, we introduce a three-tier fine-tuning framework and demonstrate that even the most lightweight depth-separable convolution tuning can achieve performance comparable to existing statc-of-ther specialist models.

---

2 Related Works
Self-supervised Learning Self-supervised Learning (SSL) extracts inherent data properties. Masked Image Modelling (MIM) predicts masked image regions from original pixel values us- ing an encoder-decoder architecture, with loss in image space [7 38 Contrastive Learning (CL) aligns representations of augmented views of the same image in an embedding space via specialized objectives [52] 39]. Combining these, the Joint-Embedding Predictive Architecture (JEPA) [40]
predicts full latent representations from context to learn robust image representations.
Restoration To address low image quality in fluorescence microscopy. Content-aware image restoration (CARE) [4] uses CNNs. Li et al. [8] improved axial resolution using a CARE-based model with physically acquired ground truth. Subsequent works integrated Swin- Transformers [36] for efficiency (SwinIR [9] or Mamba blocks 42] for long-range dependency modeling (MambalR (7)). UniFMIR 6 demonstrated that pre-traines trained foundation models generalize well for this task.
Super-resolution Super-resolution aims overcome optical limits. DeepLP [53]
employs point-
scanning for reconstruction. Diffusion-based models, including volumetric conditioning modules [S] and latent diffusion InverseSR [11], show promise for 3D brain MRT. Other approaches include local implicit image functions for flexible resolution enhancement [3] joint super-resolution and synthesis
frameworks for isotropic volumes 12]. and methods for multimodal image super-resolution [0]]
Registration Image registration aligns images by optimizing a deformation field. VoxeIMorph [54] provides learning-based 3D framework. Dual-encoder U-Nets (26), Swin-Transformer for long-
distance correspondences (TransMorph (32). and Mamba blocks 42 for efficient long-range model-
ing (MambaMorph (3). Fast 3D registration methods have been proposed by Siebert er al. [2
while Mok et al.
30 28 address large deformations with Laplacian Pyramid Networks.
Fusion Multi-modality image fusion integrates complementary information. Techniques include bidirectional stepwise feature alignment for unaligned images (BSAFusion 255). mutual enhance- ment for PAT/MRI fusion [24), and diffusion-bain based methods incorporating fusion priors (Diff-
IF (55)) or denoising diffusion models [6]] Semantic-aware strategies with registration are found
in SuperFusion (21) and MURF 22 Other notable methods encompass one-stage progressive
dense registration (23), U2Fusion 14 and Equivariant fusion [5]. Diverse strategies also include
lightweight and semantic-guided approaches (ALMFNET (17). MSGFUSION (18), dietionary-basee
and GAN-driven frameworks 19 [6]. and unsupervised methods 20)
Methods
Due to page limitations, this section primarily emphasizes our comprehensive degradation designs used for self-supervision. The Appendix provided detailed architecture of the Multi-head Hierarchy Mamba model along with the three-tier fine-tuning framework
3.1 Preliminary: Self-supervised Degradation
Self-supervised image learning be generally formulated as learning reconstruction function fo that recovers the original image x from its degraded ((x). Formally, this objective is defined as:
min E, ((,(o)
(I)
where is the sampled data, Pdata D(-) denotes degradation function applied to x, fo is the parameterized model, and is loss function (e.g.. the L2 loss or perceptual loss).
For masked image degradation, the degradation function defined as: Dimes @ M, where M E {0, 1) is binary mask with height H and width W that selectively occludes regions of z. This degradation helps the model learn to infer missing information.
Masked Image
Deformed Image For deformed image degradation, the degradation function takes the form: (ar T(x), where T(-) represents a spatial transformation (such as rotation, scaling, or warping). This degradation introduces geometric distortions that mimic real-world variations.

---

Nosiy Image For noisy image degradation, the degradation function is defined as: D Dosiso + 7) where 77 denotes additive noise (typically Gaussian noise). simulating sensor imperfections or environmental interference.
For Low-resolution image degradation, the degradation function is given by: (xx =b (z), where 4e is down-sampling operator with scale factor s, reducing the resolution of x to simulate the effects of low-resolution imaging.
Low-resolution Image
3.2 Orochi: Random Multi-scale Sampling
Random Multi-scale Sampling aims extract patches/volumes with diverse scales from raw images. Given a raw image I, the procedure consists of two main steps: (1) Multi-scale Resizing: We first generate scaled versions of the raw image capture features at different resolutions. particular, we resize 7 to scales 1/2 and 1/4 of its original size. Formally, let:
=4+ (T), f1.
where Jss (-) denotes down-sampling with factor s. (2) Random Window Sampling: For each scaled image Is we define fixed-size window K (compatible with the pre- requirements in either 2D or 3D) and perform random sampling to extract sub- patches. Let the window K have dimensions W x (or W H D for 3D data). A randomly sampled 2D patch Vg at scale s is given by:
Is T (i :
1, j:
where (i, j) is randomly chosen starting coordinate in I.. Collectively, the set of patches extracted across scales is represented as:
{Ia. {1, 1 }),
NoJ.
where N, denotes the number of patches sampled from the image at scale These multi-scale patches are then passed to subsequent degradation processes (c.g., masking, deformation, noise addition, and low-resolution conversion). By performing random sampling across multiple scales, method extended the data diversity and enabled more robust feature learning across various datasets.
3.3 Orochi: Task-related Joint-embedding Pre-training
Dual-Masking Reconstructive Fusion To better address the biomedical image fusion task, where the combination of existing contexts is crucial, we modified the conventional Masked Image Mod- elling approaches (37 3X), which typically employ a single masking strategy. Specifically, we applied two distinct masking operations to the training data thereby generating two independent masks:
FA =-=x00 ② MA EB @ MB.
(5)
where MA. e {0, 1xxw are binary masks with only partial overlap and ensure invisible information retention even after fusion. The masking probabilities are generated by:
[a(i. = 16f <],, AE
(6)
where ct; ~ U(, 1) represents a random value extracted from a uniform distribution for grid coordinates i, j. and is the masking threshold. The key innovation is that model is exposed to process both masked inputs (I,A, FB) simultaneously to recover the original image: ă = fo (x. TB), This guides the model develop robust feature extraction capabilities that identify complementary information across different masked views, and then fuse these partial observations coherently to reconstruct missing regions in both inputs.
Spatially-varying Gaussian down-sample For down-sampling, we adapt similar principles from DeepLP (53), which tested noisy down-sampling beyond uniform down-sampling in self-supervised microscopy restoration. We enhance this noisy down-sampling with spatially varying characteristics:
(iR(#)= Govar(ty (la (x + n))),
5

---

where te represents down-sampling with a random scale factor ?o denotes upsampling back to the original resolution, 7 N(0 down)) is normal distributed noise added during the down- sampling process with down ~ U0.001, 0.1), U represent uniform distribution, and Gøvar denotes matially-varying Gaussian filtering. can be defined as:
[ovar(x))L.]] = Σ (owss) - u v),,
(8)
U( mmax) ) that varies across grid coordinates i, j. This mimics the heterogeneous blurring found in optical systems.
where Jo represents a Gaussian kernel (2/3D) with standard deviation o(v,
Multi-scale Smoothed Perlin Noise Deformation For the self-supervised registration task, con- structing realistic deformation field is important. We conducted multi-scale Perlin noise fields that simulate the hierarchy variations in natural anatomical structures. Given an image x, we generate a deformation field & and its corresponding deformed image (aer(a:) ( as follows:
(9) -) a spatial transformation operator, (-) denotes pratially-varyinng Gaussian smoothing with parameter Ø, and Per(f, p) represents multi-octave Perlin noise with frequency f and persistence The multi-octave Perlin noise is specifically defined as:
Dar(2)) = T(x, 4) = G, (Per(f,p))
Per p)
Σ p
(10)
where S(.) is the simplex noise function, N is the number of octaves and coords represents the grid coordinates. This multi-scale approach generates deformation fields with varying levels of detail.
To enhance the anatomical plausibility of the deformations, we apply normalization and bound it using a tanh function: O Pinal a tanh( where a controls the maximum displacement magnitude.
Multi-stage Noise Simulation
To simulate realistic noise, we adopted a multi-stage process:
Dunise (t) (Poi((o(ma +)));
(11)
where 17
N(0, o with noise (0.0775. 0.15) represents Gaussian noise, Poi(A) denotes Poisson noise with intensity parameter À (modeling mhoton-counting statistics). and Bip represents binary (salt-and-pepa -ppppep noise that affects proportion of pixels with probability p.
These sophisticated degradation designs enable framework simulate wide spectrum of real- world imaging artifacts, encouraging the model to handle diverse image quality issues encountered.
Experiments
We conducted comprehensive comparisons strictly following the setups in published specialist models (UniFMIR (6). VCM (5). Transmorph [32] and BSAFusion (25], see Appendix [A.4 for details). Resulting in more than 30 state-of-the-art baselines across multiple benchmarks for varrous biomedical image-processipg tasks to demonstrate the effectiveness and versatility of Orochi. We color-coded the performance in Table 1 3. and 4 with Red (1st), Blue (2nd), and the row color reflects the training type with [Training free Traiiig from Scratch], [Fine- Tunnng,, and [Efficent Fine-Tuning]. See the Appendix for more details of the experiment setups and extra validation C
Generalization Capability on In-Domain Data Given that our model, Orochi, is extensively pre- trained, we expect exhibit strong generalization capabilities on in-domain data. Accordingly, in Figure 3 we demonstrate Orochi's 's zero-shot performance on various stained microscopy images [5]] (results on clinical images (50) are detailed the Appendix C.T). Panels (A)(D) illustrate Orochi's robust processing capabilities. In Panel (E), we further examine whether these outcomes align with our algorithmic expectations. For example, our Dual-Masking Reconstructive Fusion anticipates that the model learns an effective fusion strategy and leverages the existing information from both

---

(A) Fution
[R) Restoration
[E] Case Study
[C]: Registration
(P) Super Resolution
V206 145
Figure 3: In-Domain Generalization Performance of Orochi on Unseen Test Images. (A)-(D) illustrate Orochi's robust performance across various low-level processing tasks when applied to unseen testing images after pre-training. Supplementary images include the dual-masking images and naive merge results for the fusion task. Error maps for the registration task. (E) provides in in-depth case studies: for the fusion task, the centromere count emphasized both the reconstructed image and the original image (highlighted with circles); for the registration task, subtle deformations of the cell membrane are accentuated; and for the restoration and super-resolution tasks, the fine details of bright-field fieldffed images and the internal structures of DNA-stained cell nuclei are emphasized
Table Isotropic 3D volume Restoration Task. Low-high laser data pairs along the XY axis are collected and serve the training set. However, the evaluation is on both XY slices and XZ slices.
Table 2: MRI Axial Super- Task.
Two intensities of low-resolution data are trained and tested in this task, with 4mm (i.e x4 down- sampled) and 8mm (i. x8 sown-sampled)
Machal
PSNR
SSIM (XY)
PSNR (Z2)
SSIM
Methed
PSNR (dmm)
SSIM (4mm)
PSNR (kmm)
SSIM (3mm)
Detawe: CARE
Dataset: HRA
57
Li st.
23.71
24.11
Cuisic
23.84
0.76
21.50
0.63
CARB
25.00
0.60
25.26
0.64
0.59
30.91
0.1
25.98
26
0.06
062
Mambuik
25.89
0.64
27.17
065
LIFE S
01
0.95
25.1 12
031
UnIFMIR
27.12
0.66
27
0.66
InservSR T1
27.51
0.58
33.66
0.79
peane UIFMRR (PP32)
26.18
0.63
25.24
0.54
VCM 3
0.57
0.86
18:31
28.52
Orechi (Pull)
35.31
0.95
31,85
0.89
Orochi (Nocnall
29:15
29.43
(Nomma)
54
0.96
39.51
0.89
Oroehti rlught
20.77
6.71
2938
6.73
sources before reconstruction, rather than reconstructing masked regions separately. This expectation validated in the Centromere count case study, where masked A and each exhibit a partial absence of centromeres, and the model successfully performs complementary fusion and the final reconstruction does not arbitrarily generate Centromeres across extensive background regions. This precise control over fine structural details is also evident in other cases.
Image Restoration Task Table we present the performance of Orochi on the isotropic 3D volume restoration task. In microscopy imaging, the image quality along the XY plane is typically much higher than that along the XZ plane due to the inherent limitations of sequential flayerby-layerr imaging, such as in light-sheet microscopy, which leads to the formation of isotropic data. To address this, CARE 4 leverages the high-resolution XY data for training and subsequently restores the lower-resolution XZ data. On this task, Orochi not only significantly outperforms train-from-scrateee models like SwinIR 36 (+2.11 PSNR) and MambalR 7 (+ PSNR), but also comprehensively surpasses pre-trained foundation model UniFMIR 6 across both fully fine-tuned (+0.85 PSNR) and efficiently fine-tuned (+3.19 19 PSNR) configurations. An intriguing finding that our results indicate Orochi with PEFT leads the list. This outcome is plausible given that the dataset, derived from
7

---

Table Inter-patient Brain Registration Task.
Table CT-MRI Fusion Task. Volumetric MRI and CT data are sent jointly to the model, recon- structing a single fused result. This fused result would be compared with both MRI and CT input for similarity calculation (e.g. SSIM).
During training, the model goal is to input paired MRT data from distinct patients and output predie- tion of the registration flow. This flow is applied to the corresponding segmentation data to calcu-
late the dice loss. Thereby, regional deformation can be learned with supervision.
Qabl t
Qcv J
Method
HD95
SDlogJ
U2Fusion
10
0.32
6.580.80
0.41
OASIS
EMMA
15
0.29
6,695.80
1.18
Initial
3,86
ALMFoet
0.29
7,200.50
1.27
et
80.00
1.77
MsgEusion
0.19
7,090.40
0.29
Siebert et
221
81.00
1.63
0.07
MDHU
0.22
7,417.60
1:23
Mok er al.
28
82.00
1.67
0.07
UMF-CMGR 20
0.25
4,638 70
1.35
PIMed RIT
78,76
1,86
0.06
SuperFusion
P1
0.28
4,828.90
0,97
LapIRN
82.18
1.67
0.08
MURF 22
5,554.60
23
Transmorph-B
81.62
1.69
0.12
PAMRFuse
241
0.09
5,408.00
0.18
Transmorph-1.
82.22
1.66
0.12
BSAFusioo 25
0.39
4,155. 10
1.38
Mambamorph
33
81.81
1.66
0.09
DDFM L6
0.26
5,981.40
1.31
Orochi (Pull)
83.62
1.60
0.11
Oruchi (Full)
2,381.57
Orochi (Nomal))
1.65
Ornchi (Nomaal)
0.37
2,519.36
1.48
(Light)
1.73
Orochi (Light)
0.34
2,461.41
1.43
isotropic data pairs, comprises fewer than 100 total training patches. Consequently, Full Fine- Tinn-nunin or training from scratch is prone to over-fitting, (see Appendix C.3
for extra comparisons)
We next evaluated the image super-resolution capabilities of Orochi (see Table 2 Early super-resolution models typically rely on CNN-based architectures such as UniRes [0] and SynthSR [12), which are efficient yet often lack sufficient expressiveness and gener- alization ability. LIIF 13 leverages the power Implicit Neural Representations (INR) to perform implicit interpolation; however, the high training cost associated with INR limits its adaptability to real-world scenarios. More recent approaches, including Inverse [1] and VCM (5), based on powerful pre-trained Brain-Latent Diffusion Models (LDM) (59 to overcome these shortcomings. In this setting, Orochi significantly outperforms all the aforementioned architectures. At an 8mm slice thickness, Orochi achieves PSNR that is 4.01 points higher than InverseSR and 2.76 points higher than VCM. These gains demonstrate that among pre-trained models, Orochi's pre-training is markedly superior to that of Brain-LDM, both in terms of the pre-training data and purpose.
Image Super-resolution Task
Image Registration Task We further evaluated the registration task using the dataset from Lear2Reeg [60] (see Table 33. this task, brain MRI images from different patients (i.e.. inter- patients) are registered (see Appendix Appendix(C.2ffor for patient-to-atlas brain registration test), and the model's ability to handle subtle deformations assessed by measuring the similarity of the segmented brain regions after registration (e.g. Dice). Biomedical image registration has evolved from CNN- based 229 30 27 to Transformer-bast
architectures [32 (44 with even lomear-complexity models emerging in recent work. In comparison to these methods, our approach achieves Dice scores that are 2.42 points higher than ConvexAdam, 2.0 points higher than Transmorph, and 1.81 points higher than Mambamorph.
such as Mamba 33
Image Fusion Task Finally, as illustrated in Table we evaluated Orochi's performance on the image fusion task. Recent trends in this domain have integrated image registration as an auxiliary task to facilitate fusion, as demonstrated by methods such as BSAFusion [25]. UMF-CMGR (20). MURF [22]. and SuperFusion [2I] Although these models typically exhibit limited registration capabilities (see Appendix C.2 this aligns with our pursuit of developing versatile, comprehensive
model. Compared with the recent advanced model BSAFusion, Orochi outperforms on all evaluated
metrics, achieving improvements of +0.02 in Qans. -1803.53 in Q.v. and +0.07 in SSIM. Combined
with our sfate-of-the-art performance on the registration task, these results establish Orochi as the first model in this domain to achieve such performance.

---

High (GT)
Registration
Super-resolution
Figure 4: Visualization Comparison to Recent Advances. Concise visualization for each task is provided. To enhance comprehension, arrows have been incorporated to facilitate evaluation.
Registration
Super--rsolutioon
$2.00
15.00
30.00
10000
130
Permmee Coumt
Teramete Court
30000
01 prid prue jenz)
Figure 5: Vine-Tuning Efficiency VS Performance. We present the training parameter efficiency of various models alongside their corresponding results. The three-tier fine-tuning results for Orochi (Full, Normal, Light) are illustrated using a gradient green colours, from deep to light, and are connected by green dashed lines to indicate the trend. Other baselines are shown in the legend.
Visualizations In Figure 4 we provide qualitative results of Orochi. Specifically, Orochi demon- strates a superior capability in handling subtle degradations. (see Appendix C. for more)
Table 5: Pre-train Strategies VS Performance. Datasets and setups remain the same as those experiments in the previous sections.
Strutegy
Registration (Dice
Fusion (Qaa
Restoration (PSNR
Super- Resolution (PSNR ?)
MAR 337 (Single Mask) I.JEPA 40 (Dual Mask)
71.22
0.36
26.67
29.17
69.97
28.81
Orochi (TJP)
83.62
0.41
29.88
33.63
Ablation Study Comparison to Other Pre-train Strategies Table 5 we demonstrate the limitations of relying solely Masked-imas wal--iiaag--ollliiii (MIM), particularly in registration tasks. Additionally, we observe that the dual-masking approach employed in I-JEPA 40) underperforms compared Orochi. We hypothesize that this is because chunk masking is more advantageous for high-level tasks rather than the low-level focus of our study.
Ablation Study Larger 3 Better, Fine- Time-Tuning Efficiency V.S Performance As shown in Fig- 5] the number trainable parameters is not decisive factor for downstream tasks pisks-pariccularly in data-limited scenarios such as biomedical imaging. In many cases, opting for Sarrmmter-Efficiiee Fine-Tuning (using only 1-2% of the total parameter count) prevents overfitting and achieves both efficient and effective results.
5 Conclusion
We introduce Orochi, the first versatile biomedical image processor designed for low-level tasks. To enhance effectiveness, propose Random Multi-scale Sampling, which is scalable way to

---

leverage raw data from wide range of studies. The extracted data is then processed through our Task-related loint-embedding Pre- Pre-training (TJP), where a unified and robust embedding is learned from various task-related degradations. For efficiency, we developed Multi-head Hierarchy Mamba and provide a three-tier fine-tuning framework (Full, Normal, and Light). These design choices ensure high efficiency during pre-training, post-tuning, and test inference. Our experiments demonstrate that Orochi exhibits in-domain generalization capability across multiple tasks and achieves state- of-the-art performance compared to specialist models with efficient fine-tuning (less than 5% of total parameters), This suggests that constructing generalist image processor may lie more in the diversity of the dataset and the pre-training strategy than in increasing the model size naively.
6
Acknowledgements
G.D. was supported by the National Natural Science Foundation of China (Grant No. W2442028). S.Z. was supported by the National Science and Technology Major Project (Grant No. 2022ZD01 17800). The work of ISAS was supported by the "Ministerium für Kultur und Wissenschaft des Landes Nordrhein-Westfalenn and "Der Regierende Bürgermeister von Berlin, Senatskanzlei Wissenschaft und Forschung." " The work was further supported by the Bundesministerium für Forschung, Tech- nologie und Raumfahrt, BMFTR under the funding reference 1611.0272.
References
[1] Shanghang Zhang, Gaole Dai, Tiejun Huang, and Jianxu Chen. Multimodal large language models for bioimage analysis. nature methods, (1(8):1390- 1,390-1333, 2024.
[2] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan Krumholz, Jure Leskovec, Eric Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259 265229992266222222 2023.
[3] Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
[4] Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siån Culley, al. Content-awann image restoration: pushing the limits of fluorescence microscopy. Nature methods, 15(12)10900 1097, 2018.
[5] Suhyun Ahn, Wonjung Park, Jihoon Cho, Seunghyuck Park, and Jinah Park. Volumetric conditioning module to control pretrained diffusion models for 3d medical images. arXiv preprint arXiv:2410.21888 2024.
[6] Chenxi Ma, Weimin Tan, Ruian Hc, and Bo Yan. Pretraining a foundation model for generalizable fluorescence microscopy- based image restoration. Nature Methods, 2168):1558-1567, 2024.
[7] Hang Guo, Jinmin Li, Tao Dai. Zhihao Ouyang, Xudong Ren, and Shu-Ta Xia. Mambair: simple baseline for image restoration with state-space model. In European conference on computer vision, pages 222-241. Springer. 2024.
[8] Xuesong Li, Yicong Wu, Yijun Su, Ivan Rey-Suarez, Claudia Matthacus, Taylor B Updegrove, Zhuang Wei, Lixia Zhang, Hideki Sasaki, Yue Li, et al. Three-dimensional structured illumination mieroscopy with enhanced axial resolution. Nature biotechnology, 41 (9): (9):1337713119 307-1319, 2023.
[9] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference computer vision, pages 1833-1844, 2021.
[10] Mikael Brudfors, Yael Balbastre, Parashkev Nachev, and John Ashhurner. A tool for super-resolving multimodal clinical mri. arXiv preprint arXiv:1909.00 140, 2019.
[11] Jueqi Wang, Jacob Levman, Walter Hugo Lopez Pinaya, Petru-Danie Tudosiu, Jorge Cardoso, and Razvan Marinescu. Inversesr: 3d brain mri super-resolution using a latent diffusion model. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 438-447. Springer, 2023.
[12] Juan Eugenio Iglesias, Benjamin Billot, Yačl Balbastre, Azadeh Tabari, John Conklin, R Gilberto González, Daniel C Alexander, Polina Golland, Brian L Edlow, Bruce Fischl, et al. Joint synthesis of 1 mm isotropic mp-rage volumes from clinical mri exams with scans of different orientation, resolution and contrast. Neuroimage, 18206. 2021.
and
10

---

[13] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning contiquous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition. pages 0628-8638. 2021.
[14] Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, and Haibin Ling. U2fusion: A unified unsupervised image fusion network. IEFE transactions on pattern analysis and machine intelligence, 44( 44(1):502-518 2020.
[15] Zixiang Zhao, Haowen Bai, Jiangshe Zhang, Yulun Zhang, Kai Zhang, Shuang Xu, Dongdong Chen, Radu Timofte, and Lue Van Gool. Equivariant multi-modality image fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 25912-25921, 2024.
[16] Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Lue Van Gool. Ddfm:: denoising diffusion model for multi-modality image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1082-8093 2023.
[17] Pan Mu, Guanyao Wu, Jinyuan Liu, Yuduo Zhang, Xin Fan, and Risheng Liu. Learning to search a lightweight generalized network for medical image fusion. IEEE Trunsactions on Circuits and Systems for Video Technology, 34(7):5921-5934, 2023.
[18] Jinyu Wen, Feiwei Qin, Jiao Du, Meie Fang, Xinhua Wei, CL. Philip Chen, and Ping Li. Msgfusion: Medical semantic guided two-branch network for multimodal brain image fusion. IEEE Transactions on Multimedia, 26:944-995 9009444-977-00 2023.
[19] Yuchan Jie, Xiaosong Li, Haishu Tan, Fugiang Zhou, and Gao Wang. Multi-modal medical image fusion via multi-dictionary and truncated huber filtering. Biomedical Signal Processing and Control, 88: 105671, 2024.
[20] Di Wang, Jinyuan Liu, Xin Fan, and Risheng Liu. Unsupervised misaligned infrared and visible image fusion via cross modslityy image generation and registration. arXiv preprint arXiv:2205. 1876, 2022.
[21] Linfeng Tang, Yuxin Deng, Yong Ma, Jun Huang, and Jiayi Ma. Superfusion: A versatile image registration and fusion network with semantic awareness. IEEE/CAÁ Journal of Automaticu Sinica, 912):2121-211 2022.
[22] Han Xu, Jiteng Yuan, and Jiayi Ma. Murf: Mutually reinforcing multi-modal image registration and fusion. IEEE transactions pattern analysis and machine intelligence, 45( 10): 12111-114821 12-16166, 2023.
[23] Di Wang, Jinyuan Liu, Long Ma, Risheng Liu, and Xin Fan. Improving misaligned multi-modality image fusion with one-stage progressive dense registration. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
[24] Yutian Zhong, Jinchuan He, Zhichao Liang, Shuangyang Zhang, Qianjin Feng, Wufan Chen, and Li Qi. Performance of medical image fusion in high-level analysis tasks: A mutual enhancement framework for unaligned pat and mri image fusion. arXiv preprint 4rXiv:2407.03992, 2024.
[25] Huafeng Li, Dayong Su, Qing Cai, and Yafei Zhang. Bsafusion: A hidirectional stepwise feature alignment network for unaligned medical image fusion. arXiv preprint arXiv:2412.0880 2024.
[26] Jinxin Lv, Zhiwei Wang, Hongkuan Shi, Haobo Zhang, Sheng Wang, Yilang Wang, and Qiang Li. Joint progressive and coarse-to fine registration of brain mri via deformation field integration and non-rigid feature fusion. IEEE Transactions on Medical Imaging, 41(1))27788 2202228888882222222 2022.
[27] Hanna Siebert, Lasse Hansen, and Mattias P. Heinrich. Fast 3d registration with accurate optimisation and little learning for lear22reg 2021, 2021.
[28] Tony CW Mok and Albert CS Chung. Conditional deformable image registration with convolutional neural network. In Medical Image Computing and Computer Assisted Intervention-MIICAN 2021: 24th International Conference, Strasbourg, Frunce, September 27-October 1, 2021, Proceedings, Part IV 24, pages 35 45. Springer, 2021.
[29] Hanna Siebert, Christoph Großhröhmer, Lasse Hansen, and Mattias P Heinrich. Convexadam: Self- configuring dual--ppimisstiion-be based 3d multitask medical image registration. IEEE Transactions on Medical Imaging, 2024.
[30] Tony CW Mok and Albert CS Chung. Large deformation diffeomorphic image registration with laplacian pyramid networks. In Medical Image Computing and Computer Assisted Intervention- MICCAI MideCA 2020: 23rd International Conference, Lima, Peru, October 4 8, 2020, Proceedings. Part III 23, pages 211-20 211-221. Springer, 2020.
11

---

[31] B Golestani. The learn2reg 2021 miccai grand challenge (pimed team). Biomeedical Image Registration, Du- main Generalisation and -iutt---Dittibbttionn Analysis: MICCAI 2021 Challenges: MIDOG 2021, MOOD 2021, and Learn2Reg 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27- October 1, 2021, Proceedings, 13166:168, 2022.
[3] Junyu Chen, Eric C Frey. Yufan He, William Segars. Ye Li, and Yong Du. Transmorph: Transformer for unsupervised medical image registration. Medical image analysis, 82:102615. 2022.
[33] Tao Guo, Yinuo Wang, and Cai Meng. Mambamorph: a mamba- backbone with contrastive feature learning for deformable mr-ct registration. arXiv e-prints, pages 17Xiv22401, 2024.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[35] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. 100.119999 2020.
[36] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IECE/CVF international conference computer vision, pages 10012-10022, 2021.
[37] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pallern recognition, pages 16000-16009, 2022.
[38] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9653 066399663,,- 2022.
[39] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby. et al. Dinov2: Learning robust visual features without supervision. arXiv preprint ar arXiv:2304.07199 2023.
[40] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self- learning from images with joint sentt-embediing predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 15619-15629, 2023.
[41] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint arXiv:2405.21060, 2024.
[42] Albert Gu and Tri Dao. Mamba: Lincar-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
[43] Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Mattco Grella, et al. Rwkv; Reinventing mns for the transformer era. Xrxiv preprint 2rv:: 13048, 2023.
[44] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Efficient attention: Attention with linear complexities. Proceedings of the TEFE/CVF winter conference on applications of computer vision, pages 3531-3539 2021.
[45] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
[46] Gaole Dai, Yiming Tang, Chunkai Fan, Qizhe Zhang, Zhi Zhang, Yulu Gan, Chengqing Zeng. Shanghang Zhang, and Tiejun Huang. Discovering long-term effects on parameter efficient fine-tuning. arXiv preprint arXiv:2409.0677 06700, 2024.
[47] Caroline A Schneider, Wayne S Rasband, and Kevin W Elicciri. Nih image to imagej: 25 years of image analysis. Nature methods, 9(7))7 2012.
[48] Napari Contributors. napari: a multi ciltiiddimesionall image viewer for python. Zenodo https://doi. mr2//0.528//eendoo 3555620, 2019.
[49] Eleanor Williams, Josh Moore, Simon W Li. Gabriella Rustici, Aleksandra Tarkowska, Anatole Chessel, Simone Leo, Bálint Antal, Richard K Ferguson, Ugis Sarkans, et al. Image data resource: a bioimage data integration and publication platform. Nature methods, 14(8):775-78.. 2017.
12

---

[50] Claire L Walsh. P Tafforrea, WL Wagner, DJ Jafre,, A Bellier, C Werlein, MP Kühhell E Boller, S Walker- Samuel. JL Robertus, al. Imaging intact human organs with local resolution of cellular structures using hierarchical phase-contrast tomography. Nature methods, 15322):55 2021.
[51] Matheus Viana, Jianxu Chen, Theo A Knijnenburg, Ritvik Vasan, Calysta Yan, Joy E Arakaki, Matte Bailey, Ben Berry, Antoine Borenszteju, Eva M Brown, et al. Integrated intracellular organization and its variations in human ips cells. Nature, 613(7743):345 355-3344333.3434..0 2023.
[52] Mathilde Caron. Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021.
[53] Linjing Fang. Fred Monroe, Sammy Weiser Novak, Lyndsey Kirk, Cara R. Schiavon, Seungyoon Blenda
Yu. Tong Zhang, Melissa Wu, Kyle Kastner, Alaa Abdel Latif, Zijun Lin, A. Shaw, Yoshiyuki Kubota, John M. Mendenhall, Zhao Zhang. Gulcin Pekkurnaz, Kristen M. Harris, Jeremy Howard, and Uri Manor. Deep learning- based voint-scanning super-resolution imaging. Nature methods, 18:406 416, 2019.
[54] Guha Balakrishnan, Amy Zhao, Mert R. Sabuncu, John Guttag, and Adrian V. Dalca. Voxelmorph: A learning framework for deformable medical image registration. IEEE Transactions on Medical Imaging. 38(8):1788-1800 August 2019.
[55] Xunpeng Yi, Linfeng Tang, Hao Zhang. Han Xu, and Jiayi Ma. Diff-if: Multi-modality image fusion via diffusion model with fusion knowledge prior. Information Fusion, 110: 102450, 2024.
[56] Jingxue Huang, Xiaosong Li, Haishu Tan, and Xiaoqi Cheng. Generative adversarial network for trimodal medical image fusion using primitive relationship reasoning. IEEE Journal of Biomedical and Health Informatics, 2024.
[57] D Summers. Harvard whole brain atlas: www. med. harvard. edu/aanlib/home. html. Journal of Neurology, Neurosurgery Psychiatry, 74(3)288-228,, 2003.
[58] Elizabeth M Sweeney, Russell Shinohara, Navid Shice, Farrah J Mateen, Avni A Chudgar, Jennifer L. Cuzzocreo, Peter A Calabresi, Dzung L Pham, Daniel S Reich, and Ciprian M Crainiccanu. Oasis is automated statistical inference for segmentation, with applications to multiple sclerosis lesion segmentation in mri. Neurolmage: clinical, 2:402-413, 2013.
[59] Walter HI Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro F Da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso. Brain imaging generation with latent diffusion models. In MICCAI Workshop on Deep Generative Models, pages 117-126. Springer, 2022.
[60] Alessa Hering, Lasse Hansen, Tony CW Mok, Albert CS Chung, Hanna Siebert, Stephanic Häger, Annkristin Lange, Sven Kuckertz, Stefan Heldmann, Wei Shao, et al. Learn2reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the of deep learning. IEEE Transactions on Medical Imaging, 42(3):697-712, 2022.
[61] Brian B Avants, Charles I. Epstein, Murray Grossman, and James C Gee. Symmetric diffeomorphic image registration with cross-correlation evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis, 12(1):26-4) 12,2264462222 2008.
[62] Marc Modat, Gerard R Ridgway. Zeike A Taylor, Manja Lehmann, Josephine Barnes, David J Hawkes. Nick C Fox, and Sébastien Ourselin. Fast free-form deformation using graphics processing units. Computer methods and programs in biomedicine, 98(3):278-284, 2010.
[63] M Faisal Beg, Michael Miller, Alain Trouvé, and Laurent Younes. Computing large deformation metric mappings via geodesic flows of diffeomorphisms. International jourraa of computer vision, 61:139-157, 2005.
[64] Mattias P Heinrich, Oskar Maier, and Heinz Handels. Multi-modal multi-atlas segmentation using discrete optimisation and self-similarities. Visceral Challenge@ @ ISBI, 1390:27, 2015.
[65] Boah Kim, Dong Hwan Kim, Scong Ho Park, Jieun Kim, June-Gon Lee, and Jong Chul Ye. Cyclemorph: cycle consistent unsupervised deformable image registration. Medical image analysis, 71:102036, 2021.
[66] Huaqi Qiu, Chen Qin, Andreas Schuh, Kerstin Hammernik, and Daniel Rucckert. Learning diffcomorphic and modallty-invarian registration using b-splines. In Medical imaging with deep learning, 2021.
[67] Junyu Chen, Yufan He, Eric C Frey, Ye Li, and Yong Du Vit-v-net: Vision transformer for unsupervised volumetric medical image registration. arXiv preprint 2rXiv:2104.06466 2021.
13

---

[68] Wenhai Wang, Enze Xie. Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVE international conference on computer vision, pages 568-578, 2021.
[69] Yutong Xie. Jianpeng Zhang, Chunhua Shen, and Yong Xia. Cotr: Efficiently bridging and transtormer medical image segmentation. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1. 2021, Proceedings, Part III 24, pages 171-180. Springer, 2021.
[70] Hong- Yu Zhou, Jiansen Guo, Yinghao Zhang, Xiaoguang Han, Lequan Yu, Liansheng Wang, and Yizhou Yu. nnformer: volumetric medical image segmentation via a 3d transformer. IEEE transactions on image processing, 32:4036 4045, 2023.
[71] Chang Qiao, Di Li, Yuting Guo, Chong Liu, Tao Jiang, Qionghai Dai, and Dong Li. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nature methods, 18(2):194 202, 2021.
[72] Bin Xia, Yucheng Hang, Yapeng Tian, Wenming Yang, Qingmin Liao, and Jie Zhou. Efficient non-local contrastive attention for image super-resolution. In Proceedings the AAAI conference on artificial intelligence, volume 36, pages 2759-2767, 2022.
[73] Kang Xu, Austin R Graves, Gabrielle Coste, Richard L Huganir, Dwight E Bergles, Adam S Charles, and Jeremias Sulam. Cross-modality supervised image restoration enables nanoscale tracking of synaptic plasticity in living mice. Nature Methods, 20(6):935-944, 2023.
[74] Chang Qiao, Li, Yuting Guo, Chong Liu, Tao Jiang, Qionghai Dai, and Dong Li. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nature methods, 194))2194202, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions are well justified with comprehensive theoretical and experi- mental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide the full set of assumptions and a complete (and correct) proof
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide condensed implementation in the experiment section and a detailed description in the Appendix, with code submitted in the supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: A Readme.md file is attached along with the code submitted in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/pil wwwppbblccgguiicuceccccc /eegguddsss/Ceebbbinninn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the implementation details are included in the Appendix section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
Our experiments are conducted with a set random seed 42.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://uuur..............tttttta
Answer: [Yes]
Justification:
We make sure preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper currently does release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

A Experiment Setups
A.1 Baselines
Registration
Ly et al. [26]:
Uses a dual-encoder U-Net for coarse-to-fine registration.
Siehert et al.
27
Proposes fast 3D) registration approach.
Mok et al.
28
Employs conditional deformable convolutions.
PIMed 31
From the Leam2Reg challenge.
LapIRN
30
Uses a Laplacian pyramid network for large deformations.
ConvexAdam 29 : Adopts dual-optimization strategy.
TransMorph 32
Based on a Transformer architecture for capturing long-range correspon-
dences.
MambaMorph
33
Utilizes mamba blocks for efficient long-range dependency modeling.
Fuison
U2Fusion 14:
Provides a unified unsupervised fusion approach.
EMMA
13
Employs equivariant learning for fusion.
ALMENet
Searches for a lightweight generalized fusion network.
MsgFusion
18
Uses a semantic-guided two-branch network.
MDHU 19 Uses multi-dictionary learning with truncated Huber filtering.
UMF-CMGR 20
Adopts cross-modality generation and registration.
SuperFusion 21:
Combines registration and fusion with semantic awareness.
MURF 22 Reinforces multi-modal registration and fusion mutually.
IMF
23
Improves fusion with a progressive dense registration strategy.
PAM
24
Focuses on feature alignment.
Fusiin
25
Adopts bidirectional stepwise feature alignment.
DDFM 16
Utilizes a denoising diffusion model for fusion.
Super-Resolution
Cubic: Bicubic interpolation traditional baseline.
UniRes
10
Designed for super-resolving multimodal clinical MRI.
SynthSR Performs joint super-resolution and synthesis.
LIIF 13
Learns continuous image representations for implicit interpolation.
InverseSR [|]: Uses a latent diffusion model for 3D brain MRI super-resolution.
VCM 5
Applies volumetric conditioning module.
Restoration
Li al.
Improves axial resolution.
CARE
Uses content-aware network for fluorescence microscopy image restoration.
SwinIR
Employs a Woin--rranformer for efficient image restoration.
MambalR
Utilizes mamba blocks for modeling long-range dependencies.
Fine-tunes a pre-trained foundation model for generalizable fluorescence microscopy-based restoration (with pruned FP 1PF63333 variants).
UniFMIR 6
22

---

6x
58
100
Figure Preview of the Metadata List of Studies
A.2 Datasets
Pre-train
A combined multi- multi-modal biomedical image dataset aggregated from over 100 public studies,
encompassing various imaging modalities and degradation types [51] [0 49]. In Figure 6 we provide preview of the metadata of the studies we covered (Excel Form would be included in the Zip file). Since RMS method highly scalable, plan to further update this list the future and explore the borderline.
Registration
The OASIS brain MRI dataset from the Learn 2021 challenge, used to evaluate the overlap of segmented regions and the smoothness of the deformation fields 660 58).
Fuison
A CT-MRI paired fusion dataset (VIFB), which assesses the integration of complementary information across modalities [57].
Super-Resolutee Resolution
The Harvard Whole Brain Atlas (HBA), providing high-quality MRI images for evaluating
low-resolution image reconstruction [57]
Restoration
The CARE microscopy image dataset, used to evaluate the enhancement of low signal-to-
noise ratio fluorescence microscopy images 4
A.3 Metrics
Registration
Dice similarity coefficient: Computed
2|Ar n B Dice
[A] + [B
23

---

which measures the overlap between the segmented regions.
95th percentile Hausdorff Distance (HD95): Defined as the 95th percentile of the distances between boundary points of the segmented regions.
Standard deviation of the log-Jacobian determinant (SDlogJ): Calculated as the standard deviation of log( (dee(())) where is the Jacobian matrix of the deformation field. This metric reflects the smoothness of the deformation field [60]
Fuison
QAF (Qahr): Measures the quality of the fusion by evaluating the consistency between the fused image and the input modalities.
Qcv (Qwy): Assesses the contrast consistency across the fused image.
Structural Similarity Index (SSIM): Computed based comparisons of luminance, contrast, and structure between source images [25
Super-Resolution
Peak Signal-to-Noise Ratio (PSNR): Calculated as
MAXi MSE
PSNR R 10
where MAX, is the maximum possible pixel value and MSE the mean squared error between the reconstructed and reference images.
SSIM:
Evaluates perceptual similarity between the super-resolved and reference images 9
Restoration
PSNR:
As above, it measures the pixel-level fidelity between the restored image and the high-quality reference.
SSIM: Measures the structural similarity between the restored and reference images
A.4 Code-base
Pre-train
Adapted from public GitHub implementationss of the Swin-Transformer and Transmorph.
Swin-Transformer: http:///ghhhh..ccooaaacaa/t///ta
[36]:
Transmorph: https://github.c http://ghhhhaooo///uo//////////a Tran:ooforprraaaaafofommr.. for Mor_Medical_ mmage_Registration
Registration
Implemented based on the Transmorph GitHub code. Link: https://github..om com/
nunycchee244//Taaaaaoppp... wwwppphhhhraaaafffrmmmmmomm ormer MedicaMdlll_____ MetoMMeeeaallmmmaeeeeeeigiiaaii
32
Fuison
Built with reference the BSAFusion GitHub code. BDr1123/BSAFusS.n 25
Link: https://github.cr
Super-Resolution
Implemented based on GitHuu codes of InverseSR and VCM. InverseSR: https //github. com www./oom////eeeeeee
VCM: https: com/
Ahn-Ssu/ (5).

---

Restoration
Based on the UniFMIR GitHub implementation. Link: https://gittubb.cmm/ comm/cm112 /
UNi FMIR (6]
25

---

Figure 7: Model architecture of Multi-head Hierarchy Mamba (MHM). The model contains unified hierarchical Mamba encoder with replaceable decoder (e.g. regular convolution head or wist separable convolution head) for tuning
B Experiment Configurations
Multi-head Hierarchy Mamba Three-Tier Fine-Tuning Framework Figure presents comprehensive diagram of Orochi's backbone architecture. Post-tuning, the interchangeable decoder can be replaced as required. We evaluated Orochi's 's performance using our Three-Tier Tine-Tuning Framework, which includes full fine-tuning (Full, 100% parameters), regular convolution head with the encoder frozen (Normal, 10-30% parameters), and depth-wise separable convolution head (45) with the encoder frozen (Light, less than 5% parameters). The optimal results were achieved across all three tiers, underscoring the significance of selecting an appropriate tuning method based on specific requirements.
pre-trained Orochi-B (3D version) with the configuration listed in Table The 2D version has similar configuration, with slight differences some setups (e.g. batch size). We have 2 two sets pre-tree pre-training devices. The A800 80Gx8 device is used for local pre-train and the H100 40Gx8 device is for streaming pre-train.
Pre-train
We followed the same setups as our code base for each task (see Section ??), including the tuning resolution, epoch number, optimizer configurations and loss designs. The device we use for tonitunnng is NVIDIA 4090 24Gx4
Fine-tuning
26

---

Table 6: Pretraining Configuration Parameters for Orochi-B
Category
Parameter
Value / Range and Description
img_size size
(32, 224, 224)
patch_size
4
pat_merg_nr
2
in_chans
2
embed_ding _im depths
128
Model / Enccder
(4, 4, 4, 4)
drop_path_rate
0.2
if_convakip
True
out_indices
(0, 2)
ssm_cfg
None
norm_epsilon
X 10
initializer_ce _fgg
None
True
Mamba
rms_norm
True
residual_in_fp31
True
patch_nore
True
use_checkpoint
False
decoder_bb
False
decoder
ponoddr_ddpthhpparbbb..
False
Decoder
decoder_head_che chan
64
batch_size
12
1r
0.0005
weight_decay
0,01
warmup_ratio
0.1
Training
varmup_staat__factor
0.01
max_epoch
50
Optimizer / Scheduler
AdamW; WarmupCo- sine with cycles=0.5
Registration Flow Scaling
tanh(-) 0.66 (applied to the deformation field)
Registration Gaussian Sigma Range
[1.5,3.5]
Perlin Noise Octaves
4
Perlin Noise Persistence
0.5
Deformation & Augmentation
Mask Ratio
0.5
Downsampling Scale Factor Range
[0.25, 0.75)
Downsampling Noise Level Range
[0.01, 0.1]
Downsampling Gaussian Sigma Range
[0.25, 1.0]
Gaussian Noise Level Range
[0.075, 0.15]
Salt vs. Pepper Ratio
0.5
Salt & Pepper Noise Amount Range
[0.01, 0.05]
Grid Image Parameters
Grid spacing
4.
Line width =1
27

---

C Extra Results
C.1 Zero-shot Processing Biomedical Images
(A) Fusion
(B) Restoration
Masked
Nacked
Voible A&B
Fused
Noiey
(C) : Registration
(D) : Super- Resoluten
Deformed
Error Map
Eroor Hap
Registrated
GT
Low Res
Upsampled
Nisalinn
Align
Figure 8: Visualization of Orochi Zero-shot Processing to Different Degradation. (A)-(D) panel shows the ability on four tasks respectively, with the dotted line separating medical images (top) and microscopy images (bottom).
In Figure we present additional results demonstrating Orochi's 's zero-shot performance on both mieroscopy and medical images. Notably, Orochi yields satisfactory outcomes when faced with extremely severe degradation, as exemplified in panel (B), row 2, and panel (C), rows 2 and 3.
C.2 Registration Fuison
Fusion Model on Registration Task In Figure 9 we demonstrate that, despite the recent trend of pre-registration before fusion, these methods remain predominantly fusion-oriented and are not well- suited for addressing real-world registration tasks within the medical image registration community. However, Orochi represents a significant advancement in versatility, as is designed not only for this specific scenario but also to achieve superior performance across all registration tasks.
28

---

Dataset: OASIS
SuperFusion (JAS22) Dice: 0.66
MURF (TPAMI23) Dice: 0.67
BSAFusion (AAAI25) Dice: 0.69
Orochi (Ours) Dice: 0.81
Figure 9: Performance of Fusion-oriented methods on specialized registration benchmark. We provide the inputs (Move, Fix), output (flow), and evaluation data (Move/Fix seg and Deformation Grid) for each visualization.
SPECT-MRI
PET-MRI
U2Fusion
U2 F2usson
IM MMMC
MURE
RSAFusion
AMPRuuse
Orochi
Crochi (Norma))
Orochi (Normal)
(Light)
0.5
1.5
2
1.5
Figure 10: Fusion Comparison with Brain SPECT&MRI/ PET&MRI Data.
Patient to Atlas Brain Image Registration The regional deformation is learned unsupervised in Table Only the image of the atlas and the patient's brain would be used for loss calculation while training. Then we evaluate the dice score between the segmentation maps of these two brains. Since Orochi pre-trained in this unsupervised fashion, it shows excellent adaptation to this task, similar to the case with supervision.
SPECT-MRI & PET-MRI Image Fusion In Figure 10 performed comparative evaluations using state-of-the-art fusion techniques on two additional Harvard Whole Brain datasets obtained from httpt::t////tt med. harvard. edu/ aanlib/ These datasets specifically focus on the fusion of SPECT and PET imaging with MRI. The results demonstrate that Orochi outperforms recent advancements such BSAFusion and maintains superior efficiency.
29

---

Table 7: Patient to Atlas Brain Registration Task. During the training phase, the model aims to input paired MRI data from both the standard brain atlas and patient scans, output predicted registration flow. This flow is subsequently applied to the atlas data to compute the similarity between the registered atlas and the patient's scan. During the testing phase, the predicted flow is applied to the atlas brain segmentation map, and the Dice coefficient is evaluated against the patient's 's brain segmentation map.
Method
Dice
% 1g S
Dataset: IX/ 32
Affine
0.386 = 0.195
SyN
61
0.645 0.152
< 0.0001
NiftyReg
0.645 0.167
0,020
LDDMM
65
0.680 = 0.135
S 0.0001
dead<BCV
64
0.733 0.126
0.147 0.050
VoxeIMorph-1
54
0.729 # 0.129
1,590 0,339
VoxeIMorph-2
0.732 0.123
1.522 = 0.336
VoxelMorph diff
0.580 0.165
CycleMorph
65
0.737 + 0,123
1.719 & 0,382
MIDIR
0.742 0.128
0,0001
ViT-V-Net 67
0.734 0.14
1.609 = 0.319
PVT 68
0.727 1 0.128
1,858 0.314
0.735 = 0.135
1.292 0.342
nnFormer
70
0.747 = 0.135
1.595 0,358
Trans Bayes
0.753 0.123
1.560 0.333
TransMorph-diff'
32
0.594 = 0.163
0.0001
'rrms-orph--spl
0.761 0.122
0.0001
Trrph
0.754 : 0.124
1.579 0.328
Orochi (Full)
0.770 : 0.120
1.592 0.334
Orochi (Normal)
0.765 + 0.121
1.571 0.323
Orochi (Light)
0.752 + 0,126
1.499 = 0.301
30

---

8..0
26.33
$8.00
28.00
BR.GZ
Drsee
Cheth Shat
Depeii Iteena
Gribi Sagio
Figure 11: Stress Test BioSR Benchmark. We trained Orochi on four subsets concurrently, thereby reducing the cost associated with hyperparameter searching. The results were compared against baselines that involved separate hyperparameter searches for each subset. The bar chart illustrates the average PSNR values, while the lines, colour-coded according to the legends, indicate the performance metries for each respective subset.
C.3 Super-Resolution & Restoration
Stress test on joint multi-modal data image repairing In this additional validation, we aim to evaluate Orochi's 's performance under stress using an extended benchmark. The BioSR [71 benchmark comprises four distinct categories of microscopy image pairs (x2 low/high imaging quality), captured by multimodal structured illumination microscopy (SIM) system, encompassing Clathrin-Coated Pits (CCPs), Endoplasmic Reticula (ERs), Microtubules (MTs), and F-actin Filaments. Specifically, Orochi was trained on all four datasets concurrently, whereas the baseline 772 73
74 6 models
were trained separately on each dataset. This deliberate approach highlights Orochi's capability in cesource-constrained environments, where conducting hyperparameter searches for each subset is not feasible. As illustrated Figure [I despite the training constraints imposed on Orochi, an absolute improvement is still observed, further demonstrating its capability and efficiency.
D Limitations
Two limitations in our paper remain unaddressed at present. First, due to constraints on computational resources and group size, we were unable further investigate the scaling law of our method during pre-training. This limitation also indicates that our focus was restricted to low-level tasks as presented the paper. However, we firmly believe that unified model for life sciences, capable of excelling in both high-level understanding tasks and low-level generation tasks, will emerge in the future. This is also an emerging trend that has already demonstrated progress in general applications.
31


Reviews:
[{"review_id": "qBmK8Hehzt", "reviewer": "Reviewer_S4FJ", "summary": "This paper presents a pretraining method and suite of models for a variety of biomedical image processing tasks such as restoration, super-resolution, and fusion. Orochi is trained on an extremely large (>14M) and diverse collection of medical images with an efficient, multi-scale representation learning approach tailored for downstream image processing tasks. When fine-tuned on downstream tasks, Orochi outperforms state-of-the-art foundation models and specialist models specifically tailored for individual modalities and tasks.", "questions": "1.\tHow does Orochi perform when fine-tuned on downstream (discriminative) tasks like classification? I understand that this does not fit under the umbrella of \"low-level processing\", but have the authors conducted such experiments? I would hypothesize it might perform well on dense tasks like segmentation given its multi-scale pretraining.", "limitations": "Limitations are addressed in the appendix.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n- The presentation quality is among the highest I have seen while reviewing for NeurIPS. Typesetting is used effectively to guide the reader’s attention and break up passages of text. Figures are dense, informative, and visually appealing\n- The paper is both organized and written clearly. Prior work and important background are thoroughly explained. \n- Experimental efforts are comprehensive, with evaluation across many unique tasks, datasets, and modalities. Moreover, performance is compared against state-of-the-art specialized baseline approaches\n- Open-source release of code and pre-trained weights should make this a major contribution to the field\n\n**Weaknesses**:\n- I see no major weaknesses. However, I am curious to see how Orochi performs on downstream discriminative tasks like classification or even segmentation.\n\n**Minor comments**:\n- I originally misinterpreted the word “study”. In the context of medical imaging, this can often effectively mean “exam”; thus, “100 studies” seemed to be a very small amount. Context clues clarified this confusion quickly, but I would suggest potentially changing the wording here. There is probably a better way to clarify that this is an aggregate of many existing datasets.\n- Related to the above point, I would try to highlight the “size” (e.g., # images) of the composite pretraining dataset as early as possible to convey the scale and dispel any confusion\n- Figure 1: While funny (to me, at least), I’m not sure “Crappifier” is appropriate\n- L172: Change “down-sample” -> “Down-sampling”\n- L201 is not a complete sentence: “Resulting in more than 30…” perhaps simply remove the period and write “resulting”\n- Table 5: The title is confusing, “Pre-train Strategies V.S Performance”. Should this read “Vs.” as in “versus”?\n- Figure 5: Visually appealing, but difficult to read. Perhaps a different font/color choice is needed or the figure should be enlarged.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal and am encouraged to see strong results on downstream discriminative tasks. As indicated in my original review, I believe this is an excellent submission and will maintain my original rating of 6 (Strong Accept). I congratulate the authors on their great work and want to reiterate the importance of an open-source codebase and model for the research community.", "rebuttal": "We would like to extend our sincerest gratitude to the reviewer for the insightful feedback and high praise for our manuscript. We are truly honoured that our work is considered **outstanding among many excellent submissions.** A great deal of effort was dedicated to structuring the narrative, creating the figures, and designing the experiments, all with the goal of clearly and efficiently communicating the focus and contributions of our work within the limited space. It is incredibly **encouraging to have our efforts recognized, regardless of the final outcome.**\n\nWe have carefully considered all the comments and provide the following point-by-point responses:\n***\n**Q1:**\n> I originally misinterpreted the word “study”. In the context of medical imaging, this can often effectively mean “exam”; thus, “100 studies” seemed to be a very small amount. Context clues clarified this confusion quickly, but I would suggest potentially changing the wording here. There is probably a better way to clarify that this is an aggregate of many existing datasets.\n\n**A1:**\nThank you for this valuable suggestion. We used the term “study” because our primary data source, the Image Data Resource (IDR) **[1]**, uses this term to describe the original user-uploaded source data. Currently, the resource contains 138 studies, which comprise 14,042,467 images and a total of 414TB of data. To prevent ambiguity, we will avoid using the term “study” when describing other data sources in our future versions of the manuscript.\n***\n**Q2:**\n> Related to the above point, I would try to highlight the “size” (e.g., # images) of the composite pretraining dataset as early as possible to convey the scale and dispel any confusion.\n\n**A2:**\nThis is an excellent point. We agree and will be sure to emphasize the scale of the dataset in future versions of the manuscript.\n***\n**Q3:**\n> Figure 1: While funny (to me, at least), I’m not sure “Crappifier” is appropriate.\n\n**A3:**\nThank you for your feedback. The term “Crappifier” was adopted from the *Nature Methods* publication, ***\"Deep learning-based point-scanning super-resolution imaging\"*** **[2]** which systematically investigates the suitability of various synthetic noise and artifacts for real-world microscopy image restoration and super-resolution. This paper was a significant reference for us when integrating self-supervised methods for these tasks. However, your point is well-taken; given that our work extends beyond image restoration and super-resolution, a more neutral term such as “Converter” would indeed be more appropriate. We will make this change in our revision.\n***\n**Q4:**\n> L172: Change “down-sample” -> “Down-sampling”\n> L201 is not a complete sentence: “Resulting in more than 30…” Perhaps simply remove the period and write “resulting”\n\n**A4:**\nThank you for your meticulous reading. We will correct these in the next version.\n***\n**Q5:**\n> Table 5: The title is confusing, “Pre-train Strategies V.S Performance”. Should this read “Vs.” as in “versus”?\n\n**A5:**\nThank you for pointing this out. Your understanding is correct, and we will revise the title accordingly.\n***\n**Q6:**\n> Figure 5: Visually appealing, but difficult to read. Perhaps a different font/colour choice is needed, or the figure should be enlarged.\n\n**A6:**\nWe appreciate this feedback. The page limit constrained the current size of the figure for the ablation study. We will revise it for better readability in a future version of the manuscript where more space is available.\n***\n**Q7:**\n>I see no major weaknesses. However, I am curious to see how Orochi performs on downstream discriminative tasks like classification or even segmentation. (Weakness #1)\n\n>How does Orochi perform when fine-tuned on downstream (discriminative) tasks like classification? I understand that this does not fit under the umbrella of \"low-level processing\", but have the authors conducted such experiments? I would hypothesize it might perform well on dense tasks like segmentation, given its multi-scale pertaining. (Question #1)\n\n**A7:**\nWe conducted new tests on the model's performance on BTCV **[3]**, REFUGE **[4]**, and CadVidSet **[5]** segmentation datasets. We strictly adhered to the training hyperparameter settings of MedSAM2 **[6]** and carried out a complete fine-tuning following its GitHub repository. Baseline results are from the original paper. \n### Dice Score\n| Model | REFUGE-2D | CadVidSet-2D | BTCV-Aorta-3D | BTCV-Liver-3D | \n| :--- | :--- | :--- | :--- | :--- | \n| TransUNet | 0.363 | 0.316 | **0.920** | 0.969 | \n| Swin-UNet | 0.289 | 0.226 | 0.892 | **0.975** | \n| nnUNet | 0.349 | 0.418 | 0.877 | 0.948 | \n| SAM2 | 0.558 | 0.539 | 0.835 | 0.861 | \n| MedSAM2 | 0.799 | **0.861** | 0.896 | 0.865 | \n| Orochi | **0.811** | 0.857 | 0.911 | 0.950 |\n\nAlso, we conducted classification tasks on TissueMNIST (2D) and OrganMNIST (3D) datasets. All experimental setups are from the MedMNIST series papers **[7] [8] [9]**. \n\n| Methods (TissueMNIST-2D-224^2) | AUC | ACC | Methods (OrganMNIST-3D-28^3) | AUC | ACC |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| ViT-Base | 0.948 | 0.729 | ResNet-18+3D | **0.996** | **0.907** |\n| MAE-Base | 0.945 | 0.719 | ResNet-50+3D | 0.994 | 0.883 |\n| CLIP-Base | 0.922 | 0.662 | ResNet-50+2.5D | 0.974 | 0.769 |\n| EVAv2-Base | 0.940 | 0.703 | Auto-sklearn | 0.977 | 0.814 |\n| DINOv2-Base | 0.944 | 0.715 | AutoKeras | 0.979 | 0.804 |\n| Orochi | **0.950** | **0.733** | Orochi | 0.995 | 0.891 |\n\nWe demonstrated the capability of Orochi to adapt to high-level tasks; however, we aim to avoid overstating its application. Our objective remains focused on the development of a unified low-level processor, as stated since the abstract of our paper.\n***\n**Reference**\n\n**[1]** Williams, Eleanor, et al. \"Image Data Resource: a bioimage data integration and publication platform.\" Nature methods 14.8 (2017): 775-781.\n\n**[2]** Fang, Linjing, et al. \"Deep learning-based point-scanning super-resolution imaging.\" Nature methods 18.4 (2021): 406-416.\n\n**[3]** Landman BA, Xu Z, Igelsias JE, Styner M, Langerak TR, and Klein A, \"MICCAI multi-atlas labeling beyond the cranial vault workshop and challenge,\" (2015)\n\n**[4]** Orlando, José Ignacio, et al. \"Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs.\" Medical image analysis 59 (2020): 101570.\n\n**[5]** Wang, Lu, et al. \"Coronary artery segmentation in angiographic videos utilizing spatial-temporal information.\" BMC medical imaging 20.1 (2020): 110.\n\n**[6]** Zhu, Jiayuan, et al. \"Medical sam 2: Segment medical images as video via segment anything model 2.\" arXiv preprint arXiv:2408.00874 (2024).\n\n**[7]** Yang, Jiancheng, et al. \"Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification.\" Scientific Data 10.1 (2023): 41.\n\n**[8]** Yang, Jiancheng, Rui Shi, and Bingbing Ni. \"Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis.\" 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021.\n\n**[9]** Doerrich, Sebastian, et al. \"Rethinking model prototyping through the MedMNIST+ dataset collection.\" Scientific reports 15.1 (2025): 7669."}, {"review_id": "3CTdUxXr8C", "reviewer": "Reviewer_Lxhu", "summary": "This paper pretrains a foundation model for low-level biomedical image processing tasks, including restoration, super-resolution, registration and fusion. Because of the high heterogeneity of biomedical images (regarding image dimension, modality, imaging subjects and body parts, etc,), there are a large body of methods dealing with different tasks in different settings and it is usually challenging for researcher to find the proper tool for there specific need. Therefor, it is valuable to develop a unified model that can perform different tasks in different settings. This paper collect a large body of biomedical data from 100 studies, and develops a self-supervised pre-training strategy suitable for low-level image processing task. The pretrained model is then fine-tuned and compared to a set of specialist models on each task and achieves comparable or superior performance.", "questions": "please respond to the weakness points", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1.The motivation is reasonable and it is valuable to develop a unified model that can be applied to different biomedical image processing tasks.\n\n2.The paper is well written and easily to follow.\n\n3.This study collected and curated a relatively large body of biomedical data for pretraining.\n\n4.The proposed pre-training strategy seems suitable for low-level processing tasks. \n\n\nWeakness\n1.Actually, the practical value of low-level tasks is relatively low, especially when comparing to high-level tasks, such as classification, segmentation, detection. Though there are commonly used metrics for evaluating low-level tasks, their practical meaning is more demonstrated in downstream high level tasks. For example, image fusion methods are usually evaluated in downstream detection/classification tasks.  \n\n2.For each of the four tasks, evaluation is only performed on one or very small number of datasets. We know that because of the high heterogeneity of biomedical images, the performance on one dataset can hardly generalize to all other datasets with slightly different settings. The evaluation on each task is far from being enough to show that the proposed method is generally applicable for the specific task type under different settings.\n\n3.For the image fusion task, only three metrics are used to compare different methods, which is too small. Current image fusion studies typically utilize much more metrics.\n\n4.The training strategies of the proposed methods and the baselines are different in many experiments, which may make the comparison unfair.\n\n5.For the image registration task, there are a large set of different setting which bring different kinds of difficulty. Such as, registration between 2D images or 3D volumes, registration between different modality pairs or within the same modality, registration of different body parts with significant or not deformation, etc. Much more experiments are needed to conclude that a model can be generally applied to image registration task.", "was_revised": true, "final_justification": "My major concern about the insufficient validation has been clarified.", "rebuttal": "We thank the reviewer for recognizing that our **motivation is reasonable**, the paper is **well-written and easy to follow**, and that our **training methodology, data usage, and workload are appropriate** for the topic. Despite the divergence in the final score, we would like to provide a detailed point-by-point response to address your concerns and hopefully resolve any outstanding questions.\n***\n**Q1:**\n> Actually, the practical value of low-level tasks is relatively low, especially when comparing to high-level tasks. Though there are commonly used metrics for evaluating low-level tasks, their practical meaning is more demonstrated in downstream high level tasks. \n\n**A1:**\nWe agree that high-level tasks are a vital part of AI for Life Science, but we **do not agree this supports the conclusion that low-level tasks are subordinate to high-level ones**. Purely low-level processing tasks command significant attention within the community. For eaxample, starting from early efforts like Content-aware image restoration (CARE) **[1]**, which focused on fluorescence microscopy restoration and was published in *Nature Methods* in 2018, subsequent works such as BioSR **[2]** (Dataset & Benchmark), DeepLP **[3]** (Self-Supervision Learning), and UniFMIR **[4]** (Foundation Model) have repeatedly appeared in top-tier journals, each with a distinct focus. \n\nFrom a presentation standpoint, a paper should have a clear focus. As we have emphasized since the title, our goal is to develop a versatile biomedical image **processor**, not an **analyzer**. Our work is benchmarked against low-level foundation models like UniFMIR in all aspects, including writing style, experimental setup, and validation methods. In these papers, using **indirect downstream tasks to validate performance is not \"usual\"** and can be **less intuitive**, as the choice of the downstream model can influence a fair comparison of the upstream processing effects. If you could specify which articles you would like us to compare against, we would be more than happy to make improvements.\n\nFurthermore, while works focused on high-level tasks might emphasize downstream performance (e.g., the recent Cellpose 3 **[5]**, which integrates a restoration module to aid segmentation), these are not the most suitable for a direct comparison with our model. Nevertheless, we performed an additional comparison using the same noisy dataset (68 images) from their work and fine-tuned our model on Cellpose3's training set. For further details on our model's performance on high-level tasks, we invite you to see our response to Reviewer S4FJ's Q7, which includes our fine-tuning results on segmentation and classification. We hope this statement clarifies the focus of our work. Thank you for your contribution!\n| Methods | AP@0.5 ↑ | AP@0.75 ↑ |\n| :--- | :--- | :--- |\n| Noisy Image | 0.41 | 0.23 |\n| Noise2void | 0.50 | 0.26 |\n| Noise2self | 0.51 | 0.28 |\n| Cellpose3 | 0.69 | 0.41 |\n| Orochi+Cellpose | **0.71** | **0.44** |\n***\n\n**Q2:**\n> For each of the four tasks, evaluation is only performed on one or very small number of datasets. The evaluation on each task is far from being enough to show that the proposed method is generally applicable.\n\n**A2:**\nWe have included **13 datasets**, covering over **30 baselines** for comparison. At least **4 experiments** are conducted for **each task**. For fairness, we directly compared with recently published works, adopting their experimental setups and data. Specifically:\n\n* For **restoration&super-resolution**\n\nPapers: UniFMIR (*Nature Methods* 2024), VCM (*WACV* 2025), and InverseSR (*MICCAI* 2023)\n\nDatasets: CARE (*Nature Methods* 2018), BioSR-CCPs/ERs/Microtubules/F-actin (*Nature Methods* 2021), HBA (*JNNP* 2003).\n\n* For **registration&fusion**\n\nPapers: Transmorph (*MIA* 2022), ConvexAdam (*TMI* 2024), and BSAFusion (*AAAI* 2025)\n\nDatasets: OASIS (*MICCAI2021*), IXI (*MIA* 2022), VIFB-CT&MRI/SPECT&MRI/PET&MRI (*TMM* 2023).\n\nIn total, the imaging modalities directly covered in our experiments include Light Sheet Microscopy (LSM), Multimodal Structured Illumination Microscopy (M-SIM), CT, SPECT, PET, and MRI, with an even wider variety of imaging content. Objectively, we have done our best to ensure our experimental volume is substantial within the limited space of a NeurIPS submission. **Other reviewers have also acknowledged the breadth of our experiments** (**S4FJ**: *\"Experimental efforts are comprehensive, with evaluation across many unique tasks, datasets, and modalities.\"*; **eeYn**: *\"The effectiveness of the proposed method is validated through comprehensive performance comparisons with a variety of existing models.\"*). If you could point out a specific data type or study you are particularly interested in, we would be glad to include it for validation in a future version.\n***\n**Q3:**\n> For the image fusion task, only three metrics are used to compare different methods, which is too small. Current image fusion studies typically utilize much more metrics.\n\n**A3:**\nWe would first like to clarify that the three metrics were calculated following the methodology of BSAFusion (*AAAI* 2025) and   MSGFusion (*TMM* 2023), and we chose to display three key indicators for formatting and clarity. Since you did not specify which particular metrics or literature you would like us to include, we are providing additional metric results from our earlier experiments here for your reference.\n| Task | Methods | Qabf ↑ | Qcv ↓ | Qssim ↑ | Qvif ↑ | Qs ↑ |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| MRI-CT | BSAFusion | 0.39 | 4155.1 | 1.38 | 0.28 | 0.74 |\n| MRI-CT | Orochi | **0.41** | **2351.6** | **1.39** | **0.30** | **0.76** |\n| MRI-SPECT | BSAFusion | **0.77** | 110.8 | 1.07 | 0.42 | 0.79 |\n| MRI-SPECT | Orochi | 0.75 | **97.9** | **1.41** | **0.44** | **0.80** |\n| MRI-PET | BSAFusion | 0.78 | **67.7** | 1.05 | 0.42 | 0.77 |\n| MRI-PET | Orochi | **0.80** | 120.1 | **1.44** | 0.42 | **0.78** |\n***\n**Q4:**\n> The training strategies of the proposed methods and the baselines are different in many experiments, which may make the comparison unfair.\n\n**A4:**\nWe respectfully disagree with this statement. First, we clearly list our source database, codebase, and baseline scores in Section A for full transparency. Second, for all experiments, our method uses a **unified testing approach**, not a changing one. This involves three fine-tuning strategies: fully fine-tuning, replacing the decoder, and replacing with a lightweight decoder. The detailed framework diagram and experimental setup are provided in Figure 7 and Section B. If possible, we would appreciate it if the reviewer could point out which sentence or section may have led to the misunderstanding of an unfair comparison, and we will be sure to avoid any ambiguity in future revisions.\n***\n**Q5:**\n> For the image registration task, there are a large set of different setting which bring different kinds of difficulty. Such as, registration between 2D images or 3D volumes, registration between different modality pairs or within the same modality, registration of different body parts with significant or not deformation, etc.\n\n**A5:**\nOur model actually has both 2D and 3D versions, and we have already performed various registration tasks in our paper, including 2D/3D, atlas-to-patient, and patient-to-patient registration (Table 3 & 7, Figure 4 & 9). The data we use is inherently multimodal—in addition to the original MRI, it also includes its multi-labeled mask. Following the standard training procedure of works like Transmorph, we fine-tune on this task using two multi-labeled masks for supervised registration learning, while inference is performed directly on the two MRI images.\n\nAs requested, we are supplementing our results with an additional atlas-to-patient registration test on the multi-organ, four-dimensional extended cardiac-torso (XCAT) phantom data **[6]**. We have followed the same experimental setup as Transmorph, and the baseline results are taken from their paper. The final results are as follows:\n### XCAT-CT (16 organs)\n| Model | Dice ↑ | % of &#124;Jφ&#124; ≤ 0 ↑ | SSIM ↑ |\n| :--- | :--- | :--- | :--- |\n| w/o registration | 0.220 ± 0.242 | - | 0.576 ± 0.071 |\n| Affined | 0.330 ± 0.291 | - | 0.751 ± 0.018 |\n| VoxelMorph-1 | 0.532 ± 0.313 | 2.275 ± 1.283 | 0.899 ± 0.027 |\n| VoxelMorph-2 | 0.548 ± 0.317 | 1.696 ± 0.909 | 0.910 ± 0.027 |\n| CycleMorph | 0.528 ± 0.321 | 3.263 ± 1.188 | 0.909 ± 0.024 |\n| TransMorph | 0.604 ± 0.314 | 1.679 ± 0.772 | 0.918 ± 0.023 |\n| Orochi | **0.621 ± 0.355** | **1.398 ± 1.049** | **0.924 ± 0.025** |\n***\n**Reference**\n\n**[1]** Weigert, Martin, et al. \"Content-aware image restoration: pushing the limits of fluorescence microscopy.\" Nature methods 15.12 (2018): 1090-1097.\n\n**[2]** Qiao, Chang, et al. \"Evaluation and development of deep neural networks for image super-resolution in optical microscopy.\" Nature methods 18.2 (2021): 194-202.\n\n**[3]** Fang, Linjing, et al. \"Deep learning-based point-scanning super-resolution imaging.\" Nature methods 18.4 (2021): 406-416.\n\n**[4]** Ma, Chenxi, et al. \"Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration.\" Nature Methods 21.8 (2024): 1558-1567.\n\n**[5]** Stringer, Carsen, and Marius Pachitariu. \"Cellpose3: one-click image restoration for improved cellular segmentation.\" Nature methods 22.3 (2025): 592-599.\n\n**[6]** Segars, W. Paul, et al. \"4D XCAT phantom for multimodality imaging research.\" Medical physics 37.9 (2010): 4902-4915."}, {"review_id": "DR5WuHLetp", "reviewer": "Reviewer_eeYn", "summary": "Orochi is introduced as the first versatile biomedical image processor designed for diverse low-level tasks like restoration, super-resolution, registration, and fusion, overcoming limitations of task-specific specialist models. It employs a novel Task-related Joint-embedding Pre-Training (TJP) on a large, multi-scale dataset and utilizes an efficient Multi-head Hierarchy Mamba architecture. Experiments demonstrate that Orochi achieves state-of-the-art or competitive performance across tasks, even with parameter-efficient fine-tuning, highlighting the effectiveness of its data and pre-training strategies.", "questions": "Please refer to the weakness.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Orochi, a biomedical image processing model capable of handling various bio-imaging modalities and low-level tasks, addresses practical challenges faced by medical researchers and proposes a solution that can significantly reduce their computational and operational costs.\n\n2. The model is trained on a large-scale dataset (over 100 terabytes), positioning it as a promising foundation model in the biomedical domain.\n\n3. The effectiveness of the proposed method is validated through comprehensive performance comparisons with a variety of existing models.\n\nWeakness\n\n1. Limited detailed information on the diversity and balance of the pre-training dataset: While the paper highlights the large scale (100+ public studies, 100TB+ of data), it lacks clarity on the dataset’s composition—such as imaging modalities, species, pathological vs. normal cases, resolution range, and artifact types. This makes it difficult to assess how broadly applicable the claimed versatility of Orochi truly is.\n\n2. Failure cases are not discussed, making it difficult to understand the model’s limitations in real-world scenarios.", "was_revised": true, "final_justification": "The authors have adequately addressed all of my questions and concerns. Therefore, I will maintain my current score.", "rebuttal": "We sincerely thank the reviewer for the **recognition of the motivation, data selection, model training, and experimental presentation** of our work. In the following, we provide point-by-point responses to **all comments**, elaborating on areas that we were unable to detail due to space constraints. We hope this addresses your questions and earns your **further approval**. Thank you again!\n\n***\n\n**Q1:**\n> Limited detailed information on the diversity and balance of the pre-training dataset: While the paper highlights the large scale (100+ public studies, 100TB+ of data), it lacks clarity on the dataset’s composition—such as imaging modalities, species, pathological vs. normal cases, resolution range, and artifact types. This makes it difficult to assess how broadly applicable the claimed versatility of Orochi truly is.\n\n**A1:**\nWe invite you to review the **`Data_Sources.xlsx`** file located in our **Supplementary Materials**, within the **\"dataset\"** folder. This file contains the **organized metadata** that covers most of the information you mentioned (modality, species, dimension, resolution, directory...). Our **data diversity** is primarily derived from the modality diversity of the Image Data Resource (IDR) **[1]** website (which includes Light-sheet fluorescence microscopy, Digital pathology imaging, Yeast studies, High-content screening, etc.) and is supplemented by Atlas datasets (such as the Human Organ Atlas **[2]** and the Human Induced Pluripotent Stem Cell Atlas **[3]**). Due to page limitations, the main body of our paper focused on **task versatility**, with many details on data versatility expanded upon in the appendix. We hope this clarification resolves your concerns.\n\n***\n\n**Q2:**\n> Failure cases are not discussed, making it difficult to understand the model’s limitations in real-world scenarios.\n\n**A2:**\nLike all scientific papers, our work has its limitations, and we have **already included a \"Limitations\" section** in the manuscript. The nature of our training methodology and data types means that our model is **primarily focused on low-level tasks**. We have since conducted additional experiments by fine-tuning the model on two high-level tasks, namely segmentation and classification. \n\nFirst, we conducted new tests on the model's performance on a subset of BTCV **[4]**, REFUGE **[5]**, and CadVidSet **[6]** segmentation datasets. We strictly adhered to the training hyperparameter settings of MedSAM2 **[7]** and carried out a complete fine-tuning following its GitHub repository. Baseline results are from the original paper. \n\n| Model | REFUGE-2D | CadVidSet-2D | BTCV-Aorta-3D | BTCV-Liver-3D | \n| :--- | :--- | :--- | :--- | :--- | \n| TransUNet | 0.363 | 0.316 | **0.920** | 0.969 | \n| Swin-UNet | 0.289 | 0.226 | 0.892 | **0.975** | \n| nnUNet | 0.349 | 0.418 | 0.877 | 0.948 | \n| SAM2 | 0.558 | 0.539 | 0.835 | 0.861 | \n| MedSAM2 | 0.799 | **0.861** | 0.896 | 0.865 | \n| Orochi | **0.811** | 0.857 | 0.911 | 0.950 |\n\nAlso, we conducted classification tasks on TissueMNIST (2D) and OrganMNIST (3D) datasets. All experimental setups are from the MedMNIST series papers **[8] [9] [10]**. \n### Dice Score\n| Methods (TissueMNIST-2D-224^2) | AUC | ACC | Methods (OrganMNIST-3D-28^3) | AUC | ACC |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| ViT-Base | 0.948 | 0.729 | ResNet-18+3D | **0.996** | **0.907** |\n| MAE-Base | 0.945 | 0.719 | ResNet-50+3D | 0.994 | 0.883 |\n| CLIP-Base | 0.922 | 0.662 | ResNet-50+2.5D | 0.974 | 0.769 |\n| EVAv2-Base | 0.940 | 0.703 | Auto-sklearn | 0.977 | 0.814 |\n| DINOv2-Base | 0.944 | 0.715 | AutoKeras | 0.979 | 0.804 |\n| Orochi | **0.950** | **0.733** | Orochi | 0.995 | 0.891 |\n\nWe have demonstrated Orochi's capability to adapt to high-level tasks; however, we intend to avoid exaggerating its scope of application. We expect that its performance may decline to average or insufficient levels when applied to a broader range of high-level tasks, as our primary objective remains centred on the development of a unified low-level processor, as originally stated since the abstract of this paper.\n***\n**Reference**\n\n**[1]**  Williams, Eleanor, et al. \"Image Data Resource: a bioimage data integration and publication platform.\" Nature methods 14.8 (2017): 775-781.\n\n**[2]** Walsh, Claire L., et al. \"Imaging intact human organs with local resolution of cellular structures using hierarchical phase-contrast tomography.\" Nature methods 18.12 (2021): 1532-1541.\n\n**[3]** Viana, Matheus P., et al. \"Integrated intracellular organization and its variations in human iPS cells.\" Nature 613.7943 (2023): 345-354.\n\n**[4]** Landman BA, Xu Z, Igelsias JE, Styner M, Langerak TR, and Klein A, \"MICCAI multi-atlas labeling beyond the cranial vault workshop and challenge,\" (2015)\n\n**[5]** Orlando, José Ignacio, et al. \"Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs.\" Medical image analysis 59 (2020): 101570.\n\n**[6]** Wang, Lu, et al. \"Coronary artery segmentation in angiographic videos utilizing spatial-temporal information.\" BMC medical imaging 20.1 (2020): 110.\n\n**[7]** Zhu, Jiayuan, et al. \"Medical sam 2: Segment medical images as video via segment anything model 2.\" arXiv preprint arXiv:2408.00874 (2024).\n\n**[8]** Yang, Jiancheng, et al. \"Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification.\" Scientific Data 10.1 (2023): 41.\n\n**[9]** Yang, Jiancheng, Rui Shi, and Bingbing Ni. \"Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis.\" 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021.\n\n**[10]** Doerrich, Sebastian, et al. \"Rethinking model prototyping through the MedMNIST+ dataset collection.\" Scientific reports 15.1 (2025): 7669."}, {"review_id": "ka5I18uJBx", "reviewer": "Reviewer_Z17s", "summary": "A foundation model on biomedical image proposing is introduced in this draft. Multiple pre-training techniques are assembled to learn from 100+ datasets. Fine tuning is applied to the foundation model (Mamba + Swin-Transformer) for various downstream tasks such as restoration, super-resolution, fusion, and registration. The proposed model achieves comparable or superior performance compared to other task-specific models.", "questions": "A long list of pre-training techniques are available in the community. Four pre-training techniques are assembled in this draft. The underlying motivations to choose these four can be discussed with more details and analyzed with more experiments. \n\nThe backbone model uses Mamba and Swin-Transformer. Though they are not the new contributions in the draft, the backbone and fine-tuning can be briefly described in the main method section, rather than putting them entirely in the supplemental materials. A lot of engineering details in the current method section can be moved to supplement.\n\nThere are 100+ datasets with various tasks. When training the model, are all of them used at once for multi-task learning? Or, are a subset of them randomly sampled in each epoch? Or, is each of them iterated during the training? Some details on the training strategies can be added.", "limitations": "No discussion in the paper.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength: \n\n+ This paper is a demonstration of the scaling law. Using a lot of data for the pre-training and then fine-tune it for various tasks;\n\n+ Collecting 100+ datasets to build a model involves great effort; \n\nWeakness:\n\n- Biomedical image modalities can be quite different. The reader is not clear why features learned on one modality (e.g., microscopy cell images) can be used for the other modality with a large domain difference (e.g., MRI on brain images);\n\n- Some biomedical image tasks requires paired images of different modalities on the same object. As the collected 100+ datasets are quite heterogeneous, how can this problem be solved?\n\n- The method section is focused on the engineering implementation details. There is no new technique introduced.", "was_revised": true, "final_justification": "The reviewer has two major doubts on the paper:\n1. Biomedical images are very domain-specific. It's hard for the reviewer to think why the information learned from one domain (e.g., a microscopy image) could be useful for the other domain with big difference (e.g., MRI brain images). \n2. Learning from unrelated images could be another issue. If we can learn from multi modal data from the same patient regarding a certain type of disease (e.g., CT, lab test, other electronical health records, etc), it could be useful to the diagnosis, like what MedGemma did. \nDue to the concerns, I remain with the reject decision.", "rebuttal": "First and foremost, we would like to thank the reviewer for recognizing the **substantial effort** invested in our work and for the careful and diligent review. We **do include a Limitations section** in our manuscript, and here we will provide point-by-point responses to **all comments**. We aim to elaborate on aspects that were condensed due to page limitations and hope to earn your **further approval** by addressing your questions. Thank you!\n\n***\n\n**Q1:**\n> Biomedical image modalities can be quite different. The reader is not clear why features learned on one modality (e.g., microscopy cell images) can be used for the other modality with a large domain difference (e.g., MRI on brain images).\n\n**A1:**\nAs we **articulated in the first paragraph** of our introduction, the primary processing challenges for biomedical images, compared to natural images, stem from two main aspects. First is the issue of imaging device operational trade-offs: long imaging times can damage the subject, while shorter times may compromise image quality. This trade-off is a universal challenge present in **both microscopy and clinical imaging**. The second is the shortcomings of imaging modalities. For instance, MRI is superior for soft tissue imaging, while CT excels at imaging hard structures; many such **complementary modalities exist.**\n\nTherefore, **different imaging modalities are not entirely disconnected**. They share a degree of commonality and complementarity, from their underlying imaging principles to their imaging content. This understanding motivated our decision to broadly increase the diversity of our training data.\n\nFurthermore, our experiments demonstrate that a model pre-trained in this manner exhibits superior efficiency and performance when fine-tuned on various downstream tasks. This not only validates our hypothesis but also aligns with the broad consensus in the general vision domain that pre-training on diverse datasets like ImageNet leads to better performance for foundation models.\n\n***\n\n**Q2:**\n> Some biomedical image tasks require paired images of different modalities on the same object. As the collected 100+ datasets are quite heterogeneous, how can this problem be solved?\n\n**A2:**\nWe understand your concern. While downstream tasks like registration and fusion often require paired images, collecting such data at a large scale for pre-training is challenging. To address this, we developed **targeted self-supervised strategies**:\n* **For Registration:** We create a proxy task by applying multi-scale non-rigid deformations to an original image. The model's pre-training objective is to generate the registration grid that reverses this deformation. When fine-tuning on a downstream task, which may have only a small number of paired samples, our model can leverage this pre-learned ability to generate registration grids. This helps it adapt faster and more effectively, mitigating the risk of overfitting that models without such pre-training would face. \n* **For Fusion:** We build upon existing work that uses Masked Image Modeling (MIM) for fusion pre-training **[1]**. We advance this by employing a **dual-masking** strategy. This forces the model to first learn to fuse two non-overlapping masked regions of an image before predicting the content of the remaining masked area. Our ablation studies confirm that this method yields superior results compared to a standard MIM pre-training approach.\n\nIn summary, our pre-training phase does not require strictly paired data; instead, it learns through purely self-supervised methods. This pre-training paradigm allows us to scale up our dataset size and simplifies the process of incorporating new, unpaired data in the future.\n\n***\n\n**Q3:**\n> The method section is focused on the engineering implementation details. There is no new technique introduced. (Weakness # 3)\n\n> A lot of engineering details in the current method section can be moved to the supplement. (Question # 2)\n\n**A3:**\nWhile we did describe our self-supervised strategies in detail in the Method Section, **we respectfully disagree with the assessment that \"There is no new technique introduced.\"**\n\nOur writing style was intentionally modeled after widely recognized papers that introduced influential self-supervised strategies, such as MAE **[2]** (whose Method Section details its mask ratio, mask token) and I-JEPA **[3]** (which follows a similar pattern). Our paper, in fact, features a more complex design of self-supervised strategies, and detailed descriptions are unavoidable.\n\nSpecifically, our novelty lies in several areas:\n* Our Task-related Joint-embedding Pre-training (TJP) is a novel approach that simultaneously utilizes specific degradation types across multiple biomedical, domain-specific tasks as self-supervised learning strategies. To the best of our knowledge, this concept has not been previously proposed in existing literature.\n* Within each self-supervised strategy, we introduced new considerations. For example, we extended single-mask image modeling **[1]** to a dual-masking approach to better suit fusion tasks. We also improved upon the single-scale image degradation scheme in DeepLP **[4]** by implementing a multi-scale version.\n* Our Methods section also covers the key concept of our designs for data acquisition, model architecture, and post-training strategies. While we are not the first to use a Swin-Mamba framework, we are the first to apply it to this breadth of multi-task pre-training. Furthermore, our three-tiered fine-tuning strategy and its validation for different scenarios are also contributions. We will revise the manuscript to highlight these innovations more prominently in the main text in a future version with a more generous page limit.\n\nWhile these designs certainly build upon the work of others, we believe the claim of \"no new technique\" is not a fair characterization, especially when compared to other papers that focus on advancements in self-supervised strategies.\n***\n**Q4:**\n> A long list of pre-training techniques are available in the community. Four pre-training techniques are assembled in this draft. The underlying motivations to choose these four can be discussed with more details and analyzed with more experiments.\n\n**A4:**\nWe chose these four pre-training techniques (for restoration, super-resolution, registration, and fusion) not only because they represent the four primary low-level processing tasks in biomedical imaging, but more importantly, because of their deep **underlying relationships**, which we took considerable space to explain.\n\nThe need to preprocess biomedical images stems from two fundamental inadequacies of current imaging methods:\n1.  **Shortcomings of imaging modalities:** A single modality often has limitations. Therefore, a model should learn to **fuse** information from different modalities, even if they are not perfectly aligned (which requires **registration**).\n2.  **Imaging device operational trade-offs:** Even with an adequate modality, we cannot guarantee high image quality. Thus, we want the model to extract useful information even from low signal-to-noise ratio images and perform image reconstruction (**restoration** and **super-resolution**).\n\nIn our ablation studies, we demonstrated that if our TJP is removed and replaced with standard MIM methods, the performance suffers, especially on spatially-aware tasks like registration. We have additionally performed new ablations comparing TJP to DINOv2 **[5]** pre-training. As shown in the results below, the TJP approach remains more robust.\n\n|  | OASIS (Registration) | VIFB (Fusion) | CARE (Restoration) | HBA (Super-Resolution) |\n| :--- | :--- | :--- | :--- | :--- |\n| Strategy | Dice | Qabf | PSNR | PSNR |\n| MAE (single mask) | 71.22 | 0.36 | 26.67 | 29.17 |\n| I-JEPA (dual mask) | 69.97 | 0.39 | 25.02 | 28.81 |\n| DINOv2 (multi-augment) | 81.12 | 0.40 | 28.17 | 31.01 |\n| Orochi (task multi-augment) | **83.62** | **0.41** | **29.88** | **33.63** |\n***\n**Q5:**\n> There are 100+ datasets with various tasks. When training the model, are all of them used at once for multi-task learning? Or, are a subset of them randomly sampled in each epoch? Or, is each of them iterated during the training? Some details on the training strategies can be added.\n\n**A5:**\nThank you for the suggestion. Yes, **all of the studies were included** for multi-task learning, but not all the raw data was used. It is important to clarify that the original studies associated with those data did not necessarily involve deep learning, meaning properties like data dimensionality and resolution may not be directly suitable for training a deep learning model.\n\nTo leverage this diverse data, our solution involved several steps. After an initial filtering phase (to remove non-imaging data like tables and graphs), we performed multi-scale random sampling on the source data (as detailed in Section 3.2 of our paper). We then employed a hybrid training strategy using both locally stored (fixed patch/volume) and streamed data (random patch/volume).\n***\n**Reference**\n\n**[1]**  Li, Jiayang, et al. \"MaeFuse: Transferring omni features with pretrained masked autoencoders for infrared and visible image fusion via guided training.\" IEEE Transactions on Image Processing (2025).\n\n**[2]** He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n**[3]** Assran, Mahmoud, et al. \"Self-supervised learning from images with a joint-embedding predictive architecture.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n**[4]** Fang, Linjing, et al. \"Deep learning-based point-scanning super-resolution imaging.\" Nature methods 18.4 (2021): 406-416.\n\n**[5]** Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023)."}]

Author rebuttal:

Meta-review: This paper present a new foundation model for biomedical image processing and evaluates it on a diverse range of tasks. Reviewers praised the breadth of the experimental results and clarity of the presentation while raising concerns about the feasibility of transfer from a single model to such a diverse array of tasks.

While the reviews are varied in score, comprising every number in the range 3-6, the borderline positive reviewer gave high praise to the work (using the word "outstanding") and noted that their low confidence prevented them from giving a higher score. Meanwhile, the negative reviewer's final justification raised concerns about whether the model could truly be transferred to diverse tasks that seem to be addressed by the work's empirical results and did not respond to the authors' (or the AC's) pushback. Therefore I believe the paper warrants acceptance.
Decision: Accept (spotlight)