Title: Orochi: Versatile Biomedical Image Processor
Venue: NeurIPS 2025
Accepted: yes
Keywords: life science

biology

microscopy

foundation model
Abstract: Deep learning has emerged as a pivotal tool for accelerating research in the life sciences, with the low-level processing of biomedical images (e.g., registration, fusion, restoration, super-resolution) being one of its most critical applications. Platforms such as ImageJ (Fiji) and napari have enabled the development of customized plugins for various models. However, these plugins are typically based on models that are limited to specific tasks and datasets, making them less practical for biologists. To address this challenge, we introduce **Orochi**, the first application-oriented, efficient, and versatile image processor designed to overcome these limitations. Orochi is pre-trained on patches/volumes extracted from the raw data of over 100 publicly available studies using our Random Multi-scale Sampling strategy. We further propose Task-related Joint-embedding Pre-Training (TJP), which employs biomedical task-related degradation for self-supervision rather than relying on Masked Image Modelling (MIM), which performs poorly in downstream tasks such as registration. To ensure computational efficiency, we leverage Mamba's linear computational complexity and construct Multi-head Hierarchy Mamba. Additionally, we provide a three-tier fine-tuning framework (Full, Normal, and Light) and demonstrate that Orochi achieves comparable or superior performance to current state-of-the-art specialist models, even with lightweight parameter-efficient options. We hope that our study contributes to the development of an all-in-one workflow, thereby relieving biologists from the overwhelming task of selecting among numerous models. Our pre-trained weights and code will be released.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA foundation model on biomedical image proposing is introduced in this draft. Multiple pre-training techniques are assembled to learn from 100+ datasets. Fine tuning is applied to the foundation model (Mamba + Swin-Transformer) for various downstream tasks such as restoration, super-resolution, fusion, and registration. The proposed model achieves comparable or superior performance compared to other task-specific models.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n+ This paper is a demonstration of the scaling law. Using a lot of data for the pre-training and then fine-tune it for various tasks;\n\n+ Collecting 100+ datasets to build a model involves great effort; \n\nWeakness:\n\n- Biomedical image modalities can be quite different. The reader is not clear why features learned on one modality (e.g., microscopy cell images) can be used for the other modality with a large domain difference (e.g., MRI on brain images);\n\n- Some biomedical image tasks requires paired images of different modalities on the same object. As the collected 100+ datasets are quite heterogeneous, how can this problem be solved?\n\n- The method section is focused on the engineering implementation details. There is no new technique introduced.\n\nQUESTIONS:\nA long list of pre-training techniques are available in the community. Four pre-training techniques are assembled in this draft. The underlying motivations to choose these four can be discussed with more details and analyzed with more experiments. \n\nThe backbone model uses Mamba and Swin-Transformer. Though they are not the new contributions in the draft, the backbone and fine-tuning can be briefly described in the main method section, rather than putting them entirely in the supplemental materials. A lot of engineering details in the current method section can be moved to supplement.\n\nThere are 100+ datasets with various tasks. When training the model, are all of them used at once for multi-task learning? Or, are a subset of them randomly sampled in each epoch? Or, is each of them iterated during the training? Some details on the training strategies can be added.\n\nLIMITATIONS:\nNo discussion in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOrochi is introduced as the first versatile biomedical image processor designed for diverse low-level tasks like restoration, super-resolution, registration, and fusion, overcoming limitations of task-specific specialist models. It employs a novel Task-related Joint-embedding Pre-Training (TJP) on a large, multi-scale dataset and utilizes an efficient Multi-head Hierarchy Mamba architecture. Experiments demonstrate that Orochi achieves state-of-the-art or competitive performance across tasks, even with parameter-efficient fine-tuning, highlighting the effectiveness of its data and pre-training strategies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Orochi, a biomedical image processing model capable of handling various bio-imaging modalities and low-level tasks, addresses practical challenges faced by medical researchers and proposes a solution that can significantly reduce their computational and operational costs.\n\n2. The model is trained on a large-scale dataset (over 100 terabytes), positioning it as a promising foundation model in the biomedical domain.\n\n3. The effectiveness of the proposed method is validated through comprehensive performance comparisons with a variety of existing models.\n\nWeakness\n\n1. Limited detailed information on the diversity and balance of the pre-training dataset: While the paper highlights the large scale (100+ public studies, 100TB+ of data), it lacks clarity on the dataset’s composition—such as imaging modalities, species, pathological vs. normal cases, resolution range, and artifact types. This makes it difficult to assess how broadly applicable the claimed versatility of Orochi truly is.\n\n2. Failure cases are not discussed, making it difficult to understand the model’s limitations in real-world scenarios.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper pretrains a foundation model for low-level biomedical image processing tasks, including restoration, super-resolution, registration and fusion. Because of the high heterogeneity of biomedical images (regarding image dimension, modality, imaging subjects and body parts, etc,), there are a large body of methods dealing with different tasks in different settings and it is usually challenging for researcher to find the proper tool for there specific need. Therefor, it is valuable to develop a unified model that can perform different tasks in different settings. This paper collect a large body of biomedical data from 100 studies, and develops a self-supervised pre-training strategy suitable for low-level image processing task. The pretrained model is then fine-tuned and compared to a set of specialist models on each task and achieves comparable or superior performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1.The motivation is reasonable and it is valuable to develop a unified model that can be applied to different biomedical image processing tasks.\n\n2.The paper is well written and easily to follow.\n\n3.This study collected and curated a relatively large body of biomedical data for pretraining.\n\n4.The proposed pre-training strategy seems suitable for low-level processing tasks. \n\n\nWeakness\n1.Actually, the practical value of low-level tasks is relatively low, especially when comparing to high-level tasks, such as classification, segmentation, detection. Though there are commonly used metrics for evaluating low-level tasks, their practical meaning is more demonstrated in downstream high level tasks. For example, image fusion methods are usually evaluated in downstream detection/classification tasks.  \n\n2.For each of the four tasks, evaluation is only performed on one or very small number of datasets. We know that because of the high heterogeneity of biomedical images, the performance on one dataset can hardly generalize to all other datasets with slightly different settings. The evaluation on each task is far from being enough to show that the proposed method is generally applicable for the specific task type under different settings.\n\n3.For the image fusion task, only three metrics are used to compare different methods, which is too small. Current image fusion studies typically utilize much more metrics.\n\n4.The training strategies of the proposed methods and the baselines are different in many experiments, which may make the comparison unfair.\n\n5.For the image registration task, there are a large set of different setting which bring different kinds of difficulty. Such as, registration between 2D images or 3D volumes, registration between different modality pairs or within the same modality, registration of different body parts with significant or not deformation, etc. Much more experiments are needed to conclude that a model can be generally applied to image registration task.\n\nQUESTIONS:\nplease respond to the weakness points\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a pretraining method and suite of models for a variety of biomedical image processing tasks such as restoration, super-resolution, and fusion. Orochi is trained on an extremely large (>14M) and diverse collection of medical images with an efficient, multi-scale representation learning approach tailored for downstream image processing tasks. When fine-tuned on downstream tasks, Orochi outperforms state-of-the-art foundation models and specialist models specifically tailored for individual modalities and tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- The presentation quality is among the highest I have seen while reviewing for NeurIPS. Typesetting is used effectively to guide the reader’s attention and break up passages of text. Figures are dense, informative, and visually appealing\n- The paper is both organized and written clearly. Prior work and important background are thoroughly explained. \n- Experimental efforts are comprehensive, with evaluation across many unique tasks, datasets, and modalities. Moreover, performance is compared against state-of-the-art specialized baseline approaches\n- Open-source release of code and pre-trained weights should make this a major contribution to the field\n\n**Weaknesses**:\n- I see no major weaknesses. However, I am curious to see how Orochi performs on downstream discriminative tasks like classification or even segmentation.\n\n**Minor comments**:\n- I originally misinterpreted the word “study”. In the context of medical imaging, this can often effectively mean “exam”; thus, “100 studies” seemed to be a very small amount. Context clues clarified this confusion quickly, but I would suggest potentially changing the wording here. There is probably a better way to clarify that this is an aggregate of many existing datasets.\n- Related to the above point, I would try to highlight the “size” (e.g., # images) of the composite pretraining dataset as early as possible to convey the scale and dispel any confusion\n- Figure 1: While funny (to me, at least), I’m not sure “Crappifier” is appropriate\n- L172: Change “down-sample” -> “Down-sampling”\n- L201 is not a complete sentence: “Resulting in more than 30…” perhaps simply remove the period and write “resulting”\n- Table 5: The title is confusing, “Pre-train Strategies V.S Performance”. Should this read “Vs.” as in “versus”?\n- Figure 5: Visually appealing, but difficult to read. Perhaps a different font/color choice is needed or the figure should be enlarged.\n\nQUESTIONS:\n1.\tHow does Orochi perform when fine-tuned on downstream (discriminative) tasks like classification? I understand that this does not fit under the umbrella of \"low-level processing\", but have the authors conducted such experiments? I would hypothesize it might perform well on dense tasks like segmentation given its multi-scale pretraining.\n\nLIMITATIONS:\nLimitations are addressed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}