Title: Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Models

Medical Reasoning

Chain-of-Thought
Abstract: Multimodal large language models (MLLMs) have begun to demonstrate robust reasoning capabilities on general tasks, yet their application in the medical domain remains in its early stages. Constructing chain-of-thought (CoT) training data is essential for bolstering the reasoning abilities of medical MLLMs. However, existing approaches exhibit a deficiency in offering a comprehensive framework for searching and evaluating effective reasoning paths towards critical diagnosis. To address this challenge, we propose Mentor-Intern Collaborative Search (MICS), a novel reasoning-path searching scheme to generate rigorous and effective medical CoT data. MICS first leverages mentor models to initialize the reasoning, one step at a time, then prompts each intern model to continue the thinking along those initiated paths, and finally selects the optimal reasoning path according to the overall reasoning performance of multiple intern models. The reasoning performance is determined by an MICS-Score, which assesses the quality of generated reasoning paths. Eventually, we construct MMRP, a multi-task medical reasoning dataset with ranked difficulty, and Chiron-o1,  a new medical MLLM devised via a curriculum learning strategy, with robust visual question-answering and generalizable reasoning capabilities. Extensive experiments demonstrate that Chiron-o1, trained on our CoT dataset constructed using MICS, achieves state-of-the-art performance across a list of medical visual question answering and reasoning benchmarks. Codes are available at https://github.com/Yankai96/Chiron-o1


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach for searching and evaluating effective reasoning paths towards critical diagnosis in the medical domain. The authors construct a multi-task medical reasoning dataset with ranked difficulty and a new medical MLLM devised via a curriculum learning strategy. Extensive experiments demonstrate that Chiron-o1 achieves state-of-the-art performance across a list of medical visual question answering and reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- A multimodal medical dataset comprising QA pairs is introduced, which can contribute to the medical community.\n- The authors propose a new multimodal medical model that demonstrates outstanding reasoning abilities in handling both in-domain and out-of-domain complex clinical problems.\n- The proposed approach achieves competitive performance compared to previous SOTA medical MLLMs through extensive experiments across multiple benchmarks.\n\nWeaknesses:\n\n- The proposed medical dataset is basically a reorganization of existing medical datasets. There is no new high-quality dataset annotated in this paper. The contribution is limited.\n- As shown in Figure 5, the dataset is imbalanced across different body systems.\n- The source and resolution of the images in datasets are different and imbalanced. Also there is no high-resolution images, limiting the usefulness of the proposed dataset\n- The novelty of the Multi-Stage SFT is limited. It is basically the curriculum learning pattern. There is also no ablation and experiment results supporting that it is better than every single single-stage learning.\n- More qualitative results is needed for different test set.\n\nQUESTIONS:\nI wonder if the multi-stage learning paradigm is generalizable across different MLLMs. The ablations about this is only conducted on MMMU with Chiron-o1-8B.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nChain-of-Thought (CoT) data is crucial for enhancing the reasoning capabilities of medical multi-modal large language models (MLLMs). The authors propose a Mentor-Intern Collaborative Search (MICS) method, which generates optimal CoT data by evaluating the scores of different reasoning paths. Based on this approach, they construct the MMRP dataset and train the Chiron-o1 model, which achieves outstanding performance on multiple VQA and reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The experiments in the paper are comprehensive, the structure is clear, and the writing is well-organized.\n2. The Mentor-Intern Collaborative Search (MICS) strategy is straightforward and plays an important role in improving the quality of CoT data.\n\nWeaknesses:\n1. MICS relies on the intern's feedback to refine the CoT paths generated by the mentor; however, the method essentially depends on the mentor model's capability to produce CoT data. Is the mentor model itself sufficiently competent to generate reliable and high-quality reasoning data? This point warrants further discussion.\n2. The paper presents several examples (e.g., Figure 3, Figure 20) where the prompts provide extensive sign-related information, which is almost equivalent to revealing the answer. The reviewer is unsure whether the model is genuinely leveraging image features, or is simply generating responses based on the content of the prompt. The authors should validate this.\n3. In many real clinical scenarios, such as medical screening or physical examinations, there is often no sign-specific prompt and only imaging data is available—the model needs to generate a diagnostic report. How does the model perform under such conditions?\n4. If the prompt contains incorrect sign/disease-related information (e.g., the patient reports abdominal pain and is assumed to have appendicitis, but actually has a bowel obstruction), will the model be misled and provide suggestions along the wrong path? The authors should investigate and verify this.\n\nQUESTIONS:\n1. Is it possible for the dataset constructed in this paper to be open-sourced? Making it publicly available would be a valuable contribution to the entire research community.\n2. Medical imaging is an inherently rigorous discipline, and the capabilities of general-purpose MLLMs are limited in medical contexts. While optimizing the MLLM output distribution using certain metrics (such as the MICS-score) can to some extent improve the quality of generated CoT data, can CoT data generated without leveraging dedicated medical knowledge or tools like knowledge graphs truly be considered reliable? How can the correctness of the underlying medical logic be ensured? I hope the authors can provide a response on this issue.\n3. The authors need to demonstrate that the proposed MLLM does in fact incorporate imaging information in its output, rather than relying solely on prompts and textual knowledge the model has acquired.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **MICS**, a novel strategy for generating a high-quality CoT dataset from VQA samples. Using MICS, the authors curate **MMRP**, a medical reasoning dataset for SFT of language models. The authors also train **Chiron-o1**, a new multimodal medical reasoning model, through a three-stage training using MMRP and evaluate it on different datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- **MICS** is a novel and well-motivated framework for generating high-quality chain-of-thought (CoT) data. It provides a generalizable method that can be applied to existing datasets to enrich them with step-by-step reasoning traces.\n- The authors introduce **MMRP**, a multimodal medical reasoning dataset, which could serve as a valuable resource for training and evaluating medical MLLMs.\n- The paper includes thorough **ablation studies** that effectively demonstrate the impact of MICS and their proposed three-stage training strategy. These analyses help validate the importance of each design choice.\n\n**Weaknesses**\n- The paper lacks clarity in some aspects of the **evaluation setup** and **implementation details**. For example, it is unclear which exact versions of baseline models (e.g., **LLaVA-Med**, **Gemini**) were used in comparison. Additionally, important training details such as the number of GPUs used, total GPU hours, and compute infrastructure are missing (information that is required by the checklist).\n- The **evaluation scope could be improved**. Since Chiron-o1 is trained on the datasets used in Table 1, it would be more convincing to include evaluations on **unseen datasets** such as *OmniMedVQA [1]* and *MediConfusion [2]*, to better assess generalization. Moreover, the absence of **proprietary models** (e.g., GPT-4o, Gemini 2.5 Pro) in Table 2 is not well justified, even though the table focuses on medical reasoning, including these models would provide useful reference points.\n- While the proposed **MICS** strategy is promising, the paper would benefit from a more detailed analysis of **failure cases** or **limitations**. For example, in what types of reasoning tasks does MICS fail to generate effective paths? Are there modalities or body systems where it underperforms?\n\n**References**  \n[1] Cheng, T., et al. *OmniMedVQA: A Benchmark for Generalized Medical Visual Question Answering*. CVPR, 2024.  \n[2] Sepehri, M. S., et al. *MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models*. ICLR 2025.\n\nQUESTIONS:\n1. Could the authors clarify the versions of the baseline models used for evaluation? Specifically, the paper mentions that Gemini 2.5 Pro is used as a mentor model in the MICS process, but in the baseline comparison (e.g., Table 1), the cited paper corresponds to Gemini 1.5 Pro. Why is a stronger model used for MCIS but not included as a baseline?\n\n2. There appear to be discrepancies between the numbers reported in Table 1 and those in the original papers. For example, the LLaVA-Med paper reports accuracies of 61.52, 84.08, and 37.95 on VQA-RAD, SLAKE, and PathVQA (based on open evaluation for LLaVA-based version), whereas the numbers shown in Table 1 differ. Could the authors explain the source of these discrepancies? Were all baselines re-evaluated under a consistent setup?\n\n3. For the VQA benchmarks, what evaluation protocol was used, open-ended or multiple-choice (closed-ended) answering? This distinction significantly affects accuracy metrics and should be clarified for fair comparison.\n\n4. Have the authors explored the effect of reinforcement learning (RL) fine-tuning after the three-stage SFT process on Chiron-o1? Given the success of GRPO in Med-R1 and MedVLM-R1, it would be interesting to assess whether RL could further improve the reasoning capabilities of Chiron-o1.\n\n5. There are several issues reported regarding a lack of clear visual understanding in multimodal language models, where they often fail to capture fine-grained image details, which is a crucial ability in the medical domain. Have the authors investigated whether the MICS strategy helps mitigate this issue? Does Chiron-o1 still suffer from these limitations?\n\n6. It is not clear whether the intern models in the MICS process have access to the image input. Based on Figure 2, it appears they are only given the question and partial reasoning steps. If they do not see the image, could this harm their performance?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Chiron-o1, a multimodal large language model for medical reasoning, along with a novel training methodology called Mentor-Intern Collaborative Search (MICS).\nKey Contributions:\n1. MICS Strategy: A collaborative framework where \"mentor\" models (GPT-4o, Gemini) generate reasoning steps while \"intern\" models (smaller open-source models) evaluate their quality. The system iteratively searches for effective reasoning paths by having interns complete reasoning from mentor-generated steps and selecting paths based on success rates.\n2. MMRP Dataset: A three-part multimodal medical dataset containing:\n57,630 filtered, text-only QA for basic knowledge\n5,878  image-text alignment pairs \n8,328 complex reasoning scenarios across clinical contexts\n3. Chiron-o1 Model: Trained using 3-stage curriculum learning (text QA ->  image-text alignment -> complex reasoning with MICS-generated data), achieving good performance on medical VQA and reasoning benchmarks.\nThe core innovation is MICS, which creates high-quality medical reasoning data through collaborative model evaluation rather than single-model generation or expensive human annotation. This enables more rigorous reasoning paths that significantly improve performance on complex clinical scenarios compared to existing medical MLLMs.\n\nSTRENGTHS AND WEAKNESSES:\n1. Strengths\n\na. Quality\nDesign: The paper conducts comprehensive evaluations across 7 benchmarks, including both in-domain (MMRP) and out-of-domain (MedXpertQA_MM) scenarios, demonstrating good generalization capabilities.\nAblation studies: Systematic analysis of MICS components, training strategies, and dataset configurations supports design.\nMethodology: The framework here addresses a real problem in medical CoT data generation - the lack of quality evaluation mechanisms.\nConsistent improvements: Chiron-o1 achieves gains across multiple benchmarks compared to baseline models.\n\nb. Clarity\nWell-structured presentation: The paper flows well.\nClear technical exposition: The MICS algorithm is explained well and visualizations help provide more clarity.\n\nc. Significance\nAddresses healthcare need: Medical reasoning is a complicated process, not always captured in many models; improved AI capabilities could have substantial impact.\nScalable approach: MICS provides a framework for generating high-quality reasoning data that could be applied beyond medical domains.\nClear Results: Demonstrates clear advancement over existing medical MLLMs, particularly in reasoning tasks.\n\nd. Originality\nNew collaborative search approach: The mentor-intern framework for reasoning path evaluation is creative and well-motivated by educational analogies.\nEvaluation metric: MICS-Score provides a set way to assess reasoning path quality using multiple model perspectives.\nComprehensive dataset construction: The three-stage MMRP dataset design with medical content could help with other models\n\n2. Weaknesses\n\na. Quality\nThe MMRP dataset would benefit from clinician sampling and testing. There is the risk of data not being as representative or as clinically logical as expected.\nLimited theoretical analysis: The paper lacks theoretical justification for why the mentor-intern collaboration should work better than alternatives\nComputational cost concerns: MICS requires multiple API calls to expensive models (GPT-4o, Gemini) and numerous intern model evaluations, making it potentially too costly in multiple respects for large-scale deployment. The expense should be recognized to include environmental cost, as these models risk using high amounts of electricity, water and  furthering climate change and the benefit may not be as clear when paired with downstream effects. Showing what the relative costs and benefits may be, even if very rough, would help.\nEvaluation limitations:\nTest size for benchmarks\nBERT-Score for semantic similarity may not capture medical reasoning quality adequately\n\nb. Clarity\nComplex methodology: While well-explained, the MICS algorithm involves many components that may be difficult to reproduce reliably.\n\nc. Significance\nIncremental: While effective, the core contribution builds incrementally on existing CoT and collaborative reasoning approaches rather than being an entirely new approach.\nComparison limitations: Could compare with more recent medical reasoning frameworks\n\nd. Originality\nLimited novelty of each piece: Novelty ia from the combination of study parts\nDataset overlap concerns: The MMRP dataset is built on existing medical case data, often used in such studies\n\nQUESTIONS:\n1. Computational Costs of MICS\nQuestion: Can you clarify the computational resources, API costs, and environmental impact for training Chiron-o1 with MICS? How many API calls to GPT-4o/Gemini were used, and what's the approximate cost per reasoning path generated? Could you achieve similar results using only open-source models?\n\nEvaluation criteria: Clearly describing computational costs and considering cheaper alternatives would improve the paper's practical value.\n\n2. Medical Bias and Safety\nQuestion: How do you ensure MICS doesn't increase biases or mistakes from the original medical data? Did you check if the mentor-intern method introduces systematic biases? Could you share details about dataset demographics, representation of diseases across different groups (e.g., malaria, sickle cell, emerging diseases), and methods for identifying or preventing bias? Did medical experts help you assess bias risks?\n\nEvaluation criteria: Clearly discussing bias risks and showing concrete steps to reduce them would make the approach safer and more trustworthy.\n\n3. Clinical Validation and Next Steps\nQuestion: What are your plans for clinical validation? Did medical professionals help confirm the clinical usefulness of the reasoning patterns generated? Can you outline specific next steps required for clinical evaluation?\n\nEvaluation criteria: Clearly outlining plans for clinical validation and showing involvement from medical professionals would strengthen confidence in real-world applicability.\n\n4. Comparison with Human Medical Reasoning\nQuestion: How does Chiron-o1’s reasoning quality compare to human medical experts? Can you highlight areas where the model performs particularly well and situations where human oversight is still necessary?\nEvaluation criteria: Provide insights from subject matter experts to support comparisons and strengthen your conclusions.\n\nLIMITATIONS:\nTechnical Limitations\nCould discuss computational costs and API expenses required, which could limit accessibility\nHeavy reliance on closed-source models (GPT-4o, Gemini) creates vulnerability to API changes, pricing modifications, or service discontinuation.\nLimited discussion of whether BERT-Score and MICS-Score truly capture medical reasoning quality, or potential biases in the evaluation framework. Bringing in expert opinion could help as well.\nGeneralization boundaries: Insufficient analysis of when/where the approach might fail or perform poorly (ie rare diseases (though examples quite rare), emerging diseases, different medical systems).\nMissing Societal Impact Discussion\nCould discuss accountability when AI-generated reasoning leads to incorrect medical decisions or diagnoses.\nEnvironmental impact: The computational intensity of MICS, requiring multiple API calls to large models and extensive inference, contributes to climate and environmental risks.\nBias amplification and error reproduction: The training data from existing sources may perpetuate existing medical biases, diagnostic errors, and systematic mistakes already present in medicine.\nHealthcare worker deskilling and automation bias: Risk that medical professionals may become overly dependent on AI reasoning, potentially diminishing critical thinking skills and leading to automation bias where clinicians uncritically accept AI-generated conclusions.\nWould help to discuss regulatory hurdles expected for clinical deployment\n\nMissing Safeguards Discussion\nClinical validation: Need further clinical evaluation before deployment, could discuss\nHuman oversight mechanisms: How to ensure appropriate human supervision\nBias monitoring: Strategies for detecting and mitigating biased reasoning patterns\nFailure mode detection: Methods for identifying when the system is operating outside its capacity", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nChiron-o1 is a novel approach for training VLMs to be better at medical visual and reasoning tasks. Through the use of curriculum learning, the authors first train the model to become grounded with clinical cases and their corresponding treatments (stage I). In stage II, the model is then trained with images with their corresponding diagnoses. Finally, the model is trained with their novel mentor-intern collaborative search strategy where a more powerful mentor model is generates a reasoning prefix before the an intern model completes the remaining reasoning. If the intern model reasons well and is able to complete the chain frequently to arrive at the answer, then it is clear that the approach yields a successful thought.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. Without using any reinforcement learning, the authors are able to get significant improvement with just fine-tuning data. \n\n2. The mentor-intern collaboration is a novel approach which allows for diversity in thoughts and thinking by having multiple mentor and intern models attempt to complete the question. \n\n3. Approach has been thoroughly benchmarked with general medical VQA and VQA requiring extensive reasoning, which demonstrates that the approach has a lot of merit in both simple and complex scenarios. \n\nWeaknesses: \n\n1. While the authors demonstrate that their model is better than other medical reasoning approaches, comparison against other tree-branching approaches would better help resolve the novelty of this approach. \n\n2. DeepSeek-V3 is the only model being used for validation for the various reasoning chains, but other models should be used too to prevent biases. \n\n3. Intermediate reasoning steps are not independently verified, they are only assessed based on if the final answer is correct or not.\n\nQUESTIONS:\n1. It's still not entirely clear why you had to incorporate VQA as a separate item for fine-tuning in Equation 6. I understand that the MMRP is  not enough which is why you added VQA, but you need to further explain this point, especially since MMRP includes stage 2 which involves fine-tuning with text-image pairs. \n\n2. What is the performance of different number of intern/mentor models? Is it the case that a single intern/mentor model dominated the completion in most cases? \n\n3. Can you report how does the number of API calls/tokens change compared to rejection fine-tuning with just the intern models?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-23"}