U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding
Authors: Anjie Le
Henan Liu
wangyue
Zhenyu Liu
Rongkun Zhu
Taohan Weng
Jinze Yu
Boyang Wang
Yalun Wu
Kaiwen Yan
Quanlin Sun
Meirui Jiang
Jialun Pei
Siya Liu
Haoyun Zheng
Zhoujun Li
Alison Noble
Jacques Souquet
Xiaoqing Guo
Manxi Lin
Hongcheng Guo
Venue: ICLR 2026 Poster (2026)
Keywords: medical ultrasound
benchmark
large vision-language model
Primary area: datasets and benchmarks
TL;DR: We present U2-BENCH, the first benchmark for evaluating LVLMs on ultrasound, spanning 15 anatomical regions and defines 8 clinically inspired tasks across 50 ultrasound application scenarios.

Abstract: Ultrasound is a widely-used imaging modality critical to global healthcare, yet its interpretation remains challenging due to its varying image quality on operators, noises, and anatomical structures. Although large vision-language models (LVLMs) have demonstrated impressive multimodal capabilities across natural and medical domains, their performance on ultrasound remains largely unexplored. We introduce U2-BENCH, the first comprehensive benchmark to evaluate LVLMs on ultrasound understanding across classification, detection, regression, and text generation tasks. U2-BENCH aggregates 7,241 cases spanning 15 anatomical regions and defines 8 clinically inspired tasks, such as diagnosis, view recognition, lesion localization, clinical value estimation, and report generation, across 50 ultrasound application scenarios. We evaluate 23 state-of-the-art LVLMs, both open- and closed-source, general-purpose and medical-specific. Our results reveal strong performance on image-level classification, but persistent challenges in spatial reasoning and clinical language generation. U2-BENCH establishes a rigorous and unified testbed to assess and accelerate LVLM research in the uniquely multimodal domain of medical ultrasound imaging.

Published as a conference paper ICLR 2026
U2- U2-BENCH: BENCHMARKING LARGE VISION-LANGUAGE MODELS ON ULTRASOUND UNDERSTANDING
Anjie Le Henan L.u Yue Wang" Zhenyu Liu', Rongkun Zhu Taohan Weng'., Jinze Yu Boyang Wang Yalun Wu', Kaiwen Yan Quanlin Sun 6 Meirui Jiang".7 Jialun Pei", Siya Liu Haoyun Zheng'. Zhoujun Li J. Alison Noble", Jacques Souquet Xiaoqing Guo 12,5 Manxi Lin 19 Hongcheng G.3
1 Dolphin AI 2 University of Oxford
3 Fudan University Hong Kong Baptist University University Cambridge Esonic Imaging Technical University of Denmark 10 Independent
4 Beihann University
The Chinese University of Hong Kong
ABSTRACT
Ultrasound is a widely-used imaging modality critical to global healthcare, yet its interpreta- tion remains challenging due to variability in image quality caused by operator dependency, noise, and anatomical complexity. Although large vision-language models (LVLMs) have demonstrated impressive multimodal capabilities across natural and medical domains, their performance ultrasound remains largely unexplored. We introduce U2-BENCH, the first comprehensive benchmark to evaluate LVLMs on ultrasound understanding across classification,, detection, regression, and text generation tasks. U2-BENCH aggregates 7,241 cases spanning 15 anatomical regions and defines 8 clinically inspired tasks, such as diagnosis, view recognition, lesion localization. clinical value estimation, and report generation, across 50 ultrasound application scenarios. We evaluate 23 state-of- the-art LVLMs, both open- and closed-soures source, general-pure purposs and medical-specific Our results reveal strong performance on image-level classification, but persistent challenges in spatial reasoning and clinical language generation. U2-BENCH establishes a rigorous and unified testbed to assess and accelerate LVLM research in the uniquely multimodal domain of medical ultrasound imaging.
1 INTRODUCTION
Ultrasound (US) is one of the most widely used imaging modalities in global healthcare - essential in obstetrics, emergency medicine, cardiology, and low-resource settings while its interpretation remains notoriously difficult (Hewson & Bedforth, 2023). Compared to modalities such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and whole-slide imaging (WSI), which offer higher spatial resolution, consistent image quality, and standardized anatomical views, ultrasound is real-time and low-cost but highly operator-dependen and frequently affected by imaging artifacts (Sharma let al., 2021). In addition, in contrast to these modalities, US is dynamically presenting three-dimenseee (3D) anatomies image sequences. Therefore, accurate interpretation of US demands not only visual pattern recognition in the images, but also understanding of anatomy and capturing of dynamic spatial-context reasoning, typically requiring extensive prior domain expertise (Wang et al. 2022). These challenges have limited the applicability of carlier artificial intelligence (AI) models. However, recent advances in medical
Datascts and code are available /ttpr/attttt/////t//ttttottetaaa http::/wwwaatteeeetttttttttttttt leaderboard is available at: hhtpps: http:::///oopiinnsooooe www..u.dd...tu.............. wttphuub...uu---bbhheee
The

---

Published as conference paper ICLR 2026
Querle oily slyleppe decleriinnionn ot sid us mnd mnedaas
Gustas, poecill aostcaend ofomate Biliigg
Da corter
Sometics aieanantte sedoard aara pori ure when ducii lepttin.. Oedon
Use
per atherd colegn aegeiiii underte
Seude
cerirr zer and
photter propene atruauud diudo wonggoolic feney ceneteeti snjeeteeen.. Outae
cearroo rnvoestmbbe iembatt tillomm
DD
Quaries dear pinaa
toupe bunders fmaath srattin vensureee
VRA
Oytice
00
Figure 1: Examples of the 8 benchmark tasks in U2-BENCH across diverse anatomical regions. Each callout, consisting of the question prompt, expected output format, and sample output, highlights a representa- tive ultrasound application scenario of the corresponding task. Tasks involve discase diagnosis (DD), view recognition and assessment (VRA), lesion localization (LL), organ detection (OD), keypoint detection (KD), clinical value estimation (CVE), report generation (RG) and caption generation (CG).
large vision- langunge models (LVLMs) have shown promise overcoming these barriers (Chen et al. 2024b Xia et al. 2024 Huang al., 2025 potentially offering robust multimodal understanding of complex, noisy, and context-rich ultrasound data.
While progress in medical LVLM has been rapid, most previous models and benchmarks focus those less noisy and static imaging modalities (Ji al. 2022 Huang et al. 2023 Sivasubramanian et al. (202)), leaving the complexities of ultrasound largely unaddressed. Prior efforts in ultrasound Al are typically based on small, task task-specific datasets (Xiao et al. (0225), such as fetal plane identification (Guo et al. (2024) pathology segmentation (Indelman al. [2024] Ravishankar al. (203)). As model capabilities grow, public, balanced benchmark for ultrasound understanding is needed to evaluate whether emerging LVLMs can generalize beyond static medical vision tasks, to those requiring spatial reasoning and contextual understanding of anatomical structures.
To address these challenges, we introduce U2-BENCH, the first benchmark holistically evaluating current LVLMs for ultrasound understanding across diverse tasks and anatomies. The dataset use comprises 7,241 cases across anatomical regions, involving breast, heart, lung, etc, covering diverse clinical tasks and 50 application scenarios. Each task belongs to of the four categories: (1) classification (i.e., disease diagnosis, view recognition and assessment), (2) detection (i.c., lesion localization, organ detection, keypoint detection), (3) regression (i.e., clinical value estimation), (4) text generation (i.e., report generation, caption generation). Samples are selected to ensure balance across data sources, anatomies, and task types, to enable robust evaluation and alleviate sppeset- bias. Several examples in U2-BENCH are shown in Fig.
2

---

Published as a conference paper ICLR 2026
We benchmark 20 LVLMs, including both open- and closed-source, general- -preros- and medical-spocialized models, on a diverse set of US tasks. U2-BENCH makes the following key contributions:
Comprehensive Dataset: We release the first publicly available benchmark comprising 7,241 ultrasound cases spanning 15 anatomies and 8 clinical tasks, covering 50 application scenarios. Each case is annotated with task-aligned labels in unified format and paired with carefully designed prompts, enabling standardized and reproducible evaluation.
Task Suite and Evaluation:
We define an eight-task taxonomy spanning disease diagnosis, view recognition and assessmeni, lesion localization, organ detection, keypoint detection, clinical value estimation, report generation, and caption generation. Each task reflects real-world clinical work- flows and is paired with standard evaluation metries. We also introduce an aggregate metric to provide a unified assessment of each model's overall capability in ultrasound understanding.
We conduct the first large-scale evaluation of LVLMMs ultrasound, uncovering consistent trends across model families: models achieve strong performance on image-level disease diagnosis and clinical value estimation tasks, but degrade on spatial reasoning tasks such as view recognition and organ detection. Clinical report generation tasks remain particularly challenging. Performance gains from model scaling can be limited, and compact models occasionally outperform larger ones on certain tasks, suggesting that targeted training may be more impactful than scale alone in ultrassuud understanding.
Empirical Insights:
2 RELATED WORK
Large Vision-Language Models. LVLMs such as GPT-4V (OpenAl 2023). Claude (Anthropic) 2024 Gemini (Anil et al., 2023). DeepSeek-VLL VL (DeepSeek-Al et al. 2024). LLa VA (Liu et al.) [023a)). Qwen- VI. (Bai et al. 2023b), and MiniGPT4 (Zhu et 2l23) have emerged as capable of handling tasks like image captioning. visual question ans answering, and multimodal reasoning. These models are trained on large-scale image-text pairs (Sharma et al. 2018: Schuhmann et al.] 2022), and their performance has been extensively evaluated in domains such as question answering, mathematics, and
multimodal systems
science (Chen et al. 202 I] Sun et al. 2023 Wang et al. 2023 Huang et al. 2022 Liu et al. 2023b). However, their clinical reliability remains underexplored.
To address this gap, several medical-speciallea 2023b focuses on X-ray, CT, and MRI for tasks such medical report generation, VQA, and discase identification. RadFM (Wu et al. 2023a) further supports both 2D and 3D modalities. MedDr (He et al. 2024) extends to radiology, pathology, dermatology, retinography, and endoscopy, introducing a serrieall-augmentee diagnosis strategy. Lingshu (Xu et al. (2225) is a recent medical LVLM that covers multiple imaging modalities. Yet, these models exclude ultrasound. Med-Gemini (Team 2024) and MedGemma (Sellergren et al., 2025) span numerous modalities including ultrasound, though their capability in this domain is limited caption generation.
LVLMs have been proposed. MiniGPT-Med (Wu et al.
Multimodal Benchmarks for Large Vision-Language Models. Several benchmarks assess general- domain LVLMs. MMBench (Liu et al.) 2023c). MMT-Bench (Ying et al., 2024), and SEED-Bench (Li ct al. 2023a) evaluate general-domain LVLMs through bilingual multiple-choice questions, large-scale visual reasoning tasks, and generative comprehension across image and video VQA, respectively. However, these benchmarks emphasize general- purpose visual understanding and omit clinically grounded evaluation.
Early medical VQA datasets like VQA-RAD (Lau et al. 2018). VQA-Med (Ben Abacha et al. 2019). and Path (He al. 2020) offer radiology pathology saage-question pairs but not designed for evaluating LVLMs. GMAA-MMBench (Chen et al.) 2024a) introduces a large-scale VQA-style benchmark for medical LVLMs, yet contains only about 1.4k ultrasound cases primarily focused on classification
3

---

Published as conference paper ICLR 2026
Thyruid
1465
Cervis
358
Lunn
108
Brewed 1080
Macuulosloletal System
316
Pancreos
101
Carotid Arteryy
Skin
Kb
Hiver 80G
prechialplesuss
Kidney
100
CVE
RG
C6
Heart
502
Colon
766
Prostade
100
Figure 2: Distribution of benchmark tasks across 15 anatomical regions in U2-BENCH. The colored boxes next to each anatomy name indicate the benchmark tasks available for that anatomy, with each color corresponding to one of the eight core tasks (legend shown on the right). The blue bar represents the total number of samples for each anatomy region, with its length proportional to the sample count. Multiple tasks may share samples from same anatomical region, depending on annotation availability and clinical relevance.
and segmentation 6 anatomies, and does not evaluate broader model capabilities such as clinical value estimation or structured report generation. In contrast, our U2-BENCH focuses exclusively on ultrasound and includes diverse set of clinically meaningful tasks and anatomical regions. We have also included comparison with existing ultrasound foundational datasets in Appendix
3 U2-BENCH
Overview. U2-BENCH is designed to holistically assess the capabilities of LVLM ultrasound tasks. Section 3.1 introduces the eight clinically inspired tasks involved in evaluation, which reflect essential diagnostic and reasoning abilities in ultrasound understanding. Section 3.2iionn.22eee details our benchmark construction pipeline, including dataset curation, preprocessing, and task-specific prompting. Section 3.3 summarizes the statistical property of the resulting dataset, which comprises 7,241 cases across anatomies.
3.1
TASK DEFINITIONS
U2- B2-BENCH focuses on four core capabilities: classification, detection, regression, and text generation, to systematically evaluate the performance of LVLMs on ultrasound-relatee tasks. We define eight tasks based on common ultrasound use cases, designed to probe range of multimodal abilities, including anatomy recognition and clinical reporting. The task set was informed by typical sonography workflows and refined with input from domain experts to ensure practical relevance. Together, these tasks provide structured benchmark for assessing L.L.M performance across diverse ultrasound application scenarios. The eight tasks are follows:
Discase Diagnosis (DD). This task requires the model to identify the presence and severity disease condition, such grading in the Breast Imaging Reporting and Data System, based on the appearance of the ultrasound image. The task evaluates the ability of LVLMs to extract high-level semantic features and generate clinically aligned diagnostic predictions.
View Recognition and Assessment (VRA). In clinical practice, accurate diagnosis relies on the clear presentation of anatomical structures from specific angles, referred ultrasound standard planes. This task evaluates the ability model to assess image quality and classify scans into standard planes corresponding different anatomical structures, such the fetal head or abdominal long axis.
4

---

Published as conference paper ICLR 2026
Lesion Localization (LL). Given a diagnostic image, the LVLM is asked identify the location of a lesion, such as suspicious breast mass, by selecting from nine predefined spatial categories such as upper left, center, or lower right. This task evaluates the spatial reasoning, saliency alignment, and ability to detect subtle structural abnormalities of LVLMs.
Organ Detection (OD). This task involves identifying the presence and boundaries of target organs in the ultrasound field of view, such liver, kidney, nerve. It assesses coarse-grained visual recognition under challenges unique to ultrasound, such as acoustic shadowing, inter-patient variability, and orientation ambiguity from manual probe handling.
Keypoint Detection (KD). In measurement tasks such as fetal biometry and adult echocardiography, precise localization of anatomical landmarks is critical for deriving clinically meaningful measurements. This task evaluates the fine grained spatial understanding and geometric reasoning ability of the model, which are essential for tasks like skeletal length and chamber size estimation.
Clinical Value Estimation (CVE). This task involves predicting continuous clinical parameters derived from ultrasound images, such as lesion size, left ventricular ejection fraction, or liver fat percentage. It covers both anatomical and functional indicators relevant to diagnosis, treatment planning, and longitudinal monitoring, and evaluates whether the model can perform image-to-value regression by mapping visual inputs to clinically meaningful quantitative outputs.
Report Generation (RG). The model is prompted to generate structured clinical report based visual input, following the format of example reports provided in the prompt. This task evaluates the ability of LVLM perform medical language generation and produce outputs that align with standard ultrasound reporting practices.
Caption Generation (CG). The model asked generate a concise anatomical description of a diagnostic image, guided by example captions provided in the prompt. This task evaluates basic visual-language alignment and the ability verbalize structural features in a clinically appropriate manner of LVLM.
3.2 DATA CURATION AND PROCESSING
In this section, following the approach of previous benchmark constructions (Chen al. 2024c| Xu et al. 2023 Zhong et al. 2023), we outline the three key steps used to build U2-BENCH: (1) data collection and sampling (2) data cleaning, format unification and quality verification, and (3) task-specific prompt design.
Figure 3 summarizes the data processing pipeline.
Data Selection and Sampling. We construct U2-BENCH by sampling 7,241 ultrasound studies from 40 licensed datasets. In particular, 25 out of the 40 datasets were initially identified and compiled by Alsharid ct al. (2025). For a more comprehensive dataset-level characterization of that data, refer readers that paper.
These datasets were selected represent a wide range of diagnostic tasks, anatomical regions, and clinical contexts. While the original datasets were independently curated and clinically annotated, we performed stan- dardization, sampling, and quality checks to ensure consistency across tasks and enable reliable, reproducible benchmarking. Some datasets contribute multiple benchmark tasks based on their available annotations and clinical relevance.
To reflect real-world clinical data distributions and prevent data leakage, adopt task-specific, patient-level sampling strategy. Sampling is performed at the subject level rather than the image level to preserve intra- patient consistency. Importantly, during the sampling stage, datasets corresponding to clinically high-priority but data-sparse tasks were intentionally oversampled based on guidance from collaborating clinicians. To ensure anatomical coverage, include data from 15 anatomical regions: fetus, thyroid, breast, heart, liver,
5

---

Published as conference paper ICLR 2026
Data Gathering
Task Definition
Data Preprocessing
Consulting clinical experts relevant clinical scenario
fermat,
standardization
JSON
Manual selection relevant domains and datasets
language
Balancing between tasks, anatomy and domain
Task Taxonomy
data
Capabllity
Clivical
Cauifattin
Nonis neet
40
cleaning, selection
Multimodal data gathering from the Internet
Inbis Livilidee
and quality checks
Eagii Detaciio
Prompt designing
Fagresoo
Uicensing checks and approval
U2-BENCH
Figure Overview of the U2-BENCH construction pipeline. The benchmark is built through three stages: (1) data gathering from 40 licensed ultrasound datasets spanning 15 anatomical regions, (2) task definition across 8 clinically inspired tasks grouped into four core capabilities: classification, detection, regression, and text generation, (3) data preprocessing, including annotation standardization. metadata unification, image/frame selection, and quality verification. This unified pipeline ensures benchmark consistency and clinical relevance across diverse ultrasound scenarios.
cervix, carotid artery, musculoskeletal system, kidney, prostate, skin, lung, pancreas, brachial plexus, and colon.
Data Cleaning, Format Unification, and Quality Verification. All data in U2-BENCH are standardized into a unified format to support consistent parsing and evaluation across the dataset. Ultrasound scans are converted to uniform image format. For video sequences, a small number of representative frames are sampled per study control evaluation cost while retaining key diagnostic content. Task-relevant metadata, including anatomy labels, measurements, and reports, is preserved in a structured schema. Segmentation masks are converted to bounding boxes. Texts are translated into English using a medically guided translation pipeline, with ambiguous terms resolved via curated glossary and final verification by clinicians.
To ensure the reliability of U2-BENCH, we adopt both automated and manual quality assurance procedures during data preparation.
(1) Automated Filtering. During data preprocessing, we systematically check for missing labels, inconsistent invalid annotations, and corrupted or unreadable files. Samples that fail these checks are discarded.
(2) Manual Verification. team of 10 annotators manually reviewed all cases using a cross-validation protocol, where each data point independently assessed by at least three annotators. Specifically, an engineer first check the validity of the metadata in the json file, then biomedical expert independently checked for label-image consistency, measurement units and standardized anatomical terminology. clinician then performed a final review of the diagnostic consistency on all processed cases while writing the task-specific prompts.
6

---

Published as a conference paper ICLR 2026
Task-Specific Prompt Designing. To ensure consistent model behavior and fair comparability across tasks, we design structured prompts for each of the 50 application scenarios, consisting of three components: (1) a clinical role definition to set context and expertise, (2) a task-specific instruction aligned with standard sonography workflow, and (3) an output format specification, such as classification options, value ranges, reference output examples. Detailed prompts are included Appendix D An ablation study on the impact prompt design is presented in Section 5.2
3.3 STATISTICS
U2-BENCH comprises 7,241 ultrasound studies spanning benchmark tasks and 15 anatomical regions. Table Appendix dettails the number of cases per task. Classification and detection constitute the largest shares, with 2,999 and 2,921 cases, respectively, while generation and regression tasks provide targeted evaluation of report synthesis and clinical value estimation.
Figure summarizzs the distribution across anatomical regions. Thyroid and breast ultrassundd together account for more than one-third of all cases. This is because of their high clinical prevalence and broad diagnostic utility. Many anatomies support multiple tasks for instance, fetal ultrasound used for classifica- tion and regression enabling multi-task evaluation within a unified anatomical context. This composition ensures broad coverage across modalities, tasks, and body regions, supporting robust and clinically grounded assessment of LVLM performance.
4 EXPERIMENT
4.1 EVALUATION SETTINGS
We evaluated U2-BENCH on 20 LVLMs, both open-source and closed-soured source. detailed list of the exact model versions evaluated and additional experimental details are provided in Appendix C, with the full implementation and hyperparameter configurations available our public code repository. Detailed prompts are given in Appendix D
4.2 EVALUATION PROTOCOL
We employed standard metrics aligned with clinical relevance and prior LVLM benchmarks. Classification tasks were evaluated with accuracy and Fl score. For detection-related tasks, we initially evaluated models using the ground-trui bounding box or coordinate outputs. However, many LVLMs failed reliably generate valid coordinates or follow bounding- box formatting instructions. To enable stable and comparable evaluation across models, we therefore simplified the detection tasks into a 9-class position-classifiction formulation, where each region corresponds to a coarse spatial sector of the image. Under this formulation, detection tasks utilized accuracy as the metric to assess localization correctness. Regression tasks report Root Mean Squared Error (RMSE). Mean Absolute Error (MAE), and percentage within tolerance (%_tol). Generation tasks were assessed with BLEU-4 as percentage, ROUGE, and BERTScore (Zhang* et al. 2020) to capture both lexical and semantie similarity. All metries were computed using ground- labels from the original dataset and standardized outputs with the format specified by the prompts across models to ensure fair comparison.
U2-Score. We design quantitative score provide overall evaluation metric for the ultrasound understanding capability of model. The U2-SCORE defined as a weighted combination of the metrics across all tasks, which is mathematically equivalent computing case-level average, consistent with prior
7

---

Published as conference paper ICLR 2026
work (Chen al. 2024a). This can be formulated as:
and de S U2-Score = widy, where We En
where represents the number of tasks, wy is the corresponding task weight, which is computed from the proportion of the sample number of the /-th task. This can mitigate the imbalance issue of sample size in different tasks. Here, de denotes the value of the selected metric of the /-th task. More details are included Appendix
4.3 EVALUATION RESULTS
We present a comprehensive comparison of multimodal models on the U2-BENCH benchmark (Table I aiming to identify key performance trends across tasks and model types. more detailed example cases and error analysis is included in Appendix C
Closed-Source Models Lead. Closed-source models continue dominate, with Dolphin-V V1 achieving the highest overall score of 0.5835, substantially outperforming all other models. The next strongest proprietary model, GPT-5, reaches a U2-Score of 0.3250, followed by Gemini-2.5-Pros 1ii------rrr-riieee at 0.2968. While the best open-source model, DeepSeek-VL2. attains a competitive score of 0.2630, the gap to closed-source systems remains significant. These results highlight that despite rapid advances in open-source approaches, closed-source models still benefit from access to larger proprietary datasets and tailored optimization, giving them a clear performance edge.
Task Difficulty Varies Significantly. Image classification tasks remain the most tractable, with Dolphin-VI achieving the highest accuracy 0.682 on DD, and several other models exceeding 0.48. contrast, spatial reasoning and text generation remain difficult: no model surpasses 0.160 accuracy on KD, and all models fall below 7.5 BLEU on RG. Regression tasks are also challenging: only the closed-souree Qwen-Max reduces RMSE to 0.1248, while all open- source models remain above 0.1675.
Scaling Brings Diminishing Returns. Within the Qwen-2.5-VL family, scaling from 3B to 72B parameters yields consistent performance gains. While larger models achieve lower CVE RMSE, improvements in language generation and spatial reasoning tasks plateau, suggesting that excessive scaling may lead overfitting on superficial visual patterns, ultimately harming clinical text generation capabilities.
Domain-Specific Models Excel in Reasoning. Medical-domain models such MedDr show competitive performance reasoning tasks (c.g.. CVE RMSE= 0.214; CG BERT = -81.21), outperforming many general- purpose systems in structured clinical evaluation. Similarly, MedGemma-4B-it achieves the second-best CVE performance (RMSE = O. 0.7, highlighting the advantage of domain adaptation for quantitative reasoning. However, these models still lag behind larger general multimodal models on visual classification. For example, Qwen-72B achieves a DD Fl of 0.456, compared to MedDr's 's 0.312. This suggests that domain comain-specialized models are particularly effective for semantie and reasoning- tasks, while general purpose models retain edge in coarse-grained visual recognition.
4.4 LIMITATIONS AND FUTURE OUTLOOK FOR ULTRASOUND LVLMS
While existing LVLMs demonstrate impressive general multimodal capabilities, our results reveal fundamental limitations in ultrasound-specfft. perception and clinical reasoning.
8

---

Published as a conference paper at ICLR 2026
Table 1: Results of different models on the U2-BENCH. We utilize green (Ist), blue (2nd), and yellow (3rd) backgrounds to distinguish the top three results within different models. The "U2-Score" column represents the quantitative score defined in Section 4.2 BLEU scores are taken to zero
To calculate the U2-SCORE for random guessing, the
Madak
FTT
VIA
FT7
KD
MTA
CYE
9,307
BPRTST
CG
12 13050
Tanken Cuansing
34345
0.4155
0.3105
0.3184
CXCBO
0.133
0:472
15.77
MakIFFMMec Mad
37661
0.180
0.122
0.799
01643
0.72%
0,500
333259
64000
20,1000
746606
3U300
47.7900
$1.500
Metly
0.49%
0.301
00028
01903
12.309
33,009
21626
$1.311
19974
01:991
0.38%
01100
00846
D1843
01654
0 0.3055
0L2019
1.555
5.811
123027
13354.1
86/1413
71,050
0,300
Queer 3.5 VL. Sueract
4882
0.380
0.38
0.30
0.100
00755
01080
0.3414
0.3015
1,500
13.000
08.140
14.750
38.000
77.380
03365
Qeecr 21-- bourdd
03.200
02209
01385
01150
01060
0:323
122233
3152
3,900
133800
123850
20.000
44:291
91.000
03321
3.4200
0.310
0.36
6111
03568
0.133
3.350
0.2506
133335
2,550
265400
783880
11-280
3-3900
77.300
0.000
01100
01100
0.9966
0.2580
0.1148
03362
03333
00050
038.3
0.160
0.224)
0
19.352
5,230
15.3800
13.210
33390
22:9900
75,090
Misiial Souil Focd
43990
0.94%
0.941
0/664
0.110
019/3
0.063
1.675
01331
1830
11.000
712700
307700
03:130
01.100
0.796
3.3591
0,2377
0,201
02141
01100
0.124
0 0.366
0L33T
33 13.1731
0.700
0.540
12,400
1.400
15.300
75:43
01287
0191
01161
01643
0.353
71.500
357700
GFE3
2.339
13:50
04873
03358
02262
01157
0.183
0.5977
0L1573
1355
1.5911
$348)
662302
15999
233115
72.230
03220
Geneail 1--thu
2.781
0.2347
0.30
00818
02700
03692
0.275
0.220
403231
33430
11,500
25,00
45/200
20020
Chande Soocet
33331
01500
01:51
0.156
00510
00753
9 1354
0.150
08:015
12.3700
66.700
1.3000
16,900
71,000
2.4896
0
0.902
008-1
01543
11335
102543
19:339
3,900
11,000
135850
30810
20/3303
02.300
0,243
Dilphas/V
3.819
0.515
0.997
[093))
0/000
82000
02400
0,227
33300
15:1130
73.7387
5:0034
760111
02.900
0500
Ducise Keypoint Detection: Clinical Value Esimation: Report Generaiin; =Cpption Generation.
VRA
View
and
L1,
Localizaion; 01
Organ Detection:
Weak Perception of Ultrasound Structures. Models struggle with recognizing relative spatial relationships between anatomical structures, as reflected by their poor performance on detection tasks, and often fail to capture subtle echogenicity patterns that essential for clinical diagnosis. This likely stems from the lack of large-scale ultrassuud-speccffi image-caption pretraining data and the inherently noisy, heterogeneous nature of ultrasound imaging. Improving perception would require curated ultrasound datasets, ultrasound-aware pretraining objectives, and architectures or adapters with explicit spatial-reasoning capabilities.
Clinical Ultrasound Tasks Are Far More Complex Than Generic LanguaLa Tasks. Ultrasound spans more than 15 clinical subspecialties, each with distinct anatomical structures, scanning planes, and diagnostic criteria. For example, fetal biometry requires standardized abdominal circumference (AC) or head circumference (HC) views, while cardiac ultrasound relies parasterall long-axis apical four-chamber views. A clinically useful LVLM must therefore understand specialty-specffic anatomy, follow established scanning protocols, and reason according diagnostic workflows.
5 ANALYSIS
5.1 INSTRUCTION FOLLOWING ANALYSIS
Table shows that contemporary models are already highly adept at parsing prompts and adhering to output specifications: six of the seventeen systems achieve a perfect score on the DD benchmark. The remaining models lag only slightly behind. The medical-oriented MiniGPT-Med (Alkhaldi et al. 2024) and MedDr (He) et al. 2024) deliver middling results, while Qwen-3B and Qwen- 72B (Bai et al., (023b) close the gap rapidly their parameter counts increase. Claude-3.7 (Anthropic) 2025) score of 0.942 is largely attributable occasional formatting omissions. For every non non-perfect model, the deviation from the maximum is under six percentage points, and no systematic failures observed. We also note that some models occasionally
9

---

Published as conference paper ICLR 2026
refuse to answer due internal safety constraints, producing responses such as "insufficieet information" "I cannot provide medical advice", rather than simply failing to follow instructions. Table Instruction following comparison across different models.
Taok
Nookds
D0
0J32
0.61
0J66
0.383
0/96
1,80
1,000
1100
0.87
0/95
0.999
1,000
1J00
10000
10R2
5.2 PROMPT WITH OR WITHOUT ANATOMY
We investigate whether explicitly naming the anatomical region in the prompt significantly changes the diagnostic accuracy of LVLMs in ultrasound. To this end, we treat the two prompt variants as paired conditions applied to the same set of inputs and evaluate the statistical significance of their differences using McNemar's test.
Specifically, for each image Ty, we generate two prompts:
With anatomy: "You radiologist amalysing (anatomy) ultrasound image, please analyze..."
No anatomy: "You are a radiologist analysing an ultrasound image, please analyze..."
Each pmompt-image pair is forward- pewaard-assedd through the model five times, with the final prediction determined by majority vote. This produces paired outcomes (y.ith y, givihouut for each image. Experiment was conduced on 521 breast and thyroid studies from our dataset, the following paired contingency table presents the result for model 1onnii-2.00-Po-EE..
Table 3: Effect of anatomy tokens in prompt design. Paired outcomes of 521 samples comparing prompts with and without anatomy tokens. Each entry shows the number of samples that outcome combination.
No-anatomy prompt
With-anatomy prompt
Correct
Incorrect
Correct Incorrect
209 (both correct)
64 (only anatomy correct)
26 (only no-anatomy correct)
222 (both incorrect)
McNemar's exact test yields a test statistic x2 16.04 with p 6.2 10 providing strong evidence that the two conditions differ. Specifically, prompts with anatomy tokens achieve an accuracy of 52.4% versus 45.1% % without, gain +7.3 percentage points.
The McNemar test confirms that the inclusion of anatomy information in the prompt significantly improves diagnostic accuracy, rejecting the null hypothesis of no difference between prompt types.
6 CONCLUSION
Ultrasound is essential to global healthcare but remains difficult to interpret. We present U2-BENCH, the first benchmark for evaluating LVL.Ms on ultrasound understanding. It includes 7,241 cases across 15 anatomical regions and defines clinical tasks for 50 application scenarios. Evaluating 20 LVLMs, we find their strong performance in classification but persistent challenges in spatial reasoning and clinical text generation, suggesting a future direction for improving LVLMs on ultrasound interpretation.
10

---

Published as a conference paper ICLR 2026
Acknowledgements Research conducted Dolphin AI. We acknowledge support from Hong Kong Research Grants Council (RGC) Early Career Scheme grant (Grant No. 22203525) and Fudan University.
Reproducibility Statement We have taken extensive measures to ensure the reproducibility of our work. The benchmark dataset (7.241 ultrasound cases across 15 anatomies and tasks) is publicly available on 4open F Detailed descriptions of dataset curation, preprocessing. and
HuggingFace, and the complete evaluation toolkit is released anonymously at wwwence/rr/-Beennhheeeee
quality verification procedures are provided in Section 3.2 and Appendix C-D, including sampling strategies, annotation protocols, and prompt templates. The full list models evaluated, along with task-specific metrics and the aggregate U2-Score formulation, is given in Section 4.2 and Appendix C. For reproducibility of theoretical and statistical analyses (c.g., McNemar test for prompt design), contingency tables and test statistics are reported in Section 5.2.
REFERENCES
Marah I Abdin, Jyoti Aneja, Harkirat S. Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi- 4 technical report. CoRR, 0200000 2024. doi: 10.48550/ARXIV.24122220
URL https
48550/ 2412. 08905
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in Brief, 28: 104863, 2020. ISSN 2352-3409. doi: https:///oii 100 1016/j. wttp://.bb...11119.. 2019. 104863.
URL. https: / htt.a//wwwa.ccc.......oooooaaata
Asma Alkhaldi, Raneem Alnajim, Layan Alabdullatef, Rawan Alyahya, Jun Chen, Deyao Zhu, Ahmed Alsinan, and Mohamed Elhoseiny. Minigpt-med: Large language model a general interface for radiology diagnosis. arXiv preprint vxxiv:2407.04106 2024.
Mohammad Alsharid, Xiaoqing Guo, Qianhui Men, Pramit Saha, Divyanshu Mishra, Rahul Ahuja, Cheng Ouyang, and J Alison Noble. On the public dissemination and open sourcing of ultrasound resources, datasets and deep learning models. npj Digital Medicine, 2025.
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou. Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312.11805, 2023.
Anthropic. Claude 3.5 sonnet. Anthropic News, June 2024. URL https://ww..nntinopc...c.///e claude-3: 3- --nnne-----oonet
Anthropic. Claude 3.7 sonnet. Anthropic News, February 2025. URL http:p///www.atttroocccccco//eer claude- 3onnett
Jieyun Bai. Psfhs, April 2024. URL https:://oiioorg/////// 0t811/eeo2221/ 10969427

---

Published as a conference paper ICLR 2026
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang. Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv: I669, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision- assoonlanguage model with versatile abilities. CoRR, abs/2308. 12966, 2023b.
Asma Ben Abacha, Sadid A. Hasan, Vivek V. Datla, Joey Liu, Dina
and Henning Müller. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Working Notes of CLEF 2019, volume 2380 of CEUR Workshop Proceedings, Lugano, Switzerland, September 9-12 2019. 272.
CEUR-WS.org. URL //
Yuan Bi, Zhongliang Jiang, Ricarda Clarenbach, Reza Ghotbi, Angelos Karlas, and Nassir Navab. Mi- segnet: Mutual information based us segmentation for unseen domain generalization, 2024. URL https
7/arxiv. orrg/v.orrrro www.oog//////a/ae /ttp//3/3... 12649
Xavier P. Burgos-Articn David Coronado- Gutierrezz,, Brenda Valenzuela-Alcam Elisenda Bonet-Carne Elisenda Eixarch, Fatima Crispi, and Eduard Gratacós. Fetal planes db: Common maternal-fetal ultra- soundimages, June 2020. URL, /ttttss://oii.rr// 5281/zenodo. 3904280
Michal Byra, Grzegorz Styczynski, Cezary Szmigielski, Piotr Kalinowski, Lukasz Michalowski, Rafal Paluszkiewicz, Bogna
Krzysztof Zieniewicz, Piotr Sobieraj, and Andrzej Now- icki. Dataset of b-mode fatty liver ultrasound images, August 2018. URL http:::/doi..orggggtgo: 5281/
zenodo. 1009146
ByteDance. Doubao model series.
https::/www.doulee com/ 2024. Accessed: 2025-05-14.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint urXiv:2107.03377 2021.
Pengeheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J Seibel, Junjun He, and Yu Qiao. Gmai--mmeench: A comprehensive multimodal evaluation benchmark towards general medical ai, 2024a. URL httpp::t///tii.tttt/// http:////aaaiii.oorab/////////
03361
Pengeheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J Seibel, Yu Qiao, and Junjun He. Gmaiimmmennhh A comprehensive multimodal evaluation benchmark towards general medical ai. In A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 94327-94427. Curran Associates, Inc., 2024b. URL httht:pp//// httpttppsp///prrreeiigggggg www.eeiiiiiinnnnnniinooeeee wttpspprcpppppeeeeeeeeeeeeee httppppprr.illleepprr///////// #ile/abbee22f66044722233///333/3 oaapsr/faaoofffoeeeeeererrrattae PaweoeeeeePa----aaaaaeeaee
and Benchmarks Trasss__rrrco__ pdf
Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. Tombench: Benchmarking theory of mind in large language models. In Lun-We Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024,
12

---

Published as a conference paper ICLR 2026
Bangkok, Thailand, August 11-16, 2024, pp. 15959-15983. Association for Computational Linguisties, 2024c.
Matthew Christensen, Milos Vukadinovie, Neal Yuan, and David Ouyang. language foundation model for echocardiogram interpretation. Nature Medicine, (005))14411-445 May 2024. ISSN 1546-170X. doi: 10.1038/s41199) 15120444959-022400099999 http::tt/////ooorrgg///// 10t.
Karine Souza Da Correggio, Roberto Noya Galluzzo, Luís Otávio Santos, Felipe Soares Muylaert Barroso, Thiago Zimmermann Loureiro Chaves, Alexandre Sherlley Casimiro Onofre, and Aldo von Wangenheim. Fetal abdominal structures segmentation dataset using ultrasonic images, 2023. URL httpt:tttt/ooooooo
10. 12e11763/444 10.11632/4gcmmsdscc3
DeepSeek-AI, Deepseek-ve v2: strong, economical, and efficient mixture-of-eepprts language model, 2024.
DeepSeek-Al, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin. Huazuo Gao, Hui Li, Hui Qu, L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang. Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. Chen, R. L. Jin, Ruigi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. Deepseck-v3 technical report. CoRR, abs/2412. 19437, 2024. doi: 10.44550/RXXV..21222000
URL http:: org/10 48550/arXiv 2412. 19437
Atharva Divekar and Atharva Sonawane. Leveraging ai for automatic classification of peos using ultrasound imaging, 2024. URL http://ttt//aa...ooo////////////
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang. Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. VImevalkit: An open-source toolkit for evaluating large mudti-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198 11011, 2024.
FALLMUD. FALLMUD: Fasiile lower leg muscle ultrasound dataset. https: cea. fr/index php/fallmud/ 2025. Dataset composed of 812 lower leg muscle ultrasound images with segmentation masks, used for muscle structure analysis and injury prevention.
Xiaoqing Guo, Qianhui Men, and J. Alison Noble. MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video, October 2024. URL
03761 arXiv:2408.0376) [cs].
Sunan He, Yuxiang Nie, Hongmei Wang, Shu Yang. Yihui Wang, Zhiyuan Cai. Zhixuan Chen, Yingxue Xu, Luyang Luo, Huiling Xiang, Xi Lin, Mingxiang Wu, Yifan Peng, George Shih, Ziyang Xu, Xian Wu, Qiong Wang, Ronald Cheong Kin Chan, Varut Vardhanabhuti, Winnie Chiu Wing Chu, Yefeng Zheng, Pranav Rajpurkar, Kang Zhang, and Hao Chen. Gsco: Towards generalizable in medicine via generalist-speciiliss collaboration, 2024. U httpps:/aarxw.oor//aaaasssse
15127
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003 10286, 2020.
13

---

Published as conference paper ICLR 2026
David W. Hewson and Nigel M. Bedforth. Closing the gap: artificial intelligence applied ultrasoundguuied regional anaesthesia. British Journal of Anaesthesia, 130(3):245-244 2023. ISSN 0007-0912. doi: htt.://.........................
https://doo.org//0.111116 2j..2222..220005 12.000. 2/0/990007091122206222
URL https::/wwwt sciencedirect. com//science//rttclle.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv: Arxiv-2201.07207 2022.
Xiaoshuang Huang. Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang, and Yehui Yang. Towards multimodal large language model with pixel-level insight for biomedicine. Proceedings of the
AAAI Conference on Artificial Intelligence, 39(4): Apr. 2025, doi: (0.1609/ammi.v3944.333 URL https://ojs aaai org hetg//inex....o hhtp//////aarrtcee////eee///////
Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J. Montine, and James Zou. A visual-Language foundation model for pathology image analysis using medical twitter. Nature Medicine, 29(9):2307-2311 1038/
Sep 2023. ISSN 1546-170X.
doi: 1..1088/41191-002--000000 URL
541591-023-02504 02504-3
Hedda Cohen Indelman, Elay Dahan, Angeles M. Perez-Agosto, Carmit Shiran, Doron Shaked, and Nati Daniel. Semantic Segmentation Refiner for Ultrasound Applications with Zero-Shot Foundation Models, April 2024. URL http::/t/aaxi..rrgg//g/o// /2040 16325 163355 [cs] version: 1.
Yuanfeng Ji, Haotian Bai, Chongjian GE, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, and Ping Luo. Amos: large-scale abdominal aagti benchmark for versatile medical image segmentation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems, volume 35, pp. 36722-36732. Curran Associates, Inc., 2022. URL wtw.httpss/// htt.://0......a--aor-oooooeataa.
ptattpss//ppooceedigg...ot neuripss / http://wweeciiiiee.eeerrrs/eeees
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
Jing Jiao, Jin Zhou, Xiaokang Li, Menghua Xia, Yi Huang, Lihong Huang, Na Wang, Xiaofan Zhang, Shichong Zhou, Yuanyuan Wang, and Yi Guo. USFM: universal ultrasound foundation model generalized to tasks and organs towards label efficient image analysis. Medical Image Analysis, 96: 103202, August 2024. ISSN 13618415, doi: 10. 10.101 0016:..media.2244. 103202. URL httptttt//ikkiiighuuu elsevier com/
rerr1ewee/p11/111811444444449
Sekeun Kim, Pengfei Jin, Sifan Song, Cheng Chen, Yiwei Li, Hui Ren, Xiang Li, Tianming Liu, and Quanzheng Li. Echofm: Foundation model for generalizable echocardiogram analysis. arXiv preprint arXiv:2410.23413. 2024.
Markus Krönke, Christine Eilers, Desislava Dimova, Melanie Köhler, Gabriel Buschner, Lilit Schweiger, Lemonia Konstantinidou, Marcus Makowski, James Nagarajah, Nassir Navab, Wolfgang Weber, and Thomas Wendler. Tracked ultrasound and deep neural network-based thyroid segmentation reduce interobserver variability in thyroid volumetry. PLOS ONE, 17(7): 07 2022. doi: 10. pone.0268550. URL httpps:///oi..o org/10 1371 1371/journal. pone. 0268550
Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific Data, 501):10021, Nov 2018. 0tt.1/...131//00 2018.251
ISSN 2052-4463 doi: 10. 1038/sdata.2018. URL httpt:///o: wtt.://....//......ooooo.o
14

---

Published as conference paper ICLR 2026
Alexandra Laverde Saad, Abdulhadi Jfri, Rubén García, Irene Salguero, Constanza Martinez, Hirune Cem- brero, Gastón Roustán, and Fernando Alfageme. Discriminative deep learning based benignity/mallinaaee diagnosis of dermatologic ultrasound skin lesions with pretrained artificial intelligence architecture. Skin Research and Technology, 28, 08 2021. doi: 10.1 1/srt. 13086.
Sarah Leclere, Erik Smistad, João Pedrosa, Andreas Østvik. Frederic Cervenansky, Florian Espinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Mare Jodoin, Thomas Grenier, Carole I Lartizien, Jan D' D'hooge, Lasse Lovstakken, and Olivier Bernard. Deep learning for segmentation using an open large-scale dataset in echocardiography. IEEE Transactions on Medical Imaging. 38(9):2198-22 22100, 2019. doi: 10.1109/TMI.2099..0 2900000
Ramona Leenings, Maximilian Konowski, Nils R. Winter, Jan Ernsting. Lukas Fisch, Carlotta Barkhau, Udo Dannlowski, Andreas Lügering, Xiaoyi Jiang, and Tim Hahn. C-trus: A novel dataset and initial benchmark for colon wall segmentation in transabdominal ultrasound. In Alberto Gomez, Bishesh Khanal, Andrew King, and Ana Namburete (eds.), Simplifying Medical Ultrasound, pp. 101-111, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-73647-66
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-hench: Benchmarking multimodal lims with generative comprehension. arXiv preprint 1riv:2337.166125, 2023a.
Jiajia Li, Pingping Zhang, Teng Wang, Kaixuan Wang, and Bin Sheng. Lepset, June 2023b. URL https: //doi.org/10. 5281/zenodo...oo 8641285
Jun Li, Tongkun Su, Baoliang Zhao, Fagin Lv, Qiong Wang, Nassir Navab, Ying Hu, and Zhongliang Jiang. Ultrasound report generation with cross-modality feature alignment via unsupervised guidance, 2024. URL
https://arxiv.orr/bbb//// 00644
Zhi Lin, Junhao Lin, Lei Zhu, Huazhu Fu, Jing Qin, and Liansheng Wang. new dataset and baseline model for breast lesion detection in ultrasound videos. In Linwei Wang, Qi Dou, P. Thomas Fletcher, Stefanie Speidel, and Shuo Li (eds.), Medical Image Computing and Computer Assisted Intervention MICCA/ 2022, pp. 614 623, Cham, 2022. Springer Nature Switzerland.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. AgentBench: Evaluating LLMS as agents. arXiv preprint arXiv:2308..m 03688, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? mrxivv2307.06288 2023c.
Yaosheng Lu, Mengqiang Zhou, Dengjiang Zhi, Minghong Zhou, Xiaosong Jiang, Ruiyu Qiu, Zhanhong Ou, Huijin Wang, Di Qiu, Mei Zhong, Xiaoxing Lu, Gaowen Chen, and Jieyun Bai. The jnu-ifm dataset for segmenting pubic symphysis-fetal head. Data Brief, 41:107904, 2022. ISSN 2352-3409. doi:
https://doi.org/10..0001 101l/////..0..0...ooooooooooo w/tp:///...bbb..0.. 2tb.22222.107000 URL hhtpps://www http://wwwwwwwwwwwwwc......c.cce com/sccience/articlle
p11/S23523409228811 160
Fadillah Maani, Numan Saced, Tausifa Salcem, Zaid Farooq, Hussain Alasmawi, Werner Dichl, Ameera Mohammad, Gareth Waring, Saudabi Valappi, Leanne Bricker, and Mohammad Yaqub. FetalCLIP: Visual-Language Foundation Model for Fetal Ultrasound Image Analysis, February 2025. URL http:
7/arxiv. wwwpxxvvoogg//aao wtt..rrggabs///....... 14807 arXiv:2502. 14807 [eess] version: 1.
15

---

Published as conference paper ICLR 2026
Azouz Maroua. Algerian ulttrasound images thyroid dataset (auitd). https://www. kaggle. com/datasets Ahouzmaroua//lgeriiia trasound- images- thyroid-dataset- aui td 2020. Kaggle dataset contain- ing 3-class thyroid ultrasound images (Benign, Malignant, Normal) collected from hospitals Setif, Algeria. Accessed May 2025.
Kristen M. Meiburger, Guillaume Zahnd, Francesco Faita, Christos Loizou, Catarina Carvalho, David Stein- man, Lorenzo Gibello, Rosa Maria Bruno, Francesco Marzola, Ricarda Clarenbach, Martina Francesconi, Andrew Nicolaides, Aurelio Campilho, Reza Ghotbi, Efthyvoulos Kyriacou, Nassir Navab, Maura Griffin, Andrie Panayiotou, Rachele Gherardini, Gianfranco Varetto, Elisahetta Bianchini, Constantinos Pattichis, Lorenzo Ghiadoni, José Rouco, and Filippo Molinari. DATASET for "Carotid Ultrasound Boundary Study (CUBS): an open multi-center analysis of computerized intima media thickness measurement systems and their clinical impact", 2021. URL ptttp:://doi.org/111 17632/ 17632/f
Adrien Meyer, Aditya Murali, Didier Mutter, and Nicolas Padoy. UltraSam: A Foundation Model for Segmentation Datasets, November 2024. URL httttt://///iv.ooo
Ultrasound using Large ab5/2411. 16222 MrXiv:2411.1622 [eess] version: 1.
Yuhao Mo, Chu Han, Yu Liu, Min Liu, Zhenwei Shi, Jiatai Lin, Bingchao Zhao, Chunwang Huang, Bingjiang Qiu, Yanfen Cui, Lei Wu, Xipeng Pan, Zeyan Xu, Xiaomei Huang, Zaiyi Liu, Ying Wang, and Changhong Liang. Hover-trans: Anatomy-aware hover-transformeg for roi-free breast cancer diagnosis in ultrasound images, 2022. URL httpt:ttp//sxiiiii org
abs/2205. http://w...r..////..////// 08390
NeuronXJTU and palkia1999, Kfgnet: Source code for video classification using ultrasonic data. https:
//github. wwwm//eurooxJTU/KFFGCCC 2023. GitHub repository, accessed May 2025. Includes Baidu
Netdisk link to ultrasonic data.
Novin, C. Alvarez, J. B. Renner, Y. M. Golightly, and A. Nelson. Features of knee and multijoint osteoarthritis by sex and race and ethnicity: A preliminary analysis in the johnston county health study. Journal of Rheumatology, pp. jrreumm.02330447. 2023-0479, September 2023. doi: 10.38999jrheumm2223--00 http://eeeum..24479999 Epub ahead of print, published September 15, 2023.
OpenAL. Gpt-4 technical report. PREPRINT; 2023.
Collaborator Pahuni Choudhary. Carotid artery ultrasound and color doppler: Multimodal carotid artery imaging with comprehensive patient health records. https://www/ http:://wwwkkkgleeee com/datasets/ jahunichoudharyyccrrtte
2023. Kaggle dataset. In- cludes ultrasound and color Doppler images from 18 patients with structured health records. Licensed under Apache 2.0. Accessed May 2025.
Anna Pawlowska, Anna Swvierz-Pierkkowske Agnieszka Domalik, Dominika Jagus, Piotr Kasprzak, Rafat Matkowski, Lukasz Fura, Andrzej Nowicki, and Norbert Zotek. Curated benchmark dataset for ultrasound based breast lesion analysis. Scientific Data 11(1):148, Jan 2024. ISSN 2052-4463. doi: 1031038/
s411977-22-022984- URL https: www.o..///00. 01880/1010100//41155-------88880
Lina Pedraza, Carlos Vargas, Fabián Narváez, Osear Durán, Emma Muñoz, and Eduardo Romero. An open access thyroid ultrasound image database. In 10th International symposium on medical information processing and analysis, volume 9287, pp. 188-193. SPIE, 2015.
Bharath Prabakaran, Paul Hamelmann, Erik Ostrowski, and Muhammad Shafique. Fpus23: An ultrasound fetus phantom dataset with deep neural network evaluations for fetus orientations, fetal planes, and anatomical features. IEEE Access, PP:1-1, 01 2023. doi: 10.1 109//CCCSSSSSC 202.33332..0232888111
16

---

Published as a conference paper ICLR 2026
Hariharan Ravishankar, Rohan Patil, Vikram Melapudi, Harsh Suthar, Stephan Anzengruber, Parminder Bha- tia, Kass-Hout Taha, and Pavan Annangi. SonoSAMTrack Segment and Track Anything on Ultrasound Images, November 2023. BRLLittpp//axxi..or///bss
16872 arXiv:2310. 1672 [eess].
Vuppala Adithya Sairam. Ultrasound breast images for breast cancer. htpps: /www: kaggle com/datasets/
aryashah2k wwwp:///w/wuutttsouoooooorrreeee wewrresst-iagee--eeee for---------------------eeeeee 2020. Kaggle dataset. CCO: Public
Domain license. Accessed May 2025.
María Sofía Sappia. Acouslic-ai Abdominal circumference operator- agnostic ultrasound measurement low-income countries using artificial intelligence, July 2024. URL https:://oi..rg//// 0tt.///0..22//zzo/d 12697994
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 22125278-25294 2022.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint 20Xi::2507.05201, 2025.
Carla
Victor M. Campello, Jordina forrents-Barrent Yahya Ali Ahmed, Mustafa Elattar, Benard Ohhne--Botwe Pempho Nyangulu, William Stones, Mohammed Ammar, Lamya Nawal Benamer, Harriet Nalubega Kisembo, Senai Goitom Sereke, Sikolia Z. Wanyonyi, Marleen Temmerman, Eduard Gratacós, Elisenda Bonet, Elisenda Eixarch, Kamil Mikolaj, Martin Grønnebæk Tolsgaard, and Karim Lekadir. Generalisability of fetal ultrasound deep learning models to low-resource imaging settings in five african countries. Scientific Reports, 13(1):2728, Feb 2023. ISSN 2045-2322. doi: 10.1038/ UtLhhtts:///do. http:///ooooorg/////// 1008000331555 4415//-125553 0200-222-2------
Wei Shao and Wayne Brisbane. Micro-ulrasound prostate segmentation dataset, January 2024. URL https://doi.orge 2281/2eenodo118775599
Harshita Sharma, Lior Drukker, Aris T. Papageorghiou, and J. Alison Noble. Machine learning- Seannnggbbased analysis of operator pupillary response to assess cognitive workload in clinical ultrasound imaging. Computers in
Biology and Medicine, 135:104589, 2021, ISSN 0010-4825. doi: https://doi.org/10..000 101 com.ooooooedeieoooooommimeeeeedd
2021.104589 htt/shtps::// //wepp:://wwwwr www. sciencedirect. com/ wwwssscceeeeatieeeeee acciencc//artiicl///////000 04825210008333
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual 2018.
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
Matthew Shun-Shin. The imperial ai echocardiography dataset. https: //uni wwwpuniittiiiininnnn net 2023. An open-access dataset of over 7500 labele- echocardiographic images for Al development in cardiology. IRAS: 279328, REC: 20/SC/0386. Accessed May 2025.
Rohit Singla, Cailin Ringstrom, Grace Hu, Victoria Lessoway, Janice Reid, Christopher Nguan, and Robert Rohling. The open kidney ultrasound data set. In Bernhard Kainz, Alison Noble, Julia Schnabel, Bishesh Khanal, Johanna Paula Müller, and Thomas Day (eds.), Simplifying Medical Ultrasound, pp. 155-164, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-44521-7.
Sithursan Sivasubramaniam, Cedric Osei-Akoto, Yi Zhang, Kurt Stockinger, and Jonathan Fürst. Smm-rext-to--uery: Synthetic multi-model medical text-to-query benchmark. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37, pp. 88627-88663. Curran As-
17

---

Published as a conference paper ICLR 2026
sociates, Inc., 2024. URL https://prooeeddigg.eeuu wttp:pps.ccppp....eeee //tps/////ppp/ll//////////////// aB2566661226666632221700000-00r0
and Benchmarks Bee Track. pdff
Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Sci- Eval: A multi-level large language model evaluation benchmark for scientific research. arXiv preprint arXiv:2308. 13149, 2023.
Google Team. Advancing multimodal medical capabilities of gemini. CoRR, abs/2405.03162. 2024.
Abhishek Tyagi, Abhay Tyagi, Manpreet Kaur, Richa Aggarwal, Kapil D. Soni, Jayanthi Sivaswamy, and Anjan Trikha. Nerve block target localization and needle guidance for autonomous robotic ultrasound guided regional anesthesia, 2024. URL htttp:://arxxw.orrr/bb////
03717
Thomas L. van den Heuvel, Dagmar de Bruijn, Chris de Korte, and Bram van Ginneken. Automated measurement of fetal head circumference using 2d ultrasound images, July 2018. URL https://dol org/10. 5281/2enod.. 1327317
S. Vitale, I. Orlando, E. larussi, and I. Larrabide. Improving realism in patient-specific abdominal ultrasound simulation using cyclegans. International Journal Computer Assisted Radiology and Surgery, 15(2): 183-192, February 2020. doi: 10.1007/s1 11077//115488
Epub 2019 Aug 7.
Qianye Wang, Yipeiand Yang. Lior Drukker, Aris Papageorghiou, Yipeng Hu, and J. Alison Noble. Task model-specific operator skill assessment in routine fetal ultrasound scanning. International Journal of Computer Assisted Radiology and Surgery, 17(8):1437-1444 Aug 2022. ISSN 1861-6429, doi: 10.1007/s1 022-02244-yy.
URL/https://doi org.0/........0.. 1007/s11 548 022-02642-
Xiaoxuan Wang. Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SeiBench: Evaluating college-level scientific problem- solving abilities of large language models. arXiv preprint arXiv:2307.1063: 2023.
Nina Wiedemann, Dianne de Korte-de Boer, Matthias Richter, Sjors van de Weijer, Charlotte Buhre, Franz A. M. Eggert, Sophie Aarnoudse, Lotte Grevendonk, Steffen Röber, Carlijn M.E. Remie, Wolfgang Buhre, Ronald Henry, and Jannis Born. Covid-blues a prospective study on the value of ai in lung ultrasound analysis. IEEE Journal of Biomedical and Health Informatics, pp. 1-12, 2025. doi: 10. 100. 3543686.
Untari Novia Wisesty, Irba Fairuz Thufailah, Ria May Dewi, Adiwijaya, and Jondri. Study of segmentation technique and stereology to detect peo follicles on usg images. Journal of Computer Science, 14(3): 351-359, Mar 2018. doi: w..8844jcssp.2211.35....3 2018. 2018.351..33 359.359
/tLhhttp::/tthecciiubb. httpab///o//bbbtttt//cssssss
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&33d medical data, 2023a. UKL/htttp::/arxiivoo/////b
02463
Chaoyi Wu, Xiaoman Zhang. Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data, November 2023b. URL. http: //arxiv.org/aatee wtt.o////////aaa//e////// 02463, arXiv:2308.0244 [cs].
Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, Wenhao Zheng, Zhaoyang Wang. Xiao Wang. Xuchao Zhang, Chetan Bansal, Marc Niethammer, Junzhou Huang, Hongtu Zhu, Yun Li, Jimeng Sun, Zongyuan Ge, Gang Li, James Zou, and Huaxiu Yao. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang
18

---

Published as a conference paper ICLR 2026
(eds.), Advances in Neural Information Processing Systems, volume 37, pp. 140334-140365. Curran Associates, Inc., 2024. URL httpps:/ppooceedingg wwwp:/cceeeiinnnnnnni.irp....... neurips. cc/paper wwt.ee//pppe///222///.. Td67f40f8ccd55550066
Xiaolong Xiao, Jianfeng Zhang, Yuan Shao, Jialong Liu, Kaibing Shi, Chunlei Hc, and Dexing Kong. Deep learning-based medical ultrasound image and video segmentation methods: Overview, frontiers, and challenges. Sensors, 28(8), 2025. ISSN 1424-8220. doi: 10.3390/s250823361
mdpi
2tmm11444--22/2558////
XiaohuMini. Xiaohumini. XiaohuMini News, January 2025. URL http::t//aaoommmiiiii site
Qiantong Xu, Fenglu Hong. Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv: 2305. /6504, 2023.
Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint 1..Kv::2006000444 2025.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint vrXiv:2407.10671 2024.
Kaining Ying. Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi, 2024.
Sheng Zhang. Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Matthew P. Lungren, Tristan Naumann, and Hoifung Poon. Large-scale domain-specific pretraining for biomedical vision-language processing. CoRR, abs/2303.00915, 2023.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https:
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang. Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li. Jiahao Wang, Nianchen Deng. Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Gc. Kai Chen, Limin Wang. Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL ptttps:///rxivu org abse/bbb/2244 10479.
19

---

Published as conference paper ICLR 2026
APPENDICES
Within this supplementary material, we elaborate on the following aspects:
Appendix A
More Related Work and Future Work
Appendix
Safeguarding
Appendix
More Evaluation Details
Appendix
Prompt Details
Appendix
Dataset Details and License
USAGE OF LLM
LLMs were used in this work as writing and editing assistant. Specifically, they helped polish the language of the manuscript for clarity and conciseness, suggested alternative phrasings, and formatted some LaTeX tables. LLMs were not used for research ideation or experimental design, but were used assist coding and prompt design.
20

---

Published as a conference paper ICLR 2026
A LIMITATIONS AND FUTURE WORK
A.1 MORE RELATED WORK
Ultrasound Foundation Models. Several ultrasound-specific models such as USFM (Jiao et al. 2024). UltrSAM (eyyer al. 2024). and EchoFM (Kim et al.] (2024) pretrain visual backbones using self- supervised or coaneentttion-orienttn objectives. BiomedCLIP (Zhang et al. 2023). Fetal-CLIP (Maani et al. 2025), and EchoCLIP (Christensen et al. 2024) explore vision-language pretraining in biomedical domains, but are often narrow in scope (e.g., fetal or cardiac imaging only), require fine-tuning, and lack the general-purpose zero-shot capabilities of LVLMs. Our benchmark evaluates generalist LVLMs directly, across a diverse range of ultrasound tasks, without task-specific adaptation. Table 4 gives detailed comparison of existing benchmarks and datasets.
Table 4: Comparison of U2- 02-BENCH with existing benchmarks and foundation model datasets.
Dataset/ Benchmark
#Tasks
#Anatomies
#Val US Cases
Multimodal
Free-text Output
Public Available
USFM (Jino al. 2014
3
22,421
UltraSam MMyeer et al.
2024
2
58 Fetal
14,000
0
FetalCLIP (Maan et 2025
4
0
EchoCIIP Chrsslessen
2024
5
Candiac
21,484
EchoFM (Kim el al 2024
4
Cardiac
0
1
GMAL-MNTBench Chenn 2024a
1,800
U2-BENCH (Ours)
7,241
1 UUtraSam's 's US-43d is composed of public datasets, but not released as a unified benchmark.
We only count the statistics of the ultrasound part of the GMAL-MMBench dataset for fair comparison.
A.2 LIMITATIONS
Ethical and Applicability Considerations. U2-BENCH is designed as a research-oriented benchmark and is not intended for clinical deployment or diagnostic decision-making. Any real-world application of models evaluated on this benchmark would require separate validation and regulatory approval. Although all data sources are licensed or publicly available and de-identified where applicable, we acknowledge that not all ethical and demographic dimensions of fairness can be fully accounted for at this stage.
Evaluation Scope. The benchmark focuses key task categories relevant ultrasound interpreta- tion-such as anatomical recognition, diagnostic classification, and structured report generation. While these tasks are representative and grounded in clinical utility, they do not exhaust the full landscape of sonographic applications. The evaluation metrics used (e.g., accuracy, BLEU) may not capture the full subtlety of expert clinical judgment, especially in edge cases.
Ultrasound-Speet Challenges. Ultrasound imaging is highly operator-dependent and subject to artifacts such as shadowing, speckle, and angle variation. Variability in scanning protocols and lack of standardized definitions (e.g., for "standard planes")) can complicate model training and evaluation. These challenges are inherent to ultrasound and reflect real-world complexities rather than flaws the benchmark design.
A.3 FUTURE WORK
Extending Dataset Diversity and Robustness. While U2-BENCH aggregates data from a broad range of sources, further expansion to include more institutions, device types, and global populations would
21

---

Published as conference paper ICLR 2026
improve its representativeness. Future iterations of the benchmark will explore domain adaptation, adversarial robustness, and performance under distribution shifts to better simulate deployment conditions in varied clinical environments.
Model Generalization and Multimodal Reasoning. Current LVLMs still struggle with fine-grained spatial tasks, consistency across subgroups, and robust generation of clinically meaningful language. In future work, we aim to incorporate richer contextual information (e.g.. patient history, multi-view inputs) to better assess models' multimodal integration capabilities and real-world reasoning performance.
Video-Based and Real-Time Evaluation. U2-BENCH currently operates frame-based inputs ensure comparability across models. However, clinical ultrasound interpretation often involves dynamic, probe- controlled acquisition. Extending the benchmark include video sequences, real-time tasks, and longitudinal case studies will be major step toward closing the simulation-to to-iiiiic gap.
Theoretical Foundations and Causality. Our current benchmark is designed for practical performance evaluation. Future work will incorporate diagnostic reasoning audits, causal probing methods, and uncertainty quantification frameworks to deepen our understanding of LVLM behavior in high-stakes medical applications.
Standardization in Ultrasound AI. There is growing need for community consensus annotation standards, task definitions, and evaluation protocols in ultrasound Al. We hope U2-BENCH can serve as a starting point for these conversations and will actively evolve in response to feedback from both clinical and technical communities.
B SAFEGUARDING
This study involves secondary use of de-identified, publicly available or licensed ultrasound datasets for the purpose of benchmarking machine learning models. All data used in U2-BENCH are either publicly released with appropriate usage permissions or obtained through official licensing agreements. No personally identifi able information is used, and all experiments are conducted in accordance with relevant data protection and ethical guidelines. Human annotators involved in quality assurance were trained to follow data confidentiality protocols, and no clinical decision-man makinn involved any stage of this work.
C MORE EVALUATION DETAILS
C.1 DATASETS USED
Table
C.2 EXPERIMENT SETTING
We conducted experiments on U2-BENCH with both open-source and closed-source LVLMs. Uniform prompts were applied across all models. The evaluation was executed on 32 NVIDIA A800 GPUs over a period of approximately two weeks, using the OpenCompass VLMEvalKit (Duan al. 2024), with additional support from unified framework (XiaohuMimi 2025), All models were tested with temperature 0.7.
Evaluated Models. We evaluated 23 LVLMs, spanning both open-source and closed-source systems, and including both general-purposs and medical-specialle variants.
22

---

Published as conference paper ICLR 2026
Table 5: Summary of annotated datasets used in U2-BENCH, grouped by core capability and task. The "Case Number" column indicates the number of samples per task, while "Total" reflects the overall count when available. More details about the datasets are included in Appendix
Cagubility
Berest Lesion Detection Ulraaound Videos
20221 Ureast Ultsonndd
Dotaset Dhabyani 2020
Dermuelozic Ulrasound
dussilloana
[021]
DD
1411
COVID Westeminn Bizs) Ulinoond Gaded Kouioaal Auesthesia
Novin 2023 KFINer (NeumeXTT U
1949, Ultraound Brasst Thyroid Dutset AUTTD Manona Juay Nito PCOS clasefication
GUPRISSSUE [Mo 20221 LiPSet 20236
Chasification
Breast Cancer [uura
Sonawate 3049
Ulmsound
2,999
PLANESS Da [Buges Artizuu et 2000 FPUS23 Prahukuran et 2005 CAMUS Lester at 2019 Knce ulmaoond
/2000
docarct to population based connt [Novi 2. (022)) "Ihrol (Krenae et 2044
ACOUSER [Sapai A01 INU IIM Lu
er al 2022 Carctid Anery Uleasnunt and Color Doppier (P2hm houchayy A0l9t Ano KCUS elassifcation Marrun 10200 Arican
heal Stansard Plane (Sendra libicells et [023); DDnIt (Pecsara at (015) CAMUS [Lecier (00)) COBS (Mahirper alal.
VRA
1,58
30ar1 COVID-BLUD BLUES WWiedmma et 2l 203 Danset of Bimas Jally liver ulrsooun imsger (Bira 2015 The Open Kility
Ultioound Daiset a
EXouo Darssond Prestuee Segmeraation Dutsee (Stao & Bicnnnn min All Breast Ulnwouud
Images Duiaszt (A] Dhibyan ao ci A0M0 Knee ulinsound catzeet a popilation based coheet Novva del 2024
Polycystic Ovary
Ultasound Image: Dutaset Nisesy (ta) a.a.1a 2018
Usassund
Seamentation NeumoXX palka glsbes] (Fawoeska
Brishnn
Images
Danset Daabyuk
5020
The Open Kalayy Claasound Danaee Sindla ni 2013 Edogeic Da Cureszio
N03 PALATEO LALLMUD
QD
1,918
CAMUS (Levian ct (019) HC18
Guided Reponna AWeethes Tyae Aud C-TRS Diise Lecaing i (.). 202
Heard of of 2i20 Tayord Ivoms p024 CCA CCA (0 3035) Ulasonn
ALOUSLIC Al Suppin Aile
PSEttS [al
1054] JNU-FF [Lact 3022
sundiisn & sepmentaiion
Valk (0330
KD
Unry Imaging Coliabunstive Shin-Shin 2035]
Regression
kay
(019/CUM Meihirger
images da 2018
2013
Sappla X01
521
RG
Chinese Ultrsound Repor Dosset Lietal 2004
Generation
CG
200
BPUS23
Prabulamn et 2023
2023
800
Qwen2.5-VI. Series (Yang al. 2024): ; This includes Qwen2.5-VL-3B-hnstree Qwen2.5-VL-7B-
Instruct, Qwen2.5-VL-322 -1-322---- Qwve2.5-VVL72BB-nnsrrr
Medical-Specific Open-Source Models: MiniGPT-Med [W al. 2023b). MedDr (He et
2024), Lingshu, MedGemmma44 (Anil et al. 2023).
Other Open-Source Models: Phi-4-Multinodal--iae -an.rudca--nttuu-5..0050 (Abdin et al.
2024). InternVL3.
9B-Instruct (Zhu al. 2025), VA-1.--A-15313 (Liu al. 2023a), Mistral-Small33144 1-.4:ll---..22--stms-m-- Instruct-
2503 (Jiang et al.
2023). DeepSeek-VL. VL.2 (DeepSeek-Al)
2024)
Closed-Source Models: GPii-o-Mini, GPPT-4--022-088
(OeenA 2023). GPT-5, Gemini-
1.5-Pro (exp-02-05), Gemini-2-Pry (exp-02-05). Gemini-2.5-PProPPeevvie (exp-02-05 05) (Anil et al.
2023), Claude-3-Sonner (20250219) (Anthropic] 2024), Qwen-Max-2025-00 22 (Bai al. 2023a)).
(ByteDance 2024), Doiphin-VI (Model developed by Dolphin
AI)
Random Guessing: implemented by uniformly sampling from the valid answer for each classification task.
C.3 JUSTIFICATION OF U2-SCORE WEIGHTING
Table 6: Task-specific evaluation metrics and weights. The corresponding weight wa and metric used for overall score aggregation for each task.
3
5
DD VRA
11
OD
KD
CVE
CG
0.07
di Avc. Azc. Acc. Aec.
Acc.
1 IRMSE
BLEU-4
BLEU-4
23

---

Published as conference paper ICLR 2026
The U2-Score summarizes model performance across the eight benchmark tasks in U2-BENCH through weighted aggregation:
U2-Score
de,
where
Wt
Σ3 nj
d. E [0. 1]
(2)
Each task é is associated with a weight Wi proportional to its number of annotated examples ny,, and a normalized evaluation score d, representing performance on that task. This formulation ensures that the final score reflects both task competence and dataset composition.
The weighting design of U2-Score rooted in data lata-driven representation of ultrasound practice. All bench- mark tasks are constructed from licensed and publicly available datasets sourced through comprehensive and systematic search. As result, the sample distribution across tasks captures the structure of real-world ultrasound data availability and usage. Tasks with greater sample counts typically correspond to more standardized, clinically widespread, and institutionally supported workflows- as disease diagnosis and view recognition. Conversely, tasks with fewer examples often reflect more specialized applications or less frequently annotated modalities.
By aligning task weights with dataset size, the U2-Score encodes the practical significance and maturity of each task in the public ultrasound domain. This approach also promotes reliable aggregate evaluation, scores more robustly informed by tasks with greater data coverage. The resulting metric balances comprehensiveness with grounded applicability, supporting fair and reproducible model comparison the current landscape of ultrasound Al research.
24

---

Published as conference paper ICLR 2026
C.4
ERROR ANALYSIS
Diagnosis Task (DD)
Accuracy: 0.05 (1/20 modelss correct)
Reference Answer: 4A
Figure 4: Ultrasound image for Diagnosis Task 40: casc001273
Model Responses:
You are a radiologist analyzing breast ultrasound image. Your task synthesize the sonographic characteristies of any identified lesions (or lack thereof) into final ACR BI-RADS (Breast Imaging Reporting and Data System) ussessment
Prompt:
Do-3B: 2
gpt-to-2024-08-06:
Insufficient information provided to determine BI RADS category. Please describe sonographic characteristies the lesion the image.
category.
BI-RADS Ultrasound Assessment Category
Definitions
'2' (Benign): Findings definitively benign (e-g-- simple cysts, intramammary lymph nodes, stable surgical implants/changes) 0% likelihood of malignancy. Requires routine screening follow-up "3" (Probubly Benign): Findings have characteristic benign features but are not definitively henign (e.g., presumed fibroadenoma, complicated eyst). Very low likelihood malignancy (12%). Short-interval (e.g.,
MedDr: 5
qwen-max-2022 2000-max2002500-255:3
2rcallm
Baaude337-sommee2222520119 4C
gemini-220-prro-xxp-022.55 4C
deepseek-v12: 4B
qwenn25.v1.3---nstrrudt:
6-month) follow-up typically recommended.
Qxean..-v11-7B-Instrutt 4A
'4A' (Low Suspicion for Malignancy): Findings warrant biupsy but have low probubility of
gemini-1.5-pro--atett:2 2
Dwee22.5VIL.3BB-Inntte 4B
malignancy (:2% to < 10%%).
'B'' (Moderate Suspicion for Malignancy): Findings
4B
warrant biopsy an intermediate probability of
Phi-4-muttimddl--nnsttr cannot synthesize sonographic characteristics the lesions make BI-RADS assessment without the actual ultrasound images detailed description of the findings. If you provide the relevant details or images, would be able to assist you with the assessment.
malignancy 6,10% to 550%).
'4C" (High Suspicion for Malignancy): Findings
warrant biopsy with high probability of malignancy 6,50% to 195%), without the classic features of Category 5.
'5" (Highly Suggestive of Malignancy):: Findings have classic malignant features (c.g. irregular spiculated mass). Very high probability malignancy (295%). Riopsy required, and definitive action should be laken regardless of pathology results if discordant.
LLaVA-1.5-133-HH '2'
3 Do-7B: The anatomical component most vividly portrayed is the breast
Choose the single most appropriate BI-RADS assessment category from the options below. options: ['2', '3', '4A', '4B', '4C", '5'] Output format: only exact option from the list above. Do include any introductory phrases, explanations, numbering, formatting.
gpt-4o-mini:
chosen
25

---

Published as conference paper ICLR 2026
View Recognition and Assessment Tasks (VRA)
Reference Answer: hdvb
Prompt: You radiologist analyzing fetal ultrasound image. Your task is to determine the fetal presentation and orientation based on the provided ultrasound image. Specifically, identify whether the fetal head is down(hd) or up(hu). Additionally, determine if the fetal back is primarily oriented towards the ultrasound probe (vb) towards the ultrasound probe (vf). Choose single best option from the options below that accurately combines these findings. options: 'hdvb', 'hdvf', 'huvb', 'huvf" Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering. or formatting.
Model Responses:
Random Guessing: huvf
MiniGPT-Med: hdvf
MedDr: hdvb
Qwenn225- hdvf
Qren-22--V--7--Instuuct hdvb
Qwenn225-VL-22---nsttuuttt hdvb
Qween2.5-V--222--nnsturee hdvb
DeepScck- hdvf
IntermV..39IB-Instrrett hdvf
LLaVA-1.5-13B: huvb
Phi44-Mulimoola--nssruutt hdvf
Mistral-imaall3312244--nnn: hdvb
Doubaa-1.5-VVision-Poo--3 hdvb
GPT-4o-Mini: hdvf
GPT-44: hdvb
Gemini-1.5-Pro: hdvf
Gemmni--20-Pro-EExp hdvb
Cemini-2255-Pro-Previee:: hdvf
Claude-3.7-Somnet: huvb
Qwen-Max: hdvb
Dolphin-V1: hdvb
26

---

Published as conference paper ICLR 2026
Lesion Localization Tasks (LL.)
120
Reference Answer: upper left
Prompt: You are a radiologist analyzing an ultrasound image of thyroid. Your task is to identify the primary location of any visible lesion(s) relative the boundaries of the displayed image. Consider the lesion's 's center location most prominent
Model Responses:
Random Guessing: lower right
MiniGPT-Med: upper center
MedDr:
upper left
when deciding. Choose the single option from the list below that best describes this location, even if the fit is approximate. options: upper left, upper center, upper right, middle left, center, middle right, lower left, lower center, lower right, not visible Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering. formatting.
Qwven-2--VL-3B-Insstrue middle left
Qwven-22-VV-77B-nnntrrut upper left
Qwwn=225-VU-332--Instrrc upper left
Qwwen225-VL-722--Insstuu upper left
DeepSeck- VL2: upper right
InternVL.3-9B-Insttrect upper center
LLaVA-1.5-13B: middle left
Phi44-Mulimolla-Instrtttt upper center
Misttrl-Smmall-331-2BB--nnns upper left
Doubaa--.5-VViioon-Pro--... upper left
GPT-4o-Mini: upper right
GPT-44: upper left
Gemini-1,5-Pro: upper right
Gemini-2.0-Pro-Expp upper left
Gemiin225-Pro-Previeew upper right
Claude-3.7-Soneet: upper center
Qwe--Max: middle left
Dolphin-Vi upper left
27

---

Published as conference paper ICLR 2026
Organ Detection Tasks (OD)
200033
PoO 4037
RLL SAG
Reference Answer: center
Prompt: You are radiologist analyzing an ultrasound image of liver. Your task is to identify the primary location of the target organ relative to the boundaries of the displayed image. Consider the organ's center location or most prominent area when deciding. Choose the single option from the list below that best describes this location, even if the fit is approximate. options: upper left, upper center, upper right, middle left, center, middle right, lower left, lower center, lower right, not visible Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Model Responses:
Random Guessing: lower left
MiniGPT-Med: middle right
MedDr: center
Qwen-22--VL-3B-nnsttuee middle right
Qween225-VL-7BB-nsstrut:: center
Qwee:22.--1-:32--nnsrru::: center
Qwen=225-VI-722--nsttree center
DeepSeek-VI.2. VL.:: middle left
InteemtV.3-9B-Insstuutt lower center
LLa LLaVA-115-13B: middle right
Phi-4-Multioodal-Intteeee middle right
Nistral-Small:33122BB-nnt center
Doubaa-1.5-VViiion-Pro-33 center
GPT-4o-Mini: lower center
GPTT40: center
Gemini-1.5-Pro, lower center
Gemini-2.0-Pro-Exp: center
Gemini-225-ProoPPreview: lower center
Claude-3.7-Sonnet: center
Max: middle right
Dolphin-Vil center
28

---

Published as conference paper ICLR 2026
Keypoint Detection Tasks (KD)
Reference Answer: middle right
Prompt: You are radiologist analyzing an ultrasound image of heart. Your task is to identify the primary location of the key anatomical landmark point relative to the boundaries of the displayed image. Consider the landmark's precise position when deciding. Choose the single option from the list below that best describes this location, even if the fit is approximate. options: upper left, upper center, upper right, middle left, center, middle right, lower left, lower center, lower right, not visible Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Model Responses:
Random Guessing: upper center
MiniGPT-Med: middle left
MedDr:
middle right
Qwen-225-VV-3BB-nsttuuee center
Qwven225-VL-77--nsstree middle right
Qwwnn2225-V-3332--nntrrrtt middle right
Qwven2255-V--22--Instruct middle right
DeepSeek-VIA VL22 center
IntermVL.399-Innttruet middle left
LLa center
Phi44-Mutimoola-Instrcttt lower right
Mistral-Small-331-244 middle right
Doubaa-1.5-VViion-Pro-33 middle right
GPT--4-Mini: center
GPT--o: middle right
Gemini-1.5-Pro: center
Gemini-2.0-Pro-Exp: middle right
Geminn-2.5-Proo-Peevieer center
Claude-3.7-Sonnet: middle right
Qwe--Max: center
Dolphin-Vil middle right
29

---

Published as conference paper ICLR 2026
Cardiac View Evaluation Tasks (CVE)
Reference Answer: 2CH
Prompt: You are a radiologist or cardiologist specializing echocardiography, analyzing an apical view ultrasound image of the human heart. Your task is to accurately identify the specific apical view presented in the provided echocardiogram image. Carefully examine the cardiac structures visible. Determine if the image displays primarily the left ventricle and left atrium only (indicative of 2-Chamber view, 2CH), or if it clearly shows four chambers: the left ventricle, right ventricle, left atrium, and right atrium (indicative of a 4-Chamber view, 4CH). Choose the single best option from the list below that correctly identifies the view. options: 2CH, 4CH
Model Responses:
Random Guessing: 4CH
MiniGPT-Med: 4CH
MedDr: 2CH
Qween-25-VV-33--Insttuee 4CH
Qween225-VL-7B--nsttuttt 4CH
Qween225-VT--32--Imsttue 4CH
Qween225-VVL-22B--nstttttt 4CH
DeepSeek-VL2: 4CH
IntemmVL3-9B-Instruuet 4CH
LLaVA-1.5-1BB:4CH
Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Phi44-Waltinodd--Inttrttt 4CH
Mistral-Smaall3311244B-nnnt 4CH
Doubao-15--ision--rro33::: 2CH
GPTT44-Mini: 4CH
GPT-44: 4CH
Gemini-1,5-Pro: 4CH
Gemini-2.0-Pro--Exp: 4CH
Gemini-225-Pro-Preveeew 4CH
Claude-3.7-Sonnet: 2CH
Qwen-Max: 4CH
Dolphin-V1: 2CH
30

---

Published as conference paper ICLR 2026
Cardiac Grading Tasks (CG)
M0
TOOMS
LCC
11c
Reference Answer: Moderate OA
Prompt: You are a radiologist analyzing an ultrasound image of left/right knee. Your task is to assess the severity of osteoarthritis (OA) using the established Kellgren- (KL) grading system.
Model Responses:
Random Guessing:
MiniGPT-Med: Questionable OA
MedDr: Moderate OA
Kellgren-Lawrene (KL) Grade Mapping
Options:
Qwenn225-VV-33--nnttuue No OA
'No OA": Corresponds to KI. Grade (No
Qwenn22--VL-7B-nnstrrce
radiographic features of OA).
Questionable OA
'Questionable OA': Corresponds to KL Grade (Doubtful JSN and possible minute osteophytes).
Qwven22.--1-332--nnstuuttt Mild OA
'Mild OA': Corresponds to KL Grade 2
Qwenn225-V1-722--nnsrree Mild OA
(Definite osteophytes and possible JSN).
DeepScek-VL.2: Questionable OA
"Moderate OA': Corresponds to KL Grade 3
(Moderate multiple osteophytes, definite JSN,
InternWI.3-9B-Instruct No OA
some sclerosis, possible deformity).
LLaVA-1.5-1BB: No OA
'eeeere OA': Corresponds KL Grade
Phi44-Multimodall-nttuue
(Large osteophytes, marked JSN, severe sclerosis, definite deformity).
Questionable OA
"Total joint replacement': Indicates the presence knee arthroplasty components (prosthesis), which replaces the native joint structures evaluated by the KL scale. options: 'Mild OA', 'Moderate OA', 'No OA', 'Questionable OA', 'Severe OA', "Total joint replacement'
Mistral-Smalll-31-24B--nnns Mild OA
Doubao-l.--issonnPPP-33.
Moderate OA
GPT-44-Mini: No OA
GPTT-4o: No OA
Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Gemini-1,5-Pro: Questionable OA
Gemini-22.0PPo-EExp Mild OA
Gemini-2.5-Pro-Preveeew:
Questionable OA
Claude-3.7-Sonnet: Mild OA
Qwe- Mild OA
Dolphin-Vis Moderate OA
31

---

Published as conference paper ICLR 2026
Report Generation Tasks (RG) Input
Prompt: You a radiologist analyzing an ultrasound image focused on the Liver. Your task is generate concise and informative radiological report based strictly on the visual findings within the provided image. Your report should describe the primary organ's appearance (size, shape, borders/capsule). its parenchymal echotexture (e.g., homogeneous, heterogeneous, echogenicity relative to reference structures), and identify any visible abnormalities (e.g., masses, cysts, fluid collections, calcifications, ductal dilation). Comment relevant adjacent structures if visualized. Use standard radiological terminology. Output format:
Strings, that is your report.
Example: The liver morphology full with a smooth capsule. The parenchymal echotexture is fine and diffusely increased. Visualization of the portal venous system suboptimal. Intrahepatic and extrahepatic bile ducts are not dilated. The main portal vein diameter is within normal limits. The gallbladder is normal in size and shape. The wall smooth and not thickened. No obvious abnormal echoes are seen within the lumen. The pancreas is normal in size and shape with homogeneous parenchymal echotexture. The panereatic duct seen within the pancreas. The spleen is normal in size and shape with homogeneous parenchymal echotexture. No obvious space-occupying lesion seen within the spleen.
dilated. No definite space-occupying lesion is
32

---

Published as conference paper ICLR 2026
Report Generation Tasks (RG) Output
Reference Answer:
The liver morphology is full with smooth capsule. The parenchymal
echotexture is fine and diffusely increased. Visualization of the portal venous system is suboptimal. Intrahepatic and extrahepatic bile ducts are not dilated. The main portal vein diameter within normal limits. No focal lesions are identified.
Model Responses:
Random Guessing:
MiniGPT-Med: The liver shows normal size and contour with homogeneous echotexture.
No focal lesions biliary dilatation noted. Portal structures appear normal.
MedDr:
The liver demonstrates normal size and contour with a smooth capsule. The parenchymal echotexture is homogeneous with normal echogenicity. No focal lesions identified. Intrahepatic biliary ducts are not dilated. Portal venous structures appear normal.
Qwen-2.5- -----V-------rreeee The liver appears normal in size with smooth contour. The parenchymal echotexture is homogeneous without focal lesions. No intrahepatic biliary dilatation is seen.
Qwen-2.5-VI.-4e -1---VVL--B--nstrruce The liver appears normal size with a smooth capsule. The parenchymal echotexture is homogeneous with focal lesions. No intrahepatic extrahepatic biliary dilatation is seen. The portal venous structures appear normal.
Qwen-2.5- 2--25-5-LLL222-nssrrre The liver appears normal in size with a smooth contour. The parenchymal echotexture is homogeneous with normal echogenicity. No focal lesions such as masses, cysts, or calcifications are identified. The intrahepatic bile ducts are not dilated. The portal venous structures appear normal. No perihepatic fluid collection is observed.
The liver demonstrates normal size and contour with smooth capsule. The parenchymal echotexture is homogeneous with normal echogenicity. No focal lesions masses identified. The intrahepatic biliary tree is not dilated. Portal venous structures appear normal. No perihepatic fluid collection seen.
Dween2.5-VL-722-Inssiee
DeepSeek-VL.2 The liver appears normal in size with a smooth contour. The parenchymal echotexture is homogeneous with normal echogenicity. No focal lesions are identified. The intrahepatic bile ducts are not dilated. The portal venous structures appear normal. No perihepatic fluid collection is observed.
The liver appears normal in size with smooth contour. The parenchymal echotexture is homogeneous without focal lesions. No intrahepatic or extrahepatic biliary dilatation is seen. The portal venous structures appear normal.
Intern
LLa VLaVA-115-13B: The liver appears normal in size with smooth contour. The parenchymal echotexture homogeneous without focal lesions. No intrahepatic biliary dilatation is seen. The portal structures appear normal.
The liver appears normal in size with a smooth contour. The parenchymal echotexture is homogeneous with focal lesions. No intrahepatic or extrahepatic biliary dilatation is seen. The hepatic vessels appear normal.
Thi-4-MMltimodal :nstruuuttt---ttteet::::
The liver demonstrates normal size and contour with a smooth capsule. The parenchymal echotexture homogeneous with normal echogenicity. No focal lesions or masses are identified. The intrahepatic biliary tree is not dilated. Portal venous structures appear normal.
Mistral-Small-331-244----
33

---

Published as conference paper ICLR 2026
The liver appears normal in size with smooth contour. The parenchymal echotexture is homogeneous with normal echogenicity. No focal lesions are identified. The intrahepatic bile ducts are not dilated. The portal venous structures appear normal.
Daubao-..5-iiiio-Prro--
The liver appears normal in size with smooth contour. The parenchymal echotexture homogeneous without focal lesions. No intrahepatic or extrahepatic biliary dilatation is seen. The hepatic vessels appear normal. No ascites or other abnormalities are identified in the visualized portion of the liver.
GPT-4o-Minii
The liver appears normal in size and shape with a smooth capsule. The parenchymal echotexture is homogeneous with no obvious focal lesions. No intrahepatic extrahepatic bile duct dilation is noted. The gallbladder is not clearly visualized in this image. No ascites or other fluid collections are present. Adjacent structures are not well visualized for further evaluation.
GPT-40:
Gemini-1.5-Pro: The liver appears normal size with smooth contour. The parenchymal echotexture is homogeneous with normal echogenicity, No focal lesions such as masses, cysts, or calcifications are identified. The intrahepatic bile ducts are dilated. The portal venous structures appear normal.
Gemini-2.0-Pro-Exxp The liver appears normal in size with a smooth capsule. The parenchymal echotexture is homogeneous with normal echogenicity, No focal lesions identified. The intrahepatic bile ducts are not dilated. The portal venous structures appear normal where visualized.
Gemini-225-PPoo-Peeiees The liver appears normal in size with smooth capsule. The parenchymal echotexture is homogeneous with normal echogenicity. No focal lesions such as masses, cysts, calcifications
identified. The intrahepatic bile ducts not dilated. The portal venous structures appear normal where visualized. No perihepatic fluid collection is seen. The gallbladder and other adjacent structures are not well visualized in this limited image. Overall impression: Normal liver parenchyma without evidence of focal lesions or biliary dilation.
The liver demonstrates normal size and contour with smooth capsule. The parenchymal echotexture appears homogeneous with mildly increased echogenicity, suggesting possible early fatty infiltration. No focal lesions, masses, or cysts are identified. The intrahepatic biliary tree is not dilated. Portal venous structures appear normal where visualized. No perihepatic fluid collection seen. Adjacent structures are not well visualized in this limited view.
Qwen-Max: The liver appears normal in size with a smooth contour. The parenchymal echotexture homogeneous with normal echogenicity. No focal lesions such as masses, cysts, or calcifications are identified. The intrahepatic bile ducts are not dilated. The portal venous structures appear normal. No perihepatic fluid collection is observed. The visualized portions of adjacent structures appear unremarkable. Overall, the liver appears sonographically normal based on this limited view.
Dolphin-V V1: The liver morphology full with a smooth capsule. The parenchymal echotexture is fine and diffusely increased. Visualization of the portal venous system is suboptimal. Intrahepatic and extrahepatic bile duets are not dilated. The main portal vein diameter is within normal limits. No focal lesions are identified.
34

---

Published as a conference paper ICLR 2026
To provide deeper insights into model performance on medical imaging tasks, we conduct a comprehensive error analysis of models across four critical ultrasound image analysis tasks: measurement, classification, segmentation, and report generation. This analysis reveals distinct error patterns and speccifc challenges that inform future model improvements.
Numerical Regression Task Analysis Among 101 total responses, the most significant challenge is the prevalence of descriptive responses instead of numerical values (53.47%). The model frequently generates interpretative text such as "The principal anatomical element visualized here is unequivocally the fetus head" rather than the expected numerical measurement (e.g.. 291.4mm) This pattern suggests fundamental misunderstanding of task requirements, where the model interprets the task as image content identification rather than quantitative measurement.
Format violations constitute 1.98% of responses, where models provide numerical values with units (e.g., "113.6 mm") despite explicit formatting constraints. Notably, 43.56% of responses follow the correct numerical format, though accuracy assessment requires comparison with ground truth values. The high rate descriptive responses indicates that current vision-language models struggle with the transition from visual analysis to precise quantitative output.
Classification Task Performance Classification tasks demonstrate superior format compliance compared to measurement tasks, with 75.66% of responses providing valid option selections from 152 total responses. However, two distinct error patterns emerge: explanatory responses (5.92%) where models provide justifica- tions rather than selections (e.g.. "There no definitive view of the fetal abdomen or pelvis to determine fetal position"), and format violations (18.42%) containing additional descriptive content alongside valid options.
The tendency toward explanatory responses reveals an interesting model behavior where excessive caution leads to task avoidance rather than best-effort selection from available options. This suggests that models may benefit from more explicit instructions emphasizing the requirement for definitive option selection even under uncertainty.
Segmentation and Localization Analysis Segmentation tasks, requiring spatial reasoning for anatomical structure localization, show moderate success with 66% valid position responses from 500 total responses. The primary error categories include invalid position terminology (27.80%) with responses like "Not visible." or "Upper right." " that contain punctuation or non-standard terms, and complete task deviation (6.20%) where models provide structural descriptions instead of positional information.
Case Study Examples: Analysis of specific segmentation cases reveals distinct model behaviors. In thyroid lesion localization tasks, while Gemini-2.5-Pro and GPT-4o consistently provide concise responses ("center"), Claude-3.7 Ulaude-3.7-Sonnel 7-Sonnet exhibits significant format violations. For instance, when tasked with identifying tumor location breast ultrasound images, Claude generated extensive explanatory text:
"This image appears be an ultrasound showing tissue layers with varying echogenicity.. cannot identify clear, definitive lesion... For proper medical diagnosis, this ultrasound should be evaluated by a qualified radiologist..."
Such responses, while demonstrating medical awareness, completely violate the specified output format requiring only location terms. This pattern suggests that Claude prioritizes safety disclaimers over task compliance in medical contexts.
Additionally, a concerning pattern emerges where multiple models consistently respond "center" regardless of actual lesion position, evidenced by reference bounding boxes indicating lesions at coordinates [0.6. 0.247] and [0.595, 0.308). This suggests potential spatial reasoning limitations or default response bias that could compromise clinical utility.
35

---

Published as conference paper ICLR 2026
The relatively high success rate in spatial localization compared to numerical measurement suggests that discrete spatial reasoning may be more accessible to current vision-languag architectures than continuous numerical estimation.
Report Generation Excellence Report generation tasks achieve the highest success rate (98%) among all evaluated tasks, with only 2% exhibiting structural misidentification and 1% showing false findings. The rare but critical errors include anatomical misidentification ("Top view of fetus head and thorax" for fetal head ultrasound) and false pathological findings ("Aneuploid fetus with abnormal facial features"). While infrequent, such errors carry significant clinical implications, potentially leading to unnecessary medical interventions or patient anxiety.
Cross-Tast Error Pattern Analysis Task difficulty ranking from most to least challenging reveals: mea- surement (43.56% success) > segmentation (66% success) > classification (75.66% success) > report generation (98% success). This hierarchy reflects the increasing complexity of transitioning from free-form text generation to structured, constrained outputs requiring precise adherence format specifications.
Common error patterns across tasks include: (1) descriptive language substitution, most prominent measurement tasks where models default to interpretative text rather than required numerical values; (2) format non-compliance, prevalent across classification and segmentation tasks despite clear formatting instructions; and (3) task misunderstanding, where models completely misinterpret task objectives, such as treating localization as structure identification.
Implications for Medical AI Development These findings highlight critical considerations for deploying vision-lan language models in medical imaging applications. The inverse relationship between task constraint and model performance suggests that current architectures excel unconstrained text generation but struggle with precise, structured outputs essential for clinical decision-making. Future developments should prioritize: (1) enhanced instruction following capabilities for constrained output generation, (2) domain-specific fine-tuning medical imaging tasks emphasizing numerical precision, and (3) robust validation mechanisms to detect and prevent false findings in clinical applications.
The analysis underscores that while large vision-languag models show promise for medical imaging applica- tions, careful task-specific optimization and human oversight remain essential, particularly for quantitative measurements and diagnostic assessments where precision directly impacts patient care.
36

---

Published as conference paper ICLR 2026
D PROMPT FOR TASKS
Prompt Template used for fetal view classification (dataset 10)
You are radiologist analyzing fetal ultrasound image.
Your task is to determine the fetal presentation and orientation based on the provided ultrasound image. Specifically, identify whether the fetal head is down(hd) or up(hu). Additionally, determine if the fetal back is primarily oriented towards the ultrasound probe (vb) or towards the ultrasound probe (vf). Choose the single best option from the options below that accurately combines these findings.
options: 'hdvb', 'hdvf", 'huvb', 'huvf'
Output format: only the exact text the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Prompt Template used for heart view classification (dataset 18)
You are radiologist or cardiologist specializing in echocardiography, analyzing an apical view ultrasound image of the human heart.
Your task is to accurately identify the specific apical view presented in the provided echocardiogram image. Carefully examine the cardiac structures visible. Determine if the image displays primarily the left ventricle and left atrium only (indicative of 2-Chamber view, 2CH), or it clearly shows all four chambers: the left ventricle, right ventricle, left atrium, and right atrium (indicative of 4-Chamber view, 4CH). Choose the single best option from the list below that correctly identifies the view.
options: 2CH, 4CH
Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
37

---

Published as conference paper ICLR 2026
Prompt Template used for (KL) grading (dataset 28)
You are radiologist analyzing an ultrasound image of left/right knee.
Your task is to assess the severity of osteoarthritis (OA) using the established Kellgren-LLarrence
(KL) grading system. Kellgren-Lawrent (KL) Grade Mapping to Options:
No OA': Corresponds to KL Grade 0 (No radiographic features of OA).
"Questionable OA': Corresponds to KL Grade (Doubtful JSN and possible minute osteo- phytes).
Mild OA': Corresponds to KL Grade (Definite osteophytes and possible JSN).
Moderate OA': Corresponds to KL Grade 3 (Moderate multiple osteophytes, definite JSN, some sclerosis, possible deformity).
Severe OA': Corresponds to KL Grade (Large osteophytes, marked JSN, severe sclerosis, definite deformity).
'Total joint replacement': Indicates the presence of knee arthroplasty components (prosthe- sis), which replaces the native joint structures evaluated by the KL scale.
Choose the single best option from the following list that accurately describes the image.
options: 'Mild OA', 'Moderate OA', 'No OA', 'Questionable OA', 'Severe OA', 'Total joint replace- ment"
Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
38

---

Published as conference paper ICLR 2026
Prompt Template used for BI-RADS classification (dataset 40)
You are radiologist analyzing a breast ultrasound image.
Your task is to synthesize the sonographic characteristics of any identified lesions (or lack thereof) into final ACR BI-RADS (Breast Imaging Reporting and Data System) assessment category.
BI-RADS Ultrasound Assessment Category Definitions:
'2' (Benign): Findings are definitively benign (e.g., simple cysts, intramammary lymph nodes, stable surgical implants/changes) 0% likelihood of malignancy. Requires routine screening follow-up.
3' (Probably Benign): Findings have characteristic benign features but not definitively benign (e.g.. presumed fibroadenoma, complicated cyst). Very low likelihood of malignancy (12%). Short-interval (e.g., 6-month) follow-up is typically recommended.
"4A' (Low Suspicion for Malignancy): Findings warrant biopsy but have low probability of malignancy (:2% 210%).
4B'' (Moderate Suspicion for Malignancy): Findings warrant biopsy with an intermediate probability of malignancy (,10% to 250%).
"4C (High Suspicion for Malignancy): Findings warrant biopsy with a high probability of malignancy (250% to 195%), without the classic features of Category 5.
'5' (Highly Suggestive of Malignancy)): Findings have classic malignant features (e.g., irregular spiculated mass). Very high probability of malignancy (295%). Biopsy is required, and definitive action should be taken regardless of pathology results if discordant.
Choose the single most appropriate BI-RADS assessment category from the options below.
options: ['2', '3', '4A' '4B', '4C', '5']
Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Prompt Template used for fetal abdomen (dataset 50)
You are radiologist analyzing ultrasound image of fetal abdomen.
Your task to determine if the presented cross-sectional view of the fetal abdomen is technically adequate for performing an accurate Abdominal Cireumference (AC) measurement according to standard obstetric guidelines. Identify the specific anatomical plane shown for the fetal abdomen. Determine if this plane meets the criteria for an optimal AC measurement (correct landmarks visible, proper transverse orientation) or if suboptimal (incorrect plane, missing landmarks, oblique/- foreshortened view, presence of interfering structures like kidneys). Choose the single best option describing the plane's suitability for AC measurement.
options: "none", 'optimal', 'suboptimal"
Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
39

---

Published as conference paper ICLR 2026
Prompt Template used for breast classification
You are radiologist analyzing a breast ultrasound image.
Your task is carefully examine the provided breast ultrasound image, evaluate any identified lesions or abnormalities based on key sonographic characteristics (including shape, orientation, margin, echo pattern, posterior acoustic features, and associated features), synthesize these features to form an overall impression about the likelihood of malignancy, and then choose the single best option from the following list that accurately summarizes this assessment.
options: (normal), benign, malignant
Output format: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Prompt Template used for thyroid classification
You are radiologist specializing in head and neck endocrine imaging, analyzing ultrasound image of the thyroid gland.
Your task is to carefully examine the provided thyroid ultrasound image, evaluate the overall thyroid gland parenchyma (echogenicity, texture, vascularity), identify any focal nodules, assess the specific sonographic features of any nodules found (including composition, echogenicity, shape, margin, gland appears normal, contains
and echogenic foci), synthesize these findings to determine benign-appearing findings, or contains findings suspicious for malignancy, and then choose the single best option from the following list that accurately summarizes this assessment.
options: (normal thyroid), benign, malignant
Output format:
only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Prompt Template used for skin cancer classification (dataset 25)
You are radiologist analyzing an ultrasound image of skin.
Your task is carefully examine the provided skin ultrasound image, evaluate the identified lesion abnormality based on key sonographic characteristics (including its location within skin layers, echogenicity, internal echo texture, shape, margins, size/depth, posterior acoustic phenomena, and vascularity assessed with Doppler), synthesize these features to form an overall impression regarding the likelihood of malignancy, and then choose the single best option from the following list that summarizes this assessment.
options: benign, malignant
Output format:
only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
40

---

Published as conference paper ICLR 2026
Prompt Template used for pancreas cancer classification (dataset 42)
You are radiologist analyzing an ultrasound image of the pancreas.
Your task is to carefully examine the provided ultrasound image of the pancreas, evaluate the gland's echotexture, size, margins, and the pancreatic duct diameter, identify any focal lesions or masses (noting their echogenicity, margins, size, and vascularity if Doppler is available), assess for associated findings such as ductal dilation (including potential "double duct" sign), vascular involvement (encasemen//hrombosii regional lymphadenopathy. or fluid collections, synthesize these findings to determine there evidence suspicious for primary pancreatic cancer versus other findings, and then choose the single best option from the following list that summarizes this assessment.
options: non- con-pancreas cancer, pancreas cancer
Output format:
only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Prompt Template used for PCOS classification (dataset 74)
You are a radiologist analyzing an ultrasound image obtained during a pelvic examination, potentially part of evaluation for Polycystic Ovary Syndrome (PCOS).
Your task is to evaluate the overall appearance of the anatomical structures presented in the ultrasound image (primarily focusing the ovaries and potentially the uterus). Consider sonographic features such as ovarian size, morphology, follicle count and distribution, stromal echogenicity, as well as any other findings that might indicate pathology. Based on this assessment, determine if the image appears generally normal or if it displays features suggestive an abnormality (which could include findings consistent with PCOS other conditions). Choose the single best option from the following list that accurately describes this overall impression.
options: AAppear abnormal', AAppears normal'
Output format:
only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
41

---

Published as conference paper ICLR 2026
Prompt Template used for PCOS classification (dataset 74)
You are radiologist analyzing an ultrasound image obtained during a pelvic examination. Crucially, assume this specific image has already been determined to show some form of abnormality. Your focus now is on the nature of that abnormality.
Your task is to specifically assess whether the abnormality present in this ultrasound image includes clear sonographic evidence consistent with a polycystic ovary. Evaluate the visualized ovarian structures, paying close attention to features commonly associated with PCOS, such as: increased number of follicles, peripheral distribution of follicles, increased ovarian volume, increased stromal echogenicity volume. Based whether these specific PCOS-related sonographic features are identifiable within the overall abnormal appearance, specifies whether the ultrasound image shows evidence/ visibility of polycystic ovary not. Choose the single best option from the following list.
options: 'Not-visible' 'Vsiibee
only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Output format:
Prompt Template used for PCOS classification (dataset 75)
You are a radiologist analyzing an ultrasound image obtained during pelvic examination, specifically being evaluated for features potentially related to Polycystic Ovary Syndrome (PCOS).
Your task is carefully evaluate the provided ultrasound image for sonographic features consistent with Polycystic Ovarian Morphology (PCOM), which is the ultrasound component relevant to PCOS detection. Analyze the visualized ovary (or ovaries), considering criteria such as increased ovarian volume, increased antral follicle count (e.g., 2 20 per ovary), peripheral follicle distribution, and / or increased stromal echogenicity volume. If sonographic features consistent with PCOM are present, select the label 'infected', otherwise 'noninfected", Choose the single best option from the following list.
options: 'infected', 'noninfected'
Output format:
only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
42

---

Published as conference paper ICLR 2026
Prompt Template used for lung parenchyma (dataset 44)
You are a radiologist or clinician skilled in performing and interpreting Lung Ultrasound (LUS), specifically analyzing ultrasound image of the lung pleura and parenchyma.
Your task carefully examine the provided lung ultrasound image, focusing on the appearance of the pleural line and the underlying lung parenchyma, identify the presence and characteristics of A-lines, B-lines (number, coalescence), and any consolidations according the defined severity scoring criteria below, and then choose the single best integer score (0, 1, 2, or 3) from the following list that accurately reflects the observed findings.
LUS Severity Score Criteria:
O: Normal lung pattern. Characterized by continuous, regular, thin pleural line with horizontal reverberation artifacts (A-lines) below it. Sliding lung sign is typically present.
I: Mild interstitial syndrome. Characterized by an indented or slightly irregular pleural line. Scattered, well-defined vertical artifacts (B-lines) visible (typically 23 B-lines per intercostal space but not coalescent).
2: Moderate interstitial syndrome or early consolidation. Characterized by a broken or significantly irregular pleural line. Multiple coalescent B-lines (small "white lung" areas) small subpleural consolidations are present.
3: Severe interstitial syndrome or large consolidation. Characterized by dense and largely extended confluent B-lines ("white lung" appearance occupying most or all of the screen) with or without large consolidations.
Options: 0, 1, 3
only the single chosen integer number from the list above. Do not include any introductory phrases, explanations, numbering, formatting.
Output format:
Prompt Template used for fatty liver classification (dataset 57)
You are radiologist analyzing static B-mode ultrasound image displaying the liver:
Your task is to evaluate the liver parenchyma in the provided image to determine the grade of hepatic steatosis. For this task, label is assigned if the image displays features consistent with fatty liver (which often correlates histologically with 75% hepatocyte steatosis), while label is assigned if such features are absent. Based on your comprehensive assessment of these sonographic features, determine whether the image displays sufficient evidence to be classified as showing fatty liver (Label 1) or not (Label 0). Choose the single best option from the following list that accurately reflects your classification.
options: 0,
Output format: only the single chosen integer number from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
43

---

Published as conference paper ICLR 2026
Prompt Template used for fetal (dataset 03)
You are radiologist analyzing a single ultrasound image acquired during fetal examination.
Your task to carefully examine the provided image, identify the primary anatomical structure or region being visualized, and determine the most appropriate description based on the standard imaging planes used in fetal ultrasound. Choose the single best option from the following list that accurately describes the main subject shown in the image.
options: 'fetal abdomen', fetal femur', fetal brain', 'fetal thorax', 'maternal cervix', "other'
only the exact text the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Output format:
Prompt Template used for throid plane classification (dataset 37)
You are a radiologist with expertise in interpreting neck and thyroid ultrasound images. You are presented with single B-mode ultrasound image focused on the thyroid gland and adjacent neck structures.
Your task to identify the Cardinal Anatomical Plane depicted in the provided ultrasound image. Choose the single best option from the following list that accurately describes the image.
options: AAxial/rrasveer Plane', 'Coronal Plane', 'Sagittal Plane"
only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or formatting.
Output format:
Prompt Template used for fetal (dataset 53)
You are radiologist analyzing a single B-mode ultrasound image obtained during fetal assessment.
Your task to carefully examine the provided ultrasound image frame identify the presence or absence of two specific anatomical landmarks: the fetal head and the maternal symphysis pubis. Based on this identification, classify the frame's content by choosing the single best option from the following list that accurately describes which of these landmarks visible. Choose the single best option from the following list that accurately describes the frames content.
options: NNoee,, 'OnlyFetaleead;; 'OnlySymphysisuubis: "SymphyyssPubiss-rFena
Output prompt:
only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or other formatting.
44

---

Published as conference paper ICLR 2026
Prompt Template used for cartoid classification (dataset 69)
You are radiologist analyzing an ultrasound image depicting portion the carotid arterial system in the neck.
Your task carefully examine the provided ultrasound image, analyzing anatomical landmarks, vessel morphology, and its position relative to other neck structures, to identify the primary carotid artery segment shown. Choose the single best option from the following list that accurately describes the main vessel visualized in the frame's content. Assume 'left carotid' and 'right carotid' refer generally to the common or internal carotid artery on that respective side, while 'external carotid' refers specifically to the external carotid artery branch. Choose the single best option from the following list that accurately describes the image.
options: "external carotid", 'left carotid', 'right carotid'
Output prompt: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or other formatting.
Prompt Template used for anatomy classification
You are an expert specialized in analyzing medical ultrasound images. You are provided with single ultrasound image frame, which could depict various parts of the human body.
Your task to analyze the provided ultrasound image and identify the primary anatomical region organ system being visualized. Choose the single best option from the following list that most accurately represents this primary anatomical subject.
options: 'fetal', 'thyroid", 'heart', 'lung', 'liver", 'carotid', 'kidney,, 'prostate', 'breast', 'other'
Output prompt: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or other formatting.
45

---

Published as conference paper ICLR 2026
Prompt Template used for knee classification
You are radiologist analyzing an ultrasound image of knee.
Your task is to classify the specific anatomical view, laterality (left/right), orientation, and any specific imaging technique or patient positioning shown in the image:
'left anterior suprapatellar longitudinal': Image of the left knee, taken from the front (ante- rior), just above the kneecap (suprapatellar), with the ultrasound probe oriented along the long axis of the thigh/patellar tendon. Standard B-mode imaging.
Same view above (left, anterior suprapatellar, longitudinal), but with Power Doppler mode activated, typically used to assess blood flow inflammation.
'left anterior suprapatellar longitudinal with power Doppler':
'lett anterior suprapatellar transverse in 30 degrees flexion': Image of the left knee, from the front (anterior), above the kneecap (suprapatellar), with the probe oriented across (transverse) the thigh, and the knee bent approximately 30 degrees.
'left anterior suprapatellar transverse in maximal flexion': Same view above (left, anterior suprapatellar, transverse), but with the knee bent as much as possible (maximal flexion).
lleft lateral longitudinal': Image of the outer side (lateral) of the left knee, with the probe oriented along the long axis of the structures (e.g., LCL, IT band).
'left medial longitudinal': Image of the inner side (medial) of the left knee, with the probe oriented along the long axis of the structures (e.g., MCL, medial meniscus).
'left posterior medial transverse': Image of the back, inner corner (posterior medial) of the left knee, with the probe oriented across (transverse) the structures (often used for Baker's cysts).
rright anterior suprapatellar longitudinal': Image of the right knee, taken from the front (anterior), just above the kneecap (suprapatellar), with the ultrasound probe oriented along the long axis of the thigh/patellar tendon. Standard B-mode imaging.
'right anterior suprapatellar longitudinal with power Doppler': Same view as above (right, anterior suprapatellar, longitudinal), but with Power Doppler mode activated.
'right anterior suprapatellar transverse in 30 degrees flexion': Image of the right knee, from the front (anterior), above the kneecap (suprapatellar), with the probe oriented across (transverse) the thigh, and the knee bent approximately 30 degrees.
'right anterior suprapatellar transverse in maximal flexion': Same view above (right, anterior suprapatellar, transverse), but with the knee bent as much as possible (maximal flexion).
"right lateral longitudinal': Image of the outer side (lateral) of the right knee, with the probe oriented along the long axis of the structures.
'right medial longitudinal': Image of the inner side (medial) of the right knee, with the probe oriented along the long axis of the structures.
"right posterior medial transverse': Image of the back, inner corner (posterior medial) of the right knee, with the probe oriented across (transverse) the structures.
Choose the single best option from the following list that accurately describes the image.
Options: 'left anterior suprapatellar longitudinal', 'left anterior suprapatellar longitudinal with power Doppler', 'left anterior suprapatellar transverse in 30 degrees flexion', "left anterior suprapatellar transverse maximal flexion', 'left lateral longitudinal', 'left medial longitudinal', 'left posterior medial transverse', 'right anterior suprapatellar longitudinal', 'right anterior suprapatellar longitudinal with power Doppler', 'right anterior suprapaellar transverse 30 degrees flexion', 'right anterior suprapatellar transverse in maximal flexion', 'right lateral longitudinal", 'right medial longitudinal', 'right posterior medial transverse'
Output prompt: only the exact text of the chosen option from the list above. Do not include any introductory phrases, explanations, numbering, or other formatting.

---

Published as conference paper ICLR 2026
Prompt Template used for lesion detection
You are radiologist analyzing an ultrasound image of thyroid.
Your task is to identify the primary location of any visible lesion(s) relative to the boundaries of the displayed image. Consider the lesion's 's center location or most prominent area when deciding. Choose the single option from the list below that best describes this location, even if the fit is approximate.
Choose the single most appropriate location from the following list:
upper left
upper center
upper right
middle left
center
middle right
lower left
lower center
lower right
not visible
Output format: only one or two word(s) representing chosen location. No additional text or formatting is allowed.
47

---

Published as conference paper ICLR 2026
Prompt Template used for organ detection
You are radiologist analyzing an ultrasound image of abdominal.
Your task is to determine the primary location, relative to the image boundaries, for each visible structure listed in liver.
Consider the structure's center or most prominent area when deciding its location.
Choose the single option from the list below that best describes the location, even if the fit approximate.
Location Options:
upper left
upper center
upper right
middle left
center
middle right
lower left
lower center
lower right
not visible
Output format: only one or two word(s) representing the chosen location. No additional text or formatting is allowed.
48

---

Published as conference paper ICLR 2026
Prompt Template used for keypoint detection
You are radiologist analyzing an ultrasound image of the heart.
Your task is determine the top inner point of the aortic valve.
Consider the structure's center or most prominent area when deciding its location.
Choose the single option from the list below that best describes the location, even the fit approximate.
Location Options:
upper left
upper center
upper right
middle left
center
middle right
lower left
lower center
lower right
not visible
Output format: only one or two word(s) representing the chosen location. No additional text or formatting is allowed.
Prompt Template used for caption generation
You are radiologist analyzing an ultrasound image focused on the {anatomy location].
Your task is generate a concise and informative caption that accurately describes the key anatomical structures and any significant findings visible in the provided ultrasound image.
Output format: A single string constituting the image caption. Output only the generated caption text itself. Do not include any introductory phrases (like Caption:), labels, explanations, or additional formatting.
Examples:
Example1: Thyroid nodule in the right lobe. TI-RADS level 3, Benign.
Example2: Thyroid nodule in the left lobe. TI-RADS level 3, Benign.
Example3: Thyroid nodule in the right lobe. TI-RADS level Benign.
49

---

Published as conference paper ICLR 2026
Prompt Template used for report generation
You are radiologist analyzing ultrasound image focused on the (anatomy location).
Your task is generate concise and informative radiological report based strictly on the visual findings within the provided image. Your report should describe the primary organ's appearance (size, shape, borders/capsule), its parenchymal echotexture (e.g., homogeneous, heterogeneous, echogenicity relative to reference structures), and identify any visible abnormalities (e.g.. masses, cysts, fluid collections, calcifications, ductal dilation). Comment on relevant adjacent structures if visualized. Use standard radiological terminology.
Output format: Strings, that is your report.
Example: The liver morphology is full with smooth capsule. The parenchymal echotexture is fine and diffusely increased. Visualization of the portal venous system is suboptimal. Intrahepatic and extrahepatic bile ducts are not dilated. The main portal vein diameter is within normal limits. The gallbladder normal in size and shape. The wall smooth and not thickened. No obvious abnormal echoes are seen within the lumen. The pancreas is normal in size and shape with homogeneous parenchymal echotexture. The pancreatic duct is not dilated. No definite space-occupying lesion seen within the pancreas. The spleen is normal in size and shape with homogeneous parenchymal echotexture. No obvious space- cecuppyc lesion seen within the spleen.
50

---

Published as conference paper ICLR 2026
E DATASET DETAILS AND LICENSE
Table 7: Summary of Annotated Datasets Used in U2-BENCH. In particular, 25 out of the 40 datasets were initially identified and compiled by /AIsharid al. (2023). and the corresponding entries here were adapted from the tables in that work. For more comprehensive dataset-level characterization of that data, we refer readers to that paper.
Dataset
Anatomy
Clinical scenarios
Task
Case
License
Fetal abdomen
Fetal brain
Fetal femur
FETAL PLANES
Fetal standard plane identification
VRA
137
CCA 4.01
Fetal thorax
DB
Burgos-
Maternal cervix
Artizzu
al.
other
2020
DDTI
Pedraza
thyroid
Thyroid nodule identification Thyroid nodule localisation
VRA
110
LL
et al. 2015
Open Kidney
US Dataset (Sigla
kidney
Kidney detection Kidney Diag view identification
VRA
110
CC-N-
et al.
2023
Fetal abdomen
Fetal diagnostic planes identifica- tion
Fetal
VRA RP
FPUS23 Prabakara
752
MIT
Fetal head
al. 2023
Fetal US report generation
Fetal legs
Echogenic
Da Cor
Fetal abdomen
Fetal abdominal organ detection
OD
102
CCA
reggio 2023)
FALLMUD FALL
Crural muscles
Muscle detection
OD
100
MOD
Micro-US Prostate
Segmentation
Prostate localisation
VRA
Dataset (Shao
Prostate
Prostate Diag view identification
LL
110
CCA 4.01
Brisbane
2024)
Heart FD
Heart FS
Heart ejection fraction estimation
VRA
CC BY-NC-
CAMUS
Leclere
Heart 2CH
Heart atrium and ventricle localisa- tion
316
SA 4.0
et a 2019
CVE
Heart 4CH
Continued next page
51

---

Published as conference paper ICLR 2026
(Continued) Table
Dataset
Anatomy
Clinical scenarios
Task
Case
License
Breast Lesion Detection
US
Breast benign
Videos
Lin et al.
Brest malignant
Breast lesion classification
Diag
171
2022
Breast US Images
Dataset
(Al
Breast cancer level classification
Diag VRA LL
Dhabyani et al.
Breast
Breast tumour localisation
210
CCO: PD
Brest Diag view identification
2020
Dermatologic U1-
trasound Images for
classifica- tion Laverde Saad 2021
Skin
Skin tumor level classification
Diag
100
ct al.
Polycystic Ovary
Ultrasound Images
Polycystic Ovary Syndrome locali-
Dataset (Wisesty
Ovary
sation
VRA
10
CCO: PDD
ct 2018
Carotid thickness estimation
VRA
CUBS MMeiurrerr
Carotid
Carotid detection Catotid Diag view identification
OD
681
CCA 4.01
2021
CVE
Knee US dataset in
Knee US KL and pain grad classifi- cation
a population-basee -based
Diag VRA
cohort (Novin 2023)
Knee
Knee Diag view identification
326
CCO 1.0
OD
Knee lesion localisation
Fetal head circumference estimation Fetal head detection
HC18
van
den
Fetal head
202
CCA 4.01
CVE
Heuvel 2018
Thyroid nodule level classification Thyroid nodule localisation
Diag I.I.
KFGNet
Neu Thyroid
206
TJTO palkia 999
2023
Thyroid Thyroid right
Thyroid al.
Krönke
Thyroid Diag view identification
VRA
563 CC BY
2022
GDPHSYSUCC 2022
M
Breast
Breast lesion classification
Diag
109
Continued on next page
52

---

Published as conference paper ICLR 2026
(Continued) Table
Dataset
Anatomy
Clinical scenarios
Task
Case
License
LEPset 2023b
Li et al.
Pancreas
Pancreatic cancer classification
Diag
101
CCA 4.01
COVID- BLUES
COVID-19 level classification Lung US caption generation Lung Diag view identification
Diag VRA CG
Wiede mann al. 2025
Lung
318
ANN 4.01
Ultrasound Guided Regional Anesthe- sia TTaggi ct al.
Brachial plexus
Brachial plexus detection
OD
179 commerical
Non-
2024)
Unity Imaging Col-
CCANN
laborative
Shun-
Cardiac
Caridac Keypoint Detection
500
4.01
Shin
2023
C-TRUS
Dataset
Leen
Colon
Colon wall detection
OD
166
ings et al.
2025
ACOUSLIC.
Fetal abdominal circumference esti- mation Fetal adominal OD
VRA
CCANCSA
Fetal abdominal
310
Al (Sappia 2024
4.01
CVE
Fetal head
Fetal head detection
PSFHS
Bai 2024
Fetal pubic sym- physis
OD
100
CCA 4.01
Fetal pubic symphysis detection
Fetal head
Fetal view identification
JNU-IFM Lu et al.
Fetal pubic sym-
Fetal head detection
VRA
202
CC 4.0
2022)
physis
Fetal pubic symphysis detection
Dataset B-mode
Liver steatosis classification
Diag VRA CVE
fatty liver US im- ages Byra et al. 2018
Liver
Liver value estimation Liver Diag view identification
222
CCA 4.01
African Fetal Stan- dard Plane (Sendra-
Fetal abdomen
Fetal brain
Fetal standard plane identification
VRA
10
CCA 4.01
Balcells et al.
Fetal femur
2023)
Fetal thorax
BrEaST Pawlowske
Breast
Breast LL
LL
100
CC 4.0
et al. 2024
Continued page
53

---

Published as conference paper ICLR 2026
(Continued) Table
Dataset
Anatomy
Clinical scenarios
Task
Case
License
Ultrasound Breast Images for Breast (airaa
Breast
Breast cancer classification
Diag
100
CCO:
Cancer
2020)
US and tion 2020
simulation
segmenta- Vitale et al.
Abdominal
Abdominal OD
OD
100
Carotid Artery UI-
External carotid
trasound and Color Doppler (Pahumi Choud right carotid
left carotid
Carotid Diag view identification
VRA
100
Apache 2.0
hary 2023)
AUTTD
Maroua
Thyroid
Thyroid lesion classification
Diag
100
2020
Auto-PCOS classi-
Polycystic Ovary Syndrome classifi
fication
Maroua
Ovary
cation
Diag VRA
218
CCA 4.01
2020>
Ploycystic Diag view identification
Auto-PCOS classi-
fication
Maroua
Ovary
Polycystic Ovary Syndrome classifi cation
Diag
100
CC 4.0
2020)
Summary of Dataset Licensing Terms. The datasets included in U2-BENCH span range of open and restricted licenses. For clarity, summarise the licensing terms here.
CCO / Public Domain (PD, PDD): Fully open; free use, modification, and redistribution without attribution, including commercial use.
CC BY CC BY 4.0 / CCA 4.01: Free with attribution; permits modification and redistribution.
CCBNC-SAAA/ CC BY-NC-SA 4.0/ variants written as ANN 4.0 CCANN 4.0 I, CCANCSA 4.01: Non-commereial use only; derivatives must adopt the same license.
BY-NCNN: Attribution required; non-commercial; no derivatives permitted. Minor naming variations follow the dataset providers' release notes.
MIT License: Permissive license allowing free use, modification, and redistribution, including commercial applications.
Apache 2.0: Permissive license with explicit patent grant.
Non-commercial data use agreement: Access provided strictly for non-commercial research; redistribution or reuse requires separate permission.
Unspecified / "_"; Publicly released datasets without an explicit license. Usage follows the terms communicated by the original authors.
54


Reviews:
[{"review_id": "6dnUA3wluQ", "reviewer": "Reviewer_goEb", "summary": "This paper introduces U2-BENCH, the first comprehensive benchmark designed to systematically evaluate Large Vision-Language Models (LVLMs) on *ultrasound understanding* tasks. The benchmark includes 7,241 ultrasound cases across 15 anatomical regions and defines 8 clinically inspired tasks (disease diagnosis, view recognition, lesion localization, organ detection, keypoint detection, clinical value estimation, report generation, and caption generation) covering 50 ultrasound application scenarios. The authors benchmark 20 state-of-the-art LVLMs (open/closed-source, general/medical-specific) using standardized prompts and evaluation metrics. Results reveal that current LVLMs perform well on image-level classification but struggle with spatial reasoning and clinical language generation.", "questions": "'??' at Line 274，missing reference", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper presents a well-motivated and comprehensive benchmark targeting an underexplored domain—ultrasound imaging. Its breadth (15 anatomies, 8 tasks) and evaluation rigor (20 models, standardized prompts) represent a good contribution.\n\n- Twenty sota LVLMs are evaluated and compared, which makes the benchmark comprehensive. This benchmark would be beneficial for the medical multimodal community", "weaknesses": "- While I appreciate the substantial effort invested in constructing this benchmark and the meticulous annotation process, the paper currently lacks sufficient conceptual or analytical insights for the research community. As noted in Lines 132–134, prior work such as GMAI-MMBench has already included ultrasound-related evaluation scenarios. Although U2-BENCH expands the dataset scale and task diversity, the incremental novelty over existing benchmarks appears marginal, focusing primarily on scope rather than methodological innovation.\n- Moreover, the experimental findings are largely expected: closed-source LVLMs outperform open-source and medical-specific models, reflecting well-known trends in multimodal evaluation. Given the proliferation of benchmark papers in the field, it would be more valuable if this work offered deeper analysis or actionable insights, such as identifying failure patterns or proposing improved solution, rather than solely reporting model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P9DULYIY7q", "reviewer": "Reviewer_FVnR", "summary": "The paper introduces U2-BENCH, a benchmark for evaluating large vision‑language models (LVLMs) on ultrasound understanding with 7,241 studies across 15 anatomies and 8 task types spanning classification, detection, regression, and text generation. It reports results for 20 LVLMs and proposes a composite “U2‑Score” that aggregates task metrics via sample‑count weights. Key findings show image‑level classification and some regression tasks are relatively strong, while spatial reasoning tasks (e.g., keypoint detection) and clinical text generation remain challenging; a prompt ablation indicates anatomy tokens boost diagnostic accuracy, and closed‑source models generally outperform open‑source ones.", "questions": "Can you report inter‑rater variability for the 10‑annotator review (e.g., per‑task agreement, adjudication rates) and quantify rejections during automated filtering to bound annotation noise ?\n\nPlease ablate prompt components beyond anatomy tokens on tasks where models struggle (e.g., KD, RG) to separate instruction‑following failures from perceptual limits.\n\nCan you define minimal clinically acceptable thresholds for CVE, DD, and RG to contextualize utility and report the fraction of predictions meeting those thresholds ?\n\nPlease clarify and detail the “standardization, sampling, and quality checks” pipeline across 40 datasets, including per‑dataset preprocessing, label harmonization, leakage controls, and cross‑dataset consistency checks.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The benchmark targets ultrasound, a clinically crucial yet under‑evaluated modality for LVLMs, with a broad task suite aligned to typical sonography workflows and clear task definitions and prompts per scenario.\n\nThe evaluation spans 20 modern LVLMs with standardized prompt formats and metrics, including most popular and current SOTA models, open- as well as closed-source.\n\nThe dataset curation aggregates many sources and applies multi‑stage QA with automated filtering plus manual review, and the authors release prompts, metrics, and an evaluation toolkit to facilitate reusability.", "weaknesses": "The composite U2‑Score’s task weights are proportional to sample counts, which conflates data availability with clinical importance and mixes heterogeneous metrics into a single scalar without uncertainty quantification, making ranking sensitivity high and potentially misaligned with clinician priorities.\n\nNo uncertainty metrics (e.g., confidence intervals, paired tests, bootstrap CIs) are reported for main tables, so small deltas across 20 models and many tasks may reflect sampling noise or prompt variance rather than true differences. This is especially problematic for small tasks like caption generation. Thus also not allowing a statistical significance test in order to determine the best models.\n\nError analysis is insufficient: failures are not stratified by artifact types, view standardization, anatomy, or acquisition protocol, limiting insight into whether errors stem from perception, spatial reasoning, or prompt sensitivity.\n\n\nDataset transparency gaps: heavy anatomical imbalance is acknowledged, but standardization, sampling, and QA procedures are described briefly without inter‑annotator agreement, rejection counts, or harmonization diagnostics across 40 sources.\n\n\nReproducibility is limited by heavy reliance on closed‑source models whose APIs evolve, and by omitted inference details (e.g., seeds, temperatures, multi‑pass voting) in the main leaderboard.\n\n\nMinor issues: a missing section reference (Line 274), a typo (“Brest”→“Breast”), and Table 1 lacks per‑task sample counts and task‑abbreviation expansions in the caption despite being the primary results table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "142niNZ3gY", "reviewer": "Reviewer_Hrs9", "summary": "This manuscript presents U2-BENCH, a comprehensive benchmark for large VLM ultrasound understanding on eight clinically-inspired classification, detection, regression and text generation tasks, with 20 SOTA LVLMs evaluated on the benchmark.", "questions": "1. For Figure 2, the statement that \"...The length of the bar reflects the number of samples for each anatomy region\" may be slightly confusing, since it is not immediately clear that this refers to the blue bar. If so, this might be clarified in the caption.\n\n2. In Figure 3, language unification is stated, but its implementation does not appear to be discussed. This appears relevant since the only dataset used for report generation (RG), the Chinese Ultrasound Report Dataset, appears to be in Mandarin.\n\n3. In Section 3.2, format unification was stated to be performed on ultrasound scans by converting to a uniform image format. It might be briefly stated as to whether any conversion loss (either in image quality due to compression, or image dimensions due to cropping etc.) was involved.\n\n4. In Section 3.2, it is stated that a small number of representative frames were sampled, for video sequences. It might be clarified as to how the sampling was performed, since this would appear to restrict video analysis scope.\n\n5. In Section 3.2, it is stated that segmentation masks are converted to bounding boxes. However, this would appear to imply a loss of granularity (since the pixel-level annotations are simplified), and a potential loss of individual objects (since bounding boxes may overlap for originally-distinct irregular objects). This limitation might be considered if relevant.\n\n6. In Section 3.2, the reference to the ablation study is missing.\n\n7. In Section 4.1, it is stated that detection tasks were converted to position classification tasks, and utilized accuracy as the metric. This appears to discount segmentation models/tasks unnecessarily, despite their clinical utility.\n\n8. In Section 4.2, individual performance metrics may be expanded from relevant literature - for example, while accuracy is the main metric for classification and detection tasks, AUROC etc. may be considered for inclusion.\n\n9. In Section 4.3 (or the Appendix), the implementation for Random Guessing might be described.\n\n10. In Appendix B, it is stated that all data was either publicly released with appropriate usage permissions or obtained through official licensing agreements (assumed for datasets with no license stated, in Appendix E). It might be clarified that these licensing agreements include redistribution and modification of the data.\n\n11. In Appendix C.2, any particular VLM parameter settings (especially relating to temperature) might be stated.\n\n12. In Appendix C.3, the normalization for $d_t$ (e.g. by assuming a normal distribution on the scores of the evaluated VLMs) might be stated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- (Quality) Application of a large range of existing VLMs to the benchmark\n\n - (Clarity) Empirical justification for weighing of tasks\n\n - (Significance) Extensive coverage of ultrasound understanding datasets, and unification into a single comprehensive benchmark", "weaknesses": "- Segmentation task underrepresented due to unification of ground truth to bounding boxes and predefined spatial localization\n\n - Preprocessing of video data may limit analysis potential", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y35fEkku3o", "reviewer": "Reviewer_nz9Q", "summary": "This paper introduces U2-BENCH, a benchmark for evaluating LVLMs on ultrasound. Its core contributions are a public dataset (7,241 cases spanning 15 anatomical regions), a systematic evaluation of 20 LVLMs, and important observations regarding model capabilities and limitations in this domain.", "questions": "1. I wonder the time cost for data collection and curation, and annotation. And how scalable is this process for future expansion?\n2. The current findings are intuitive. Could you provide a deeper analysis to uncover the underlying reasons for LVLMs' failures, which would significantly elevate the contribution?\n3. For KD/LL tasks, did you experiment with more precise evaluation methods (e.g., bounding box coordinates, point regression) to validate if the coarse-grained task is a reliable proxy for fine-grained spatial reasoning?\n4. Beyond the U2-Score, what actionable guidance does the benchmark provide for model developers to improve performance on ultrasound tasks?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The main strength of this work lies in its effort to create and release U2-BENCH for a specialized domain, which may serve as a resource for the community.\n2. The evaluation of 20 diverse LVLMs provides a preliminary landscape of current capabilities, offering a baseline for comparison.", "weaknesses": "1. The benchmark's scale and imbalance undermine its reliability. With only 7,241 cases thinly spread across 15 anatomies and 8 tasks, the effective sample size for many evaluations is small.\n2. The design of certain tasks is problematic. For instance, reducing complex spatial tasks like KD / LL to a 9-class classification problem is a significant oversimplification.\n3. The paper primarily presents a performance leaderboard but the key findings are largely intuitive. The analysis remains on the surface, which severely limits the paper's contribution beyond a mere performance report.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)