FETAL-GAUGE: A BENCHMARK FOR ASSESSING VISION-LANGUAGE MODELS IN FETAL ULTRASOUND
Authors: Hussain Alasmawi
Numan Saeed
Mohammad Yaqub
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Models
Fetal Ultrasound
Visual Question Answering
Primary area: datasets and benchmarks
TL;DR: Fetal-Gauge is the first large-scale benchmark for Vision-Language Models in fetal ultrasound, with 42k images and 93k QA pairs. Testing 15 models shows major gaps, best accuracy is 55%, highlighting challenges in dealing with fetal ultrasound.

Abstract: The growing demand for prenatal ultrasound imaging has intensified a global shortage of trained sonographers, creating barriers to essential fetal health monitoring. Deep learning has the potential to enhance sonographers' efficiency and support the training of new practitioners. Vision-Language Models (VLMs) are particularly promising for ultrasound interpretation, as they can jointly process images and text to perform multiple clinical tasks within a single framework. However, despite the expansion of VLMs, no standardized benchmark exists to evaluate their performance in fetal ultrasound imaging. This gap is primarily due to the modality’s challenging nature, operator dependency, and the limited public availability of datasets. To address this gap, we present Fetal-Gauge, the first and largest visual question answering benchmark specifically designed to evaluate VLMs across various fetal ultrasound tasks. Our benchmark comprises over 42,000 images and 93,000 question-answer pairs, spanning anatomical plane identification, visual grounding of anatomical structures, fetal orientation assessment, clinical view conformity, and clinical diagnosis. We systematically evaluate several state-of-the-art VLMs, including general-purpose and medical-specific models, and reveal a substantial performance gap: the best-performing model achieves only 55\% accuracy, far below clinical requirements. Our analysis identifies critical limitations of current VLMs in fetal ultrasound interpretation, highlighting the urgent need for domain-adapted architectures and specialized training approaches. Fetal-Gauge establishes a rigorous foundation for advancing multimodal deep learning in prenatal care and provides a pathway toward addressing global healthcare accessibility challenges. Our benchmark is publicly available at https://github.com/BioMedIA-MBZUAI/FETAL-GAUGE

Published as a conference paper at ICLR 2026
FETAL-GAUGE: A BENCHMARK FOR ASSESSING VISION-LANGUAG MODELS IN FETAL ULTRASOUND
Hussain Alasmawi Numan Saeed & Mohammad Yaqub Mohamed bin Zayed University of Artificial Intelligence (MBZUAT), UAE
ABSTRACT
The growing demand for prenatal ultrasound imaging has intensified global shortage of trained sonographers, creating barriers to essential fetal health mon- itoring. Deep learning has the potential enhance sonographers' efficiency and support the training of new practitioners. Vision-Language Models (VLMs) are particularly promising for ultrasound interpretation, as they can jointly process images and text to perform multiple clinical tasks within a single framework. However, despite the expansion of VLMs, no standardized benchmark exists to evaluate their performance in fetal ultrasound imaging. This gap is primarily due to the modality's 's challenging nature, operator dependency, and the limited pub- lic availability of datasets. To address this gap, present Fetal-Gauge, the first and largest visual question answering benchmark specifically designed to evalu- ate VLMs across various fetal ultrasound tasks. Our benchmark comprises over 42,000 images and 93, 33,000 prrestion-aswwar pairs, spanning anatomical plane iden- tification. visual grounding of anatomical structures, fetal orientation assessment, clinical view conformity, and clinical diagnosis. We systematically evaluate sev- eral state-of-the art VL.Ms, including general- purppss and medical-specifit mod- els, and reveal substantial performance gap: the best-performing model achieves only 55% accuracy, far below clinical requirements. Our analysis identifies criti- cal limitations of current VLMs in fetal ultrasound interpretation, highlighting the urgent need for domain-adaad adapted architectures and specialized training approaches. Gauge establishes rigorous foundation for advancing multimodal deep learning in prenatal care and provides a pathway toward addressing global health- care accessibility challenges. Our benchmark is publicly available here!
INTRODUCTION
Ultrasound is the primary modality for monitor- ing fetal health. In 2024, more than 132 million babies were born worldwide (Database Earth (205)), and most pregnancies involved multi- ple ultrasound scans, typically averaging about 6-7 scans over the course of routine antenatal visits, with some women receiving as many as 8-10 scans (Susu et al. (2025)) depending maternal health and resource availability. UI- trasound can detect up to 85% of major fetal anomalies (Dulgheroff et al. (2019), highlight- ing its critical role in prenatal care. However, the growing reliance on ultrasound has intensi- fied the global shortage of trained sonographers (Won et al. (2024).. Expanding the workforce alone may meet demand, given the time and resources required for training. This challenge underscores the need for innovative solutions to ensure universal access high-quality prenatal imaging.
Figure 1: Number of questions categorized by the
number of models answering correctly. Each bar represents how many of the 15 models answered a question correctly. For example, the bar at 15 represents the set of questions that all models an- swered correctly, comprising only 38 correctly an- swered questions out of total of 21,468.
"Email: Hussai i n. Alasmawi @mbzuai ace.a

---

Published as a conference paper at ICLR 2026
Deep learning (DL) methods have shown great promise in ultrasound interpretation, with advances in tasks such as plane classification (Maani et al. (2025)). biometric measurement (Qazi et al. 2023), ) and anomaly detection (Arnaout al. (2021). Taratynova et al. (2025)). DL has the po- tential to enhance efficiency by reducing scan time, improving image quality, and supporting the training of new sonographers. Among emerging DL approaches, Vision-Langgage Models (VLMs)
stand out for their ability to jointly process visual and textual information. These models enable wide range of tasks, including image classification, segmentation, report generation, and interactive question answering. While several benchmarks exist to evaluate VLMs in medical imaging, there is no benchmark dedicated to the fetal ultrasound domain, primarily due its challenging nature, de- pendence on operator expertise, and the limited public availability of annotated datasets. To address this gap, we present the first comprehensive benchmark for evaluating VLMs fetal ultrasound interpretation. We introduce the first large-scale benchmark for fetal ultrasound VL.M evaluation, constructed by integrating 13 publicly available datasets. Our contributions are:
Fetal-Gauge dataset: A benchmark of 42,036 images and 93,451 question guestion-answer answer pairs, spanning diverse anatomical regions, clinical tasks, and question types. This is the first and largest dataset enabling reproducible evaluation of VLMs in fetal ultrasound.
Comprehensive VLM evaluation: We systematically benchmark 15 state-of-the-ane art VLMs. including general-purposs. (open and closed-source) and medical-specific, under a unified evaluation framework highlighting their capabilities and limitations in assessing fetal ultrasound.
Critical performance analysis: We analyze performance for fetal ultrasound interpreta- tion, including visual grounding of different structure sizes, handling of phantom images, and qualitative error patterns. providing key insights into their limitations.
This benchmark establishes foundation for multimodal medical DL in fetal ultrasound, highlight- ing open challenges and setting the stage for future progress in applying VLMSs to real-world clinical tasks.
2
RELATED WORK
2.1 MEDICAL VISUAL QUESTION ANSWERING DATASETS
Medical Visual Question Answering (Med-VQA) has grown rapidly in recent years, producing datasets to address challenges in clinical image interpretation. Early efforts such as VQA-Med (Ben Abacha et (2019)) (4,200 radiology images, 15,292 QAs) and VQA-RAD (Lau et al. ( (3,515 QAs from 315 radiology cases) established structured question categories and manual cura- tion by clinical experts. SLAKE (Liu al. (20TT) expanded to bilingual knowledge- based setting. with 14,028 Q&As over 642 radiology images annotated with rich semantic labels.
To improve scalability, PMC-VQA (Zhang al. (2023b)) leveraged figure- caurron pairs from med- ical publications to generate 227k VQA pairs from 149k images, though its reliance on paper figures introduces noise and limited clinical realism. More recent large-scale datasets, such OmniMed- VQA (Hu et al. (2024)) (12 modalities, all from real clinical scenarios) and CAREs (Xia et al. (2024) (16 modalities with evaluation of confidence, fairness, and safety), reflect a shift toward multimodal, multi-metric evaluation. Specialized datasets like Vat (He et al. (2020) demon- strate the benefits of domain-specific collection.
However, no public fetal ultrasound VLM dataset exists, despite ultrasound being the primary imag- ing modality in prenatal care worldwide. Existing medical datasets concentrate on adult imaging modalities (CT, MRI, X-ray, histopathology), overlooking domain that demands reasoning over noisy, pperator-depende bottleneck for standardized benchmarking and hinders the development of multimodal medical DL systems capable of addressing clinically important tasks in fetal health, such as anomaly detection, gestational age estimation, and automated reporting in prenatal care.
and contains various anatomical views. This omission creates a critical
2

---

Published as a conference paper at ICLR 2026
2.2 VISION-LANGUAGE MODELS MEDICAL IMAGING
Recent advances in Vision-Language Models have catalyzed sigmificant progress in medical image understandingg, with specialized medical VLMs emerging to address domain-specific challenges. These models generally follow two primary approaches: curriculum learning with medical data adaptation and retrieval-augmented architectures.
Within the first approach, several models employ staged training strategies to adapt general vision- language capabilittes to medical domains. LLaVA-Med (Li et al. (2023) pioneered cost-efficient biomedical adaptation by combining PubMed figure-caption datasets with GPT-4 self-instruction, using a two-stage curriculum that first aligns biomedical vocabulary and then masters conversational semantics. However, its reliance on publication-derived data introduces quality limitations due to inherent noise and compression artifacts. MedVLM-RI (Pan et al. (2025)) advances this approach by incorporating explicit reasoning generation, achieving remarkable improvements from 55.11% 11% to 78.22% accuracy across MRI, CT, and X-ray tasks using Group Relative Policy Optimization (Shao al. (2024) with only 600 training samples and 2B parameters.
Large-scale foundation models aim for broader modality coverage. Lingshu (Xu et al. (2025) ad-
dresses medical VLM limitations through training across eight imaging modalities, including adult ultrasound, enabling cross-modal understanding and generalization. HuatuoGPT-Vision (Zhang al.] (203aa) scales this approach further with 34B parameter model trained on refined PubMed image-text pairs across multiple modalities, representing one of the largest medical vision-language models available.
Specialized architectures also emerge for domain-specific optimization. MedGemma (Sellergren al. (2055) combines retrieval techniques with fine-tuned Gemma 2 models, providing broad spe- cialty coverage across radiology, dermatology, pathology, and ophthalmology while emphasizing research accessibility.
Despite these advances, no medical VLM has been systematically evaluated on fetal ultrasound. The modality poses unique technical challenges for VLMs: integrating fine-grained spatial reason- ing, interpreting images with substantial inter-operator variability, and coping with artifacts absent in other medical imaging modalities, such as standardized imaging. Without structured, targeted benchmarks, these limitations remain invisible, hindering progress toward clinically useful multi- modal DL in prenatal care.
3 THE FETAL-GAUGE BENCHMARK
To address the critical gap in standardized evaluation for vision-language models in fetal ultrasound, we introduce Fetal- Gauge, a large-scale, multi-task VLM benchmark. This section details its con- struction. Section ddnfeiies the five core clinical tasks Fetal-Gauge is designed evaluate. Section 3.2 outlines the data curation and standardization pipeline. Section [33 presents the data splitting strategy and final dataset statistics. Finally, section 3.4 discusses the importance of phantom images in our dataset.
3.1 TASK DESIGN
Fetal- Tetal-Gauge is structured around five clinically distinct tasks and designed to assess model's ca- pabilities, from high-level scene understanding to fine-grained anatomical localization. Each task is formulated as multiple-choice question (MCQ), format chosen for its simplicity, amenability to straightforward evaluation, and ability to reduce the ambiguity inherent in free-text responses, thereby ensuring objective. scalable, and automated assessment. This structured approach also min- imizes biases associated with open-ended responses, prevents hallucinations, and improves the fair- ness of model assessment. Figure 2 shows sample of the images of our dataset, including the question we are using for each category. The tasks are:
Anatomical Fetal Plane Identification (PI): Models must identify the specific anatomical plane shown in the ultrasound image (e.g., abdominal, trans- thalamic). This task evalu- ates fundamental image recognition and classification capabilities. In addition, this is an essential clinical task performed by sonographer during the assessment of fetal growth.
3

---

Published as a conference paper at ICLR 2026
What is the pians In the image?
Is this a standard view?
What the fetus's orientation? ?
Does this image show normal. benign, malignant condition?
What does the red bounding box represent?
Figure 2:
Question types in the dataset, along with representative sample images for each type, highlighting the diversity of visual content and associated questions.
Clinical View Conformity (VC): Models must determine if an image meets the criteria of clinically accepted standard view, reflecting their ability assess adequacy for diagnostic use. This is an essential clinical task performed by sonographers to confirm that images are diagnostically adequate, ensuring accurate biometric measurements, reliable anomaly detection, and standardized reporting.
Fetal Orientation Assessment (FO): This task requires the model to determine the orien- tation the fetus within the scan, crucial step in clinical assessment. Clinically, knowing fetal orientation is crucial for establishing presentation, guiding measurement techniques, detecting positional abnormalities, and aiding delivery planning and parental counseling.
Clinical Diagnosis (CD): This task involves classifying the image as showing a normal, benign, malignant condition, evaluating the model's capacity for clinical diagnostic rea- soning. In practice, accurate classification enables timely clinical decision-man maasionnmaaing,. ap- propriate referrals, and patient counseling, supporting management strategies and ensuring that abnormalities are promptly identified and addressed.
Visual Grounding of Anatomical Structures (VG): Given an image with a red bounding box, the model must identify the anatomical structure highlighted within it. This task di- rectly assesses the model's spatial reasoning and fine-grained object recognition. Clinically, precise localization of structures is fundamental for measurement, monitoring fetal devel- opment, guiding based interventions, and improving inter-observer consistency in assessments.
3.2 DATASET CURATION AND STANDARDIZATION
The construction Fetal-Gauge followed a systematic pipeline unify disparate data sources into cohesive benchmark.
Source Aggregation. We began by aggregating thirteen publicly available fetal ultrasound datasets (detailed in the Appendix Table 4 This multi-source approach was crucial for ensuring diversity imaging conditions, ultrasound machinery, hospital protocols, and patient demographics, thereby promoting the development of generalizable models.
Task and Annotation Unification. One the main challenges was standardizing heterogencous annotation types (e.g., image-level labels, segmentation masks, and bounding box annotations) into unified format. For datasets containing segmentation masks, each mask was converted into a visual grounding task extracting its bounding box coordinates. The bounding box was then overlaid onto the image as red rectangle, enabling the formulation of "What does the red bounding box represent?").
questions (c.g.,
4

---

Published as a conference paper at ICLR 2026
Brain 58%
Heart
7,358
38%
11%
70%
30%
2,626
100%
VG
CV
Kidney
2,624
Maternal
2,480
31%
Thorax
100% 1,793
Other 70%
55,985
CD
27% 3%
Figure 3: Distribution of benchmark tasks across anatomical regions. Colored segments represent question categories (legend on the right), with proportions shown within each bar and total counts indicated the right. PI: Anatomical Plane Identification, VG: Visual Grounding of Anatomical Structures, VC: Clinical View Conformity, FO: Fetal Orientation Assessment, CD: Clinical Diag- nosis.
Vocabulary Normalization. To minimize label noise from heterogeneous sources, standardized answer vocabulary. This involved expanding clinical abbreviations (e-g.. "abdomcire" "ab- dominal cireumference plane") and harmonizing synonymous terms to ensure consistency across the entire benchmark. For certain datasets, the specific imaging plane within an organ was not indi- cated-fo for example, the heart was labeled simply as "heart plane" rather than specifying views such as Thrree-vesse plane" or "Four-chamber plane," and similar omissions occurred for the brain and abdomen. In such cases, we retained generic "[organ] plane" label ensure consistent terminol- ogy.
3.3 DATA PARTITIONING AND STATISTICS
Splitting Strategy. To ensure that model evaluation reflects true generalization capabilities rather than patient-specific memorization, adopted rigorous splitting strategy. Where available, preserved the original train-test splits from the source datasets. For datasets without a prespecified split, enforced strict patient-wise splits prevent data leakage. Furthermore, to robustly assess generalization to unseen data distributions, datasets with limited sample sizes were allocated exclu- sively to the excluding classes with limited clinical value (e.g.. "other"), focusing the benchmark on well-defined and meaningful clinical tasks.
(detailed in Appendix Table During this process, curated the dataset by
Dataset Scale. The Fetal-Gauge benchmark is the most extensive collection of fetal ultrasound VLM data date, comprising 42,036 images and 93,451 question -antwr pairs. This includes significant portion of phantom images (19k) for specialized task evaluation. The scale and diversity of Fetal-Gauu Fetal-Gauge provide a robust foundation for training and comprehensively evaluating modern vision-language models. Figure provides the distribution of our dataset per anatomy.
3.4 THE ROLE OF PHANTOM DATA
substantial portion of Fetal-Gauge (19k images) is composed of data from anatomical phantoms. This is not limitation but strategic feature of our benchmark. Phantoms are the standard-of-cart for training sonographers, allowing them to develop probe handling skills and learn to recognize standard planes
controlled, repeatable environment. By including this data, we enable the development of DL systems designed for clinical practice, education, and simulation. This creates pathway for future work where DL models could be trained and validated on phantoms before being deployed, or even serve as interactive training aids for human novices.
4 EVALUATIONS AND RESULTS
This section presents comprehensive evaluation of state-of-the th- VLMs the Fetal-Gauge benchmark. We first detail the experimental setup and then provide a facti-faceted analysis of model performance, both overall and per-task basis.

---

Published as a conference paper at ICLR 2026
4.1 EVALUATION SETUP
Evaluated Models. We selected 15 prominent VLMs for evaluation. This cohort includes six models with a specialization in medical imaging (Lingshu-7B (Xu et al. (202)), Lingshu-32B. McdVLM-RI (Pan et al. (2025)). MedGemma-4b-4n (Sellergren et al. (2025)). MedGemma- 2e-Gemmma227-ii,
HuatuoGPT-Vision-77 (Zhang et al. (2023a)), )), cight leading general
general-purpose purpose models (Intern
8B-Instruct (Zhu et al. (2025), Intern (L33-14B-Instruct Llama-3.2 2-1 1-B- Vision-Instruct (Dubey (Wang et al. (2024)), Qwen2.5 11-22.5--L-3225
et al. (2024)), Qwen2.5- 1----------nss-
Aya-
Vision-8b (Dash at. (225)), Aya- Wya-Vision-32b. vip-llava-7hh (Cai et (2024)) and one commercial model GPT-5 (OpenAl| (2025)). ). A random guess baseline was added to assess whether the model's high performance reflects real understanding rather than random chance. Additionally, we fine-tuned and Llama-3.2 3.2-1 IB- Fision-Instrue on our training set using LoRA (Hu ctt al. (2021)) for various amounts of epochs (referred as model x where x represents the number of epochs) to evaluate the impact of domain-specific training on model performance.
Qwen2.5-VL
Evaluation Protocol. Given the multiple-choice question (MCQ) format of Geugle we use accuracy as the primary evaluation metric. Performance is measured across the entire test set on per-task basis to enable a granular analysis.
4.2 OVERALL PERFORMANCE ANALYSIS
Our comprehensive evaluation reveals that fetal ultrasound interpretation poses significant chal- lenges for current VLMs. As illustrated in Figure ! the distribution of correct answers across our 15 evaluated models shows concerning patterns: very few questions were answered correctly by all models, with the majority of questions being answered correctly by only a small subset of models. This suggests fundamental limitations in current VLM architectures for this domain.
Table presents the detailed performance breakdown across all five evaluation tasks. The results demonstrate a clear performance hierarchy: GPT-5 achieves the highest overall accuracy 55%, followed by the Lingshu models (32B: 46%, 7B: 40%), while most other models perform at or near random chance levels (26% overall).
Table 1: Model accuracy across question types. Best accuracy per column is bold, second-best is underlined, The second block of rows corresponds fine-tuned models.
Medi
vc
DrecE
RANBUM GUBSS
0A3
0.5
R28
058
0.2
DJF
ASG. MIA-VBUN--- S8
622
038
0.3
622
633
034
422
0.8
623
INTERNNES BIE DSTTIRE
635
033
105
0.99
INTERNA BRIKTTRIK DRIKT
618
016
037
TISEERT TI
631
0111
1730
035
20
TRRSHUT TE
834
004
1030
0330
2715
LLAME RS TIB VISON INSTRCTY
641
005
0.35
0335
631
103
METIGEMRAA 270
033
0.45
0.23
0.30
0:37
602
032
0.4
0.22
0.22
0.27
628
MEDDNESS
031
0.54
0.36
0.25
022
CHENLSSPLLEE BOST BLET
033
0:56
0.22
0.32
0.27
022
QWINES-AL TIANN TRACT
604
0038
1.24
0.99
0:23
1034
VIFLLAM LAV-TS
035
1006
102
0.30
GPI3
040
602
0.30
LLAM -OBINNN DRUCT
0/8
0.45
080
LAMM-32-TI 2-1-------II-E- VISINNN DIICTS
088
0.47
039
060
0.6
628
TLAMA EETTE PEEEUINTTRU KTJ
638
018
0.33
638
GAENTESE ININTICUTA
611
078
0.30
QAPATEVE VI TENNTR ETS
611
011
11
1.99
617
VE TE TENTUR TI
231
611
035
611
60
OATNEE 38 INTRECTO
641
620
64
035
601
4.3 TASK-SPECIRIC PERFORMANCE
Anatomical Plane Identification (PI): This fundamental classification task reveals the largest per- formance variations among models. While most models struggle random chance (26%), several demonstrate meaningful capabilities: GPT-5 leads with 66% accuracy, followed by Lingshu- 32B (53%) and Llama-3.2-11B (40%).
Clinical View Conformity (VC) & Fetal Orientation Assessment (CD): All models performed at near-random levels these tasks, with none showing any meaningful performance.

---

Published as a conference paper at ICLR 2026
Visual Grounding of Anatomical Structures (VG); Spatial localization tasks reveal clear perfor- mance tiers. GPT-5 achieves the highest accuracy (58%), followed by Lingshu models (32B: 47%, 7B: 45%). Most other models cluster near the 25% random baseline, suggesting fundamental limi- tations in fine-grained spatial reasoning.
4.4 DOMAIN ADAPTATION THROUGH FINE-TUNING
After task-specific fine-tuning, 3.mma-3.2-1IB improves substantially from 33% to 85% overall curacy, with consistent gains observed across all tasks. Qwen2.5-V also shows clear improvements, increasing from 24% 52% overall aceuracy across the same set of tasks.
4.5 PHANTOM VS. CLINICAL PERFORMANCE
We evaluated model performance phantom and real ultrasound images for two tasks: Anatomi- cal Plane Identification (PI) and Visual Grounding of Anatomical Structures (VG). The evaluation dataset included 4,045 phantom VG questions, 1,146 phantom PT questions, 7,286 real VG ques- tions, and 5,417 real Pl questions.
Figure presents the comparative results across models. Performance on phantom tasks was gen- erally poor, with most models achieving accuracies close to or below the random guess baseline. In contrast, performance real clinical images was substantially higher, with nearly all models surpassing random chance levels.
Among the models tested, Lingshu-32B and GPT-5 demonstrated the strongest performance. Lingshu-32B exceeded 50% accuracy on both PI and VG real-image tasks, while GPT-5 consis- tently ranked highest overall, with balanced performance across phantom and real domains.
Phantom vs Clinical Performance of Models
Prodit
ALUAA
94
63
recuantt ofter
www. W017
Figure 4: Bar plot presenting model accuracy on phantom and clinical ultrasound images through two questions ('Plane of the image' and Bounding box label')
4.6 IMPACT OF ANATOMICAL STRUCTURE SIZE
To further evaluate visual grounding, we analyzed performance based bounding box size. Our evaluation set is composed of 7,373 small, 1,799 medium, and 2,160 large questions. Results are summarized in Table Models performed best on large structures, with accuracies often exceeding 80%. However, performance dropped sharply for medium and small targets, where accuracies were frequently below 50%.
ANALYSIS
Commercial advantage of GPT-5. GPT-5 consistently outperformed all other models across tasks, achieving the highest accuracy in both phantom and clinical datasets. As a closed-source commercial model trained aarge-scale proprietary data, it is possible that its training distribution included fetal ultrasound images or closely related medical data. This may explain its superior performance compared to open-source models, which lack access to such data.
7

---

Published as a conference paper at ICLR 2026
Table 2:
Performance comparison of models based on bounding box size. The best results are in
bold and the second-best are underlined.
Model
Small
Mediam
Large
AYA- VISION 328
0.22
0.09
0.04
VE
0.20
0:11
0.20
HUATUOGPEVISSON
0.21
0.14
0:45
INTERNVL3-14
0.20
0:20
INTERNVI 1. 1-11------SSLCCC
0.19
0.13
0.10
LINGSHU
LINGSHU-7B
0.34
0.45
0.82
LLAMA- INSTRCT
0.18
MEDIGEMMA-2 27-177
0.29
0.32
0.67
MEDGEMMA 48-IT
0.21
0.31
0:43
MEDVLM-RI
0.55
0.04
0.06
S-L INB-INSTRUCT
0.29
0.13
0.35
QWENZ S-VL 7I--INISTRUUCC
0.23
0.28
VIP-LLAWA-78
0.23
0.13
0.30
GPT3
Limited utility of existing medical VLMs. None of the evaluated medical VL.Ms reported training on fetal ultrasound data, which likely explains their limited performance. While they leverage adult MRI and CT scans, these modalities differ significantly in appearance and resolution from fetal ul- trasound. Nevertheless, such data still provide anatomical priors closer to fetal imaging than natural image distributions (e.g., cars, trees, animals). This partial domain relevance likely contributed to the modest but still insufficient performance observed in these models.
Role of ultrasound-specific training. The Lingshu models represent the only group that explicitly reported training on adult ultrasound data. This appears have been critical for their relatively strong performance among open-source models. In particular, Lingshu-32B achieved over 50% ac- curacy in Anatomical Plane Identification and Visual Grounding on real images, suggesting that exposure to ultrasound imaging, even of adults, provides transferable knowledge that aids general- ization to fetal ultrasound.
Domain adaptation and fine-grained localization. The phantom clinical comparison under- scored the persistent domain adaptation gap, with phantom images proving particularly challenging for all models. Additionally, bounding box analysis revealed that models are far more successful at grounding large anatomical structures than small or medium ones. This highlights an ongoing weakness in fine-grained localization, which is critical for clinical tasks requiring precise anatomi- cal identification.
5.1 QUALITATIVE ANALYSIS
To better understand the limitations of the models, we performed a qualitative analysis of challenging cases where many models provided incorrect answers. Figure showcases several of these instances from both clinical and phantom datasets.
Case A: Clinical Image Challenges In the first case (FigureBAA, left), most models failed identify the four-chamber plane correctly. The view is significantly zoomed out. making the key anatomical features less distinct. We observe that many models incorrectly selected the transventricular plane and the transverse kidney plane, suggesting that they were confused by the general elliptical shape present in all three planes, rather than identifying the specific internal structures.
Similarly, in the second image (Figure 5A, center), no model correctly identified the transverse kidney plane, despite its straightforward presentation. Instead, models predominantly chose the transcerebellar plane and four-chamber plane, which also present as elliptical shapes. This indicates a potential model bias towards more commonly encountered elliptical structures.
The third example (Figure 5A, right) shows that while about half of the models correctly identified the transcerebellar plane, significant number chose the transthalamic plane. This confusion is understandable, as these two planes anatomically close and can be challenging to differentiate, even for novice sonographer.
Case B: Phantom Image Challenges The images in Case B were sourced from a phantom. In the first image (Figure left), models were asked to identify the structure within the red bounding box, which is the abdomen. However, many models were incorrect, selecting options like cerebellum,
8

---

Published as a conference paper at ICLR 2026
midbrain, and head. This suggests that the models were heavily influenced by the global context of the image (which includes prominent brain structures) and did not focus exclusively on the specified region of interest (ROI).
The final two examples (Figure 5B, center and right) further demonstrate the models' difficulties. Many failed identify the legs and the femur plane correctly. This poor performance be at- tributed to several factors: the structures interest are small, the images are zoomed out, and the phantom images are significantly brighter than typical clinical ultrasounds. This brightness variation likely represents an out-of-distribution characteristic that the models were not adequately trained to handle.
In hhighe????
Beriege?
para
Trover ablenne
pide
shusben
Mokeo der
senancea Miney
hoading
#Seectors
beanding bex
Meletlew
#Sckebore
Minc renbiies
pane
Figure 5: Examples of challenging cases illustrating model failure modes. The figure shows model predictions for (A) three clinical and (B) three phantom ultrasound images. For each case, table displays the distribution of answers from 15 models, with the correct option shown in bold. These examples highlight common errors, such as confusion between similarly shaped structures and poor generalization to phantom images.
CONCLUSION
We introduce Fetal-Gauge. the first large-scale benchmark for evaluating Vision-Language Mod- els in fetal ultrasound interpretation, comprising 42,036 images and 93,451 crestion-answer pairs across five clinical tasks. Our systematic evaluation of 15 state-of-the-an VLMs reveals substantial limitations: the best- pess-rreroomin model achieves only 55% accuracy, far below clinical requirements.
Key findings highlight critical gaps in current VLM capabilities. Models struggle with fine- -ne-grained spatial reasoning, particularly for small anatomical structures, and show poor domain adaptation be- tween phantom and clinical images. While ultrasound-specife training data (as Lingshu models) improves performance. fundamental architectural limitations persist across all evaluated models.
Our benchmark establishes a foundation for developing specialized VLMs in prenatal care and re- veals urgent research priorities: ultrasound-specifie architectures, improved spatial reasoning, and robust domain adaptation strategies. The substantial performance gaps underscore both current lim- itations and opportunities for methodological innovation in medical multimodal DL. Fetal-Gauge Gauge provides the rigorous evaluation framework necessary for measuring progress toward clinically vi- able fetal ultrasound interpretation systems.

---

Published as a conference paper at ICLR 2026
THE USE OF LARGE LANGUAGE MODELS
During the preparation of this work, the authors used ChatGPT enhance writing. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.
REFERENCES
Mahmood Alzubaidi, Marco Agus, Michel Makhlouf, Fatima Anver, Khalid Alyafei, and Mowafa Househ. Large-scale annotation dataset for fetal head biometry in ultrasound images. Data in Brief, 51:109708, 2023.
Anitha. Ultrasound fetus dataset. http:://bb...rr//.11533////aaa2p
2024.
Data set.
Rima Arnaout, Lara Curran, Yili Zhao, Jami C Levine, Erin Chinn, and Anita J Moon-Grady, An ensemble of neural networks provides expert-level prenatal detection of complex congenital heart disease. Nature medicine, 27(5):882-88) 891, 2021.
S. Belciug. Fetal planes and organs. htttee://aai.rrr//002222///ebbb 2024. Data set.
Smaranda Belciug. 3vessels + gallbladder. 7323401 2022. Data set.
http::://b...rrg110.522//
Asma Ben Abacha, Sadid Hasan, Vivek V Datla, Dina Demner-Fushman, and Henning Müller. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Pro- ceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019, 2019.
Xavier P Burgos-Artizzu, David Coronado-Gutiéree Brenda Valenzuela-Alcara Elisenda Bonet- Carne, Elisenda Eixarch, Fatima Crispi, and Eduard Gratacós. Evaluation of deep convolutional neural networks for automatic classification of common maternal fetal ultrasound planes. Scien- lific Reports, 10(1): 10200, 2020.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip-llava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2024.
Chen Cui and Fajin Dong. n2rbrb9t 4f 2022.
Dataset for fetus framework. https://eo..org110..11
Karine Souza Da Correggio, Roberto Noya Galluzzo, Luis Otávio Santos, Felipe Soares Muy- laert Barroso, Thiago Zimmermann Loureiro Chaves, Alexandre Sherlley Casimiro Onofre, and Aldo von Wangenheim. Fetal abdominal structures segmentation dataset using ultrasonic images.
/tttp::/dbo..rgge 17 11tp:.......3////msssss........ 2023.
Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter Beller-Morales, al. Aya vision: Advancing the frontier of multilingual multimodality. arXiv preprint arXiv:2505.0877 008000 2025.
Database Earth. Annual population births by country in 2024, 2025. URL https:://aaaaaass Barth/populat: on/bi htwnsiitth/////ee rttss/0222
Accessed: 2025-08-16.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
F. Dulgheroff, A. B. Peixoto, C. G. Petrini, M. R. D. C. Caldas, D. R. Ramos, F. O. Magalhães, and E. Araujo Júnior. Fetal structural anomalies diagnosed during the first, second and third trimesters of pregnancy using ultrasonography: retrospective cohort study. Sao Paulo Medical Journal, 137(5):391-44)) 4000, Sep-Oet 2019. doi: 10.1590/1516-1188 011..000..0009990100 11..9.269900200999
10

---

Published as a conference paper at ICLR 2026
D. González, J. P. Barrientos, M. Perez, J. Fajardo. F. Reyna, and A. Lara. Natalia: Pbf-usl (phan- tom sweeds for fetal ultrasound scanning), 2024. URL htllhhttpp///oo org/ 5281/ [Data set].
zenodo 1 14193949
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arxiv 2021. arXiv preprint Xiv: 1et.:22060.9985. 10, 2021.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A large-scale comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22170-2211 2024.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Gemner-Fushmen generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
dataset of clinically
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vistt as- sistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 28541-28566 2023.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pp. 1650-1654. IEEE, 2021.
Fadillah Maani, Numan Saeed, Tausifa Saleem, Zaid Farooq, Hussain Alasmawi, Werner Dichl, Ameera Mohammad, Gareth Waring, Saudabi Valappi, Leanne Bricker, al. Fetalelip: A visual- language foundation model for fetal ultrasound image analysis. arXiv preprint arXiv:2502.1480) 14807, 2025.
OpenAL. GPT-5, htttp:://openai.com gpt- 2025. Large language model.
Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-rl: Incentivizing medical reasoning capability of vision-
language models (vlms) via reinforcement learning. arXiv preprint 15Xi:22502.19334 2025.
Bharath Srinivas Prabakaran, Paul Hamelmann, Erik Ostrowski, and Muhammad Shafique. Fpus23: an ultrasound fetus phantom dataset with deep neural network evaluations for fetus orientations, fetal planes, and anatomical features. IEEE Access, 1::8308- 1588388-83177,, 2023.
Mohammad Areeb Qazi, Mohammed Talha Alam, Ibrahim Almakky, Werner Gerhard Dichl, Leanne Bricker, and Mohammad Yaqub Multi-task learning approach for unified biometric estima- tion from fetal ultrasound anomaly scans. International Conference on Medical Imaging and Computer-Aided Diagnosis, pp. 52-61. Springer, 2023.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint
Carla Sendra-Balcells, Victor M Campello, Jordina Forrents-Barrent Yahya Ali Ahmed, Mustafa Elattar, Benard Ohene-Botwe, Pempho Nyangulu, William Stones, Mohammed Ammar, Lamya Nawal Benamer, et al. Generalisability of fetal ultrasound deep learning models to low- resource imaging settings in five african countries. Scientific reports, 13(1):2728, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL htlhhtta:///ee arxiv. org/ abs/24022
03300
Catalin Stoean, Nebojsa Bacanin, Wieslaw Paja, Ruxandra Stoean, Dominic Iliescu, Ciprian Patru, and Rodica Nagy. Semantic segmentation of fetal heart components in second trimester echocar- diography. Procedia Computer Science, 207:3085-3092 2022.

---

Published as a conference paper at ICLR 2026
Ruxandra Stoean, Dominic Iliescu, Catalin Stoean, Vlad llie, Ciprian Patru, Mircea Hotoleanu, Rodica Nagy, Dan Ruican, Rares Trocan, Andreea Marcu, et al. Deep learning for the detection of frames of interest in fetal heart assessment from first trimester ultrasound. In International Wlokk-Conferenee on Artificial Neural Networks, pp. 3-14. Springer, 2021.
Belay Susu, Kibir Temesgen, Sindu Ayalew, Selam Yibeltal, Tadele Emagneneh, Adem Yesuf, and Chalie Mulugeta. Prenatal ultrasound utilization and associated factors among pregnant women attending antenatal care in south wollo zone public hospitals, north east, ethiopia, 2023. Frontiers in Digital Health, 7:1547547, 2025.
Darya Taratynova, Alya Almsouti, Beknur Kalmakhanbet, Numan Saeed, and Mohammad Yaqub. Tpa: Temporal prompt alignment for fetal congenital heart defect classification, 2025. URL
sittps:/aaxat org/
httg//b////// /tt.///////// 15298
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing dision-languu model's perception the world at any resolution. arXiv preprint arXiv:2409.12 2024.
Daniel Won, James Walker, Russ Horowitz, Sandeep Bharadwaj. Edward Carlton, and Helena Gabriel. Sound the alarm: the sonographer shortage is echoing across healthcare. Journal of Ultrasound in Medicine, (2(7):1299-1333 2024.
Songxiong Wu, Hongyuan Zhang, Tingting Ye, Haoyu Xie, Ping Zeng, Qingjun Sun, Panying Wang, Bingsheng Huang, Lei Du, and Guangyao Wu. Focus: Four-chamber ultrassund image dataset for fetal cardiac biometric measurement. 2025.
52t1 14597550
Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in med- ical vision language models. Advances in Neural Information Processing Systems, 30:140334- 140365, 2024.
Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang. Cheng- hao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foun- dation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044, 2025.
Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, Xiang Wan, Benyou Wang, and Haizhou Li. Hu- atuoGPT, towards taming language model to be doctor. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10859-10885. Singapore, December 2023a. Association for Computational Linguistics. doi: 10.11653/112203, 1tn.::53../222......nnmmmmmmm findings-emnlp7255 7255. URL https: http://aaaaaatnttaaaeee org.lo/33...gg/20..... Findings-emn 725/
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305. 10445, 2023b.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
12

---

Published as a conference paper at ICLR 2026
7 APPENDIX
Table 3: Dataset-wise distribution of images and corresponding questions across training and testing splits. The values represent the number of images, with the number of associated questions shown in parentheses.
Dataset Name
Train
Test
'Total
3-Vessel View Daraset Belcing
2022
OC0)
253(253)
253(253)
PASS Correggo
2024
1,265(4,983))
923(1,276)
1,58966,259)
(lst Trimesteri Steean
3,2233,223)
6901690)
3,913(3,913)
Bchoo
Trimester Stocas
28107556)
941250)
3,001)006)
Ietall Plases
Bargos-
2,908/2,908)
18187(2.187)
5,0855,0055
Ietal Plaaes Organs Belcing
2024
2,776(10,477)
09312,704
1,47
Fetus Head Tumor Aniiha
2024
1,420)14420)
249(249)
1,669116669
FOCUS
Wu ct
2025
250/750)
50(150)
3009000)
PPUS2
Prabokaran et
2023
15,248(31,43))
3,812(78857)
19,090(39,990)
Targe Fetal Head Biometry
Alouhuidi ctal
3023
2,833/8,665)
1,715(4,737)
4,047(10,002)
MFUUPSSendaaBBBeell et
2023
2171217)
233(233)
450.450)
NatallA Gonzilee es a
2024
000)
346(346)
346,3466
NT Scan Cui & Dung
2022
1,372(10.181)
312(936)
1.14(11.117)
Tital
31,292/71,963)
10,959(21,466)
42,0136/33,451)
Table 4: Summary of fetal ultrasound datasets with descriptions, annotation types, and annotated anatomical structures or planes.
Danasee
Descrigtion
Anactation Clavsification
Annotited Structures Labels
3- Vesesl View Ditasee
Ulmoound imnges of the 3 vessels & gallblasker of scoond trinester fetisos.
3-vessel & bladder
PASS
ultmaoond lotal abdomes (tens preguanciess with viannal segmenttttion of abdominal argans fannta, umbilical vein, stom- ach. liver) support prosital diagnoticss
Segmeentition
Abdominal nosta, introbopara: umbilical vein, stom- ash, liver area
Thimesser)
(12-14 GA, Doppler color) labeled by view. Contains 6,720 images across
sweep
Clasiifiatiion
Atrioventrcclar noow (4-chumber), (LVOT) Great vessels (RVOT), Arterial arches (3 vessell, plus "other"
four main cunling planes.
(2nd Trmmester)
sweeps fetuses 121-24
Segmeniztion
egg. septum (S), atrim (1 A), norta (ort) right ventricle (RV),
(RA),
across standard heart views. Fach frame is segmeuted with 11 candiac strictuee.
yieding 1,040
Plancs
Large screening dalasct (mid sccond Timeestrr from patients labeled
Classitication
Aldomen,, Brain thalamis cerebeilum, trans ventrccatr,,llaa) Fenur, Thoras, Cervix, Other
tals. images from
classes:
four fetal planes (Abdomen. Brain, Femur, Thura).. Cervis, and Other. (Brain images are further sub classified into three views
line grained analysis.)
Fotal Planes & Organs
2D scoms of fetal morphology They are divided into different
Segmentition Classifica-
11 fotal ulinsound planes (eg. bipuriotal head, ab- donistll heart) and 18 annotated structines [e.g. bladder, porta, kidaey, cerchellum).
view planes, and the organs are segmented.
lion
Petus
Ultrasound tetal that contains normal, besugn, and maiig- nani cases. Each image annotated the frame level with one of three diagnostic.
Segmentziion Classitica.
Fecal head
tion
ROCUS
4-chmmber fetal heart imsages (sacond Trimester) monual seg- mentarion hent thorax regions biometric messurement (o cardischeracic ratio).
Segmentation
Cardiac chumbers thoracic regions
FPUS23
Phantom letal
weeks GA, 15,725
Classitication
Diugnostic pluncs, uneeiationn anstomical land marks, bounding boses anatomy
oricatation, anstoetical features, and bounding box detection.
tasks: plane ideatifiationn,
(plans/elerratloo/eemmss) and Detection Seymentition
Large Fetal Head Binmetry
Hightres fosal head ullrmoocnn images annotated by experts for richms.
Fetal brain, cavum septim pallicidum (CSP), lateral
biometry. Used for training segmentaioonnnnen algo-
MFLPP
Screening
from African centers (low-resoures settings). labeled into
Classifiation
Abdomen, Brain, Fermer Thorax
Contains coutine second trimester
mon fotal planes (Abdomenn
Femur,
AasalAA
Phantom stans by non experts af 23 weeks GA. 19,007 trames from 90 free-hund videos (POCUS device) simolating low- resourve scans. "nurplane"),
Classification
Diparieal hend plane, Abdornnal plane, Heari plane. Spine plane, Femur plane, No-plane
frame labeled fetal plane (including
NF Scan
Sigittal ultrasound images (11-14 werks GA) for NT movern- plane classification key detection syo- drome screening).
Claviiiiation and Cijject Da-
Tholami, midbeain palate, 4th ventrille cistemn magna, nuchal translucency, tip, skin, nasal bone
13

---

Published as a conference paper at ICLR 2026
Table 5: Summary of datasets with gestational age, ultrasound machines, license, patient counts, and geographic distribution.
Gestational
#Patients
Geographic Distribation
3-Vesell Datose
Trimestor
Provided
CC 4.0
15
Remnnnia
University Emergency County Hospital Craiova
Acuion Vilsson 750, Philips
169
University Hospital Polydorn Frnani
EPQQ Elite
São Thingo, Flocinadpolis
Echo (1at Trimester)
1st Trimester
GE Voluson EL0; GE Valuson ES; GE Voluson
CCBY
326
Not Provided
B6
Echo (2nd Trimester)
2nd Trimester
Not Provided
CC 4,0
8
Not Provided
Fetal Planes
2nd 1ed Trimester
GF Voluson Fs: GF Voluson 58: GE Valuson
CCBBY 4.0
1,792
Spuin - Hospinal Clinic and Hospital Sont Iran
S10, Aloka.
Dén, Barcelona
Fetal Planes & Organs
2nd Trimester
GET G...Ori GR Volnso 730 Pio
CCBRY 4.0
215
Romaniia University Emergency County
Hospital Craiova
Fetus Head
Nor Provided
Not Provided
CC
Provided
Not Provided
POCUS
Trimester
Not Provided
BY
Not Provided
Not Provided
FPUS23
Trimester
Philips EPQQ 7
Not Provided
N/A
N/A
Large Fetal Head
2nd 3rd Trimester
GE Voluson ES: GE Voluson 730
CC 4,0
551
Netherhand Radboud University Medical
Bionetry
Nijnegon
MPUP
2nd Trimester
Medical Siemens, Idan Instrmment: Mindray:
CCBB
125
Egype; Algena;
Ghans
HD3 (POCUS)
Trimesser
1,119
Shenzhen People's Houpial
Table 6: Dataset composition across fetal ultrasound tasks.
Task
Acronym
# Samples
Visual Grounding of Anatomical Structures
VG
54,601
Plane Identification
20.131
Fetal Orientation Assessment
FO
15,113
Clinical View Conformity
VC
1,684
Clinical Diagnosis
CD
1,669
14


Reviews:
[{"review_id": "XehKigjRZm", "reviewer": "Reviewer_aVZ6", "summary": "This manuscript introduces Fetal-Gauge, a large, multi-task visual question answering (VQA) benchmark for assessing vision-language models (VLMs) on fetal ultrasound. The benchmark aggregates 13 public datasets into 42,036 images and 93,451 MCQ question–answer pairs spanning five clinically motivated tasks: Anatomical Plane Identification (PI), Clinical View Conformity (VC), Fetal Orientation (FO), Clinical Diagnosis (CD), and Visual Grounding (VG). The authors standardize labels and unify heterogeneous annotations (including converting segmentations to bounding-box-based grounding questions). They enforce patient-wise splits and include a phantom subset (~19k images).\n\nThe authors evaluate 15 VLMs (general and medical-specific; plus one commercial model) with accuracy as the main metric, reporting substantial headroom: the best overall model attains ~55% accuracy; most others hover near task-specific random baselines. Fine-tuning two open models markedly improves performance (up to ~85% overall), and analyses highlight weaknesses on phantom data and on small structure grounding.", "questions": "- Is a single-frame image enough for FO or CD tasks? Sonographers may rely on multi-view reasoning, temporal sweep, and 3D relationships.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an imperative need for a standardized fetal ultrasound VLM benchmark. \n- The benchmark’s size and breadth (five tasks that mirror real clinical workflows) are substantial and thoughtfully justified. The MCQ format prioritizes objective, scalable evaluation and curbs hallucinations.\n- Unifying disparate labels/annotations (including mask-to-box conversion) and vocabulary harmonization is valuable for reproducible evaluation across datasets.\n- Evaluating 15 models (medical and general) under a unified protocol, plus showing that domain-specific fine-tuning can unlock large gains, gives the community clear baselines and targets.\n- The phantom vs. clinical breakdown and bounding-box-size analysis offer actionable insights (e.g., small-structure failures).\n- Cleary writing and presentation.", "weaknesses": "- Aggregating 13 datasets raises questions about licenses and usage permissions. Please include a table listing the license, citation, and redistribution policy for each source.\n- Although multiple datasets/hospitals are used, please report scanner vendors, probe types, gestational age ranges, and geographic distribution to understand generalizability.\n- Converting masks to boxes and asking “what’s in the box?” evaluates recognition, not localization fidelity. Consider adding a pointing game (models output coordinates of bounding box).\n- “normal/benign/malignant” in fetal ultrasound is non-trivial; please clarify labeling sources, consensus rules, and whether images are frame-level vs case-level labels.\n- Several source datasets include video sweeps; current tasks are frame-centric. Adding temporal subtasks would align with real ultrasound operation and may favor models with spatiotemporal reasoning.\n- Multiple typos. Section 4.4 seems to be incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TskajmDkZe", "reviewer": "Reviewer_ghap", "summary": "This paper presents **Fetal-Gauge**, the first benchmark specifically designed for evaluating **Vision-Language Models (VLMs)** in fetal ultrasound imaging. The benchmark consists of **42,000 ultrasound images** and **93,000 question–answer pairs**, covering five clinically relevant tasks:\n\n1. Anatomical plane identification  \n2. Visual grounding of key anatomical structures  \n3. Fetal position and orientation assessment  \n4. Clinical view conformity evaluation  \n5. Clinical diagnosis question answering  \n\nThe authors evaluate **15 representative VLMs**, including general-purpose models (e.g., GPT-4V, Gemini, LLaVA) and medical-specific ones (e.g., Med-Flamingo, BioVLM, PMC-VLM). The results show that the best-performing model achieves only **55% accuracy**, which is far below clinical requirements. Through analyses of error types, modality correlation, and task difficulty, the authors reveal major limitations of current multimodal models in handling the complex visual characteristics of fetal ultrasound—such as high noise, frequent occlusion, and operator dependency. This work aims to promote the development of **domain-adapted VLMs for medical imaging** through standardized datasets and evaluation protocols.", "questions": "1. Please clarify whether the data are collected from **real patients**. If so, provide IRB approval identifiers, the informed consent procedure (prospective/retrospective/waiver), de-identification process, and cross-institutional data-use agreements. (If the data are entirely from public sources, this is not required.)  \n2. The highest reported accuracy is only **55%**. Could you elaborate on the **clinical interpretability** and potential **risk-control mechanisms** associated with such performance levels?  \n3. Do the authors plan to build **domain-adapted models** on top of this benchmark (e.g., through medical text corpus fine-tuning, image–text alignment enhancement, or multimodal RLHF)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work establishes the **first large-scale benchmark** for fetal ultrasound with multi-task and multi-question–answer settings, covering the entire pipeline from basic recognition to diagnostic reasoning.  \n2. The dataset contains **42k images and 93k QA pairs**, each accompanied by standardized labeling schemes and metadata for ultrasound view types, ensuring clear task hierarchy and structure.  \n3. It evaluates both **general-purpose and medical-specific VLMs** under a unified QA interface, ensuring fair comparison across models.  \n4. The authors summarize **failure modes** of current multimodal models—such as boundary ambiguity, structural occlusion, and linguistic ambiguity—which provide valuable insights for developing safer and more reliable medical VLMs.  \n5. If the dataset and codes are fully released as planned, it will likely become an **important evaluation standard** for future medical VLM research.", "weaknesses": "1. The paper does not clarify the **data origin**, **IRB approval numbers**, **patient consent process**, or **de-identification pipeline** for the 42k images. Since obstetric data involve pregnant women and fetuses, the authors should describe the ethical approvals and data-sharing protocols either in the main text or the appendix.  \n2. The benchmark currently lacks **structured diagnostic**, **multi-view fusion**, and **temporal decision-making** tasks, which limits its representativeness for real-world AI-assisted prenatal screening. It is unclear whether the authors plan to include these components.  \n3. There is no quantitative analysis of **error sources** (e.g., modality, gestational stage, or device variation), nor is there an evaluation of confidence calibration, output stability, or safety metrics.  \n4. The **clinical evaluation metrics** are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GUQxxMNzbd", "reviewer": "Reviewer_4LjX", "summary": "This paper contributes Fetal-Gauge, a large-scale VQA benchmark for fetal ultrasound tasks, by integrating and processing 13 existing datasets. The authors evaluated 15 SOTA VLMs (including general-purpose and medical-specific models) on Fetal-Gauge. The primary finding is that even the best-performing closed-source model (GPT-5) achieved only 55% overall accuracy, while most open-source models performed near random chance. This reveals a substantial performance gap for current VLMs in this highly specialized domain.", "questions": "1. Figure 3 shows the distribution of the five task types per anatomical region. Providing an overall distribution of the five task types would seem more intuitive. Could you provide this? \n\n2. How are the incorrect options (distractors) for the MCQs generated? Are they sampled from an answer pool internal to the source dataset, or are they sampled globally from an answer pool spanning all datasets?\n\n3. For a given MCQ question, is the order of the options fixed, or is it randomized?\n\n4. From which source dataset(s) was the 'Clinical View Conformity (VC)' task derived? \n\n5. Is the content of Section 4.4 incomplete? The text appears to cut off. \n\n6. In Figure 3 and Table 1, should the abbreviation 'SV' be 'VC'?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper aggregates and standardizes 13 heterogeneous datasets from different sources, which have varying formats and annotations, into a unified VQA benchmark. It demonstrates a thoughtful design in data partitioning, such as preserving original dataset splits and allocating some datasets entirely to the test set to evaluate generalization.\n\n2. The paper's experimental analysis provides valuable preliminary insights, such as the substantial performance gap of existing VLMs, the critical importance of domain adaptation for future research, and specific model limitations, including poor performance on small anatomical structures and inadequate generalization to phantom images.", "weaknesses": "1. The paper's contribution is primarily an aggregation of existing datasets. The described method of 'Task and Annotation Unification' (Sec 3.2), which converts various annotation types (e.g., image-level labels, segmentation masks, and bounding box annotations) into a unified VQA format, does not demonstrate significant novelty or innovation compared to established data curation practices in the medical imaging domain.\n\n2. The 'Vocabulary Normalization' operation (Sec 3.2) is presented as a potentially important methodological contribution. However, this section is described too briefly, making it difficult to assess its impact. It is unclear which specific datasets are affected and how their vocabularies are unified (e.g., are answer vocabularies from different datasets merged?). Concrete examples would be necessary for comprehension. Furthermore, given that the benchmark uses an MCQ format where all answers are provided as options, the necessity of this normalization step is uncertain. An ablation study is required to empirically validate the importance of this operation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)