Structure is Supervision: Multiview Masked Autoencoders for Radiology
Authors: Sonia Laguna
Andrea Agostini
Alain Ryser
Samuel Ruiperez-Campillo
Irene Cannistraci
Moritz Vandenhirtz
Stephan Mandt
Nicolas Deperrois
Farhad Nooralahzadeh
Michael Krauthammer
Thomas M. Sutter
Julia E Vogt
Venue: Accepted by TMLR ()

Abstract: Building robust medical machine learning systems requires pretraining strategies that exploit the intrinsic structure present in clinical data.
We introduce Multiview Masked Autoencoder (MVMAE), a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations. MVMAE combines masked image reconstruction with cross-view alignment, transforming clinical redundancy across projections into a powerful self-supervisory signal. We further extend this approach with MVMAE-V2T, which incorporates radiology reports as an auxiliary text-based learning signal to enhance semantic grounding while preserving fully vision-based inference. Evaluated on a downstream disease classification task on three large-scale public datasets, MIMIC-CXR, CheXpert, and PadChest, MVMAE consistently outperforms supervised and vision–language baselines. Furthermore, MVMAE-V2T provides additional gains, particularly in low-label regimes where structured textual supervision is most beneficial. Together, these results establish the importance of structural and textual supervision as complementary paths toward scalable, clinically grounded medical foundation models.

Published in Transactions on Machine Learning Research (03/2026)
Structure is Supervision:
Multiview Masked Autoencoders for Radiology
Sonia Laguna"
Alaguna@ing etff.ethh.cc ch anandrea @inf. wti.fe.h..cc
Andrea Agostini"
Alain Ryser
Samuel Ruiperez-Camplln
Irene Cannistraci
Moritz Vandenhirtz
Department of Computer Science, ETH Zurich
Stephan Mandt
Department of Computer Science, UC Irvine
Nicolas Deperrois
Farhad Nooralahzadeh
Michael Krauthammer
Department of Quantitative Biomedicine, University of Zurich
Thomas M. Sutter'
Julia E. Vogt Department of Computer Science, ETH Zurich
Reviewed OpenReview: https: openreview. per/fomm?? ud=ury
Abstract
Building robust medical machine learning systems requires pretraining strategies that exploit the intrinsic structure present in clinical data. We introduce Multiview Masked Autoencoder (MVMAE), self-supervised framework that leverages the natural mulli-view organization of radiology studies learn riew-inaariant and disease-relevant representations. MVMAE combines masked image roconstruction with cross-view alignment, transforming clinical re- dundancy across projections into powerful self-supervisory signal. We further extend this approach with MVMAF-V2T, which incorporates radiology reports as an auxiliary text- based learning signal to enhance semantic grounding while preserving fully vision-based inference. Evaluated on downstrcam discase classification task on three large-scale public datasets, MIMIC-CXR, CheXpert, and PadChest, MVMAE consistently outperforms su- pervised and vision language baselines. Furthermore, MVMAE-V21 provides additional gains, particularly in low-label regimes where structured textual supervision is most benefi- cial. Together, these results establish the importance of structural and textual supervision as complementary paths toward scalable, clinically grounded medical foundation models.
Introduction
Foundation models have accelerated progress in artificial intelligence for medicine (Moor et al. et al.] 2024) and pathology (Chen et al.] 2024a Lu al.] (202)). Yet, the data rcalitics in healtheare dif- ler sharply from the internet-scale corpora that power general-purpose vision language systems. Within
2023 Pai
the radiology domain, chest radiography datasets such MIMIC-CXL (Johnson et al. 2019a). CheX-
pert [Chambon et al. 2024), Chest X-ray Wang et al. (2017), and PadChest (Bustos et al.
2020) comprise
hundreds of thousands not billions-of studies, labels are expensive and often noisy (Gündel et al. 2021
and Institutions acquisition protocols vary widely (Hassanzadeh et al. 2018 Chambon el al. 2024). As a
result, purely supervised pipelines struggle to remain performant, and generic vision-lange laiiual pretraining may underperform or require costly adaptation to domain benchmarks (Chaves al. 2024). Sclf-supervisod
*These authors contributed equally. iShared senior authorship.

---

Published in Transactions on Machine Learning Research (03/2026)
learning (SSL) offers path forward by reducing reliance expert labels (Radford et al. 2021 Tiu et al, 2022b; Azizi et al. 2021). Compared to natural images, the data generation process for radiographs defines a clear structure: one or multiple scans, along with a corresponding radiology report, constitute a radiology study. Novertheless, many medical SSL methods still treat radiographs of the same study as independent
images (Ghesu el al. 2022 Tiu et al. 2022a), disregarding the structure of clinical exams.
In practice, radiology studies are structured: they frequently contain multiple image projections (e.g., frontal or lateral), repeated acquisitions, and an accompanying report. This structure encodes strong geometric (anatomical and physiological) and semantic coherence across views that radiologists routinely exploit for interpretation. Prior work has leveraged portions of such structures, showing improvements in robustness and label efficiency (Mo & Liang] 2024 Pellegrini et al. 2025 Chen et al.) 2024b). However, reconstruction-only objectives are view-specific and ignore cross-view view correspondence (Xiao et al. 2023 ; contrastive objectives
ignore roconstruction quality (Nguyen et al. 2022 ; and vision-language approaches may depend on report availability and quality, emphasizing abstract, label-driven semantics rather than detailed spatial or visual correspondence within images (Pellegrini et. al. 2025 Chen et al. 2024b). A scalable SSL pretraining recipe for chest radiography should harness both local visual detail and study-level consistency.
Following this intuition, we propose Multiview Masked Autoencoder (MVMAE), study-centric pretraining framework that couples masked reconstruction with cross-view alignment. to learn view-invariant representa- tions without relying on external supervision, as illustrated in Figure I MVMAE treats the different image projections acquired within the same study as natural positive pairs and jointly optimizes: (i) a masked image modeling objective per view to capture the complementary information from multiple X-ray views acquired during a patient examination; and (ii) a study-level alignment objective to regularize embeddings across projections of the same medical study. This converts clinical redundancy into self-supervision. Re- flecting clinical practice, we adopt a study-level organization and a unified policy that retains single and multi-view exams, incorporates uncertain views, and aggregates multiple projections within a study. This approach avoids over-counting, preserves the internal structure radiologists actually use, and allows us to probe how performance scales with the nunber of available projections al test time. Although we focus on chest-X-rays, the same strategy could extend to any life-science domain that offers repeated or complemen- tary scans, such as longitudinal magnetic resonance imaging (MRT) multi-sequence computed tomography (CT). Additionally, no open dataset fully captures alone the variability of chest radiography (Zech et al., 2018). Institutional protocols, patient demographics, and acquisition devices differ substantially weew collections such as MIMIC-CXR, CheXpert, Chest X-Ray, and PadChest. It is therefore crucial to pretrain jointly on all four datasets, allowing the model to internalize the variability inherent to clinical imaging and produce representations that transfer effectively across institutions.
To further inject semantic grounding when reports are available, we introduce MVMAE-V2T, which aug- ments MVMAE with lightweight vision-to-text objective. Reports are used only during pretraining as an auxiliary signal in addition to imsge-based objectives; at test time, the model remains vision-only encoder. This design contrasts with CLIP-style (Zhang et al. 2025) or instruction-tuned radiology Vision-Language Models (VLMs) Chen et al. 2024b Deperrois et al. 2025 that base their objectives on image-text alignment
or text generation only.
Contributions Our work advances multimodal self-supervised learning for radiology along four main directions: (i) We introduce MVMAE, the first multi-view masked autoencoder for radiology that adapts established masked reconstruction and cross-view alignment paradigms to the structure of medical data, yielding view-invariant yet detail-rich visual representations and consistently outperforming supervised and vision language baselines, while providing better calibrated predictions. (ii) We extend this framework into V2T, which incorporates radiology reports auxiliary supervision during pretraining. The additional vision-to-text signal enhances semantic grounding and offers clear gains under limited labeled data, without requiring text at inference. (iii) We adopt a clinically realistic, study-centric evaluation policy that mirrors real-world diagnostic practice and allows analysis of how cross-view consistency scales with the number of projections. (iv) We integrate three large public datasets: MIMIC-CXR, CheXpert Plus, and PadChest, into a unified benchmark, demonstrating consistent improvements across institutions and analyzing the per-label behavior of our models. Together, these results establish MVMAE and MVMAE- V2T as a simple yet effective blueprint for scalable, structure-aware medical foundation models.
2

---

Published in Transactions on Machine Learning Research (03/2026)
(a) Pretraining
(b) Fineturing
Rec
fn
9
fo
Latign
V2T
Generated
H
Text
9
fo
Input Modalities
Generated Text
V2T
<>>
Radiology
LGR
Text Report
Target Modality
Figure 1: Framework overview. (a) Pretraining and (b) Finetuning stages of the proposed MVMAE and MVMAE-V2T frameworks. Each study includes multiple views processed jointly through masked reconstruc- tion and cross-view aligument losses. MVMAE- V2T additionally incorporates vision-to-text objective.
2 Related Work
Self-supervised pretraining for medical imaging. SSL has emerged as key strategy for medical image representation learning, where expert annotation is costly and label distributions are imbalanced. Early works adapted general-purpose pretext tasks, such as context restoration or rotation prediction (Zhuang et al. 2019) to radiological images, followed by contrastive frameworks like SimCLR (Chen al.] 2020) and MoCo (He et al. 2020),, which encourage invariance to data augmentations. Applied to chest radiographs, these methods demonstrated that large unlabeled datasets such MIMIC-CXR (Johnson et al. 2019b)
could yield strong feature encoders transferable to downstream classification and localization tasks (Azizi ct al. 2021 Tiu et al. 2022b). However, these approaches typically treat each image as an independent instance, disregarding the inherent relational structure of radiology studies, such as co-acquired frontal and lateral projections. As a result, while they succeed in learning general visual semantics, they fail to exploit the intra-study coherence and anatomical consistency that characterize radiological data.
Masked and reconstruction--asse pretraining. Masked image modeling has recently become the dominant paradigm in visual self-supervision. Masked Autoencoders (MAE) (He et al. 2022) learn dense representations by reconstructing masked patches of the input image, and domain-specific adaptations have been proposed for radiology (Xiao et al.] 2023 Mo Liang. 2024). These studies emphasize that optimal masking ratios and roconstruction targets differ from natural-image setups, and that subtle medical textures require careful tuning of pretext objectives. While MAEs capture fine-grained image statistics and low-level anatomy, their reconstruction targets remain view-specific and intra-image only. In the context of CXRs, they ignore rich source of information: the geometric correspondence between frontal and lateral views of the same exam. In practice, each radiological study provides paired observations of the same anatomy under complementary projections. Yet, existing MAE pretraining works treat them as unrelated samples, forgoing a powerful form of implicit supervision.
Multimodal and vision-language pretraining. Paired image report datasets have enabled cross- modal learning between radiographs and text. VLMs, such as McdCLIP, CheXagent, and RaDialog (Boccking et al. 2022 Pellegrini et al. 2025 Chen et al. 2024b; Deperrois el al. 2025), align visual and textual representations through contrastive or generative objectives, yielding transferable multimodal features. However, these approaches rely heavily on the availability and quality reports, which vary across institutions, and the supervision they provide is semantic rather than geometric. In other words, text describes what is seen, but not how distinct radiographic views of the same patient relate anatomically to another.
3

---

Published in Transactions on Machine Learning Research (03/2026)
Cross-view and longitudinal pretraining. A smaller number of studics have begun to explore the structural redundancy in radiology data. Multi-view or correspondence-based contrastive methods NNguyen et al. 2022 Zhou et al., 2023 Zeng et al. 2023 Agostini et al., 2025) exploit spatial and anatomical consis- tency across projections or timepoints, improving representation robustness and patient-level genoralization. Still, these works remain limited either to contrastive pairing or simple view averaging; they do not yet integrate these constraints with reconstruction-baseed learning, nor do they exploit the joint information flow between modalities at scale.
In summary, prior medical pretraining approaches have successfully adapted sulf-super and multimodal objectives to radiological data but have largely abstracted away the structured nature of clinical imaging studies. Most treat radiographs as independent images, neglecting that each study inherently couples multi- ple views, repeated acquisitions, and textual context into a semantically and geometrically coherent whole.
Our work builds on this obscrvation by explicitly loveraging the structure of chest radiography, paired frontal lateral projections, a sell-supervisory signal. In contrast to previous view-agnostic reconstruction repert-dependent alignment, our method couples masked reconstruction with cross- view regularization and vision-to-text reconstruction, yielding representations that are both detail-preserving and view-invariant. This approach moves toward domain-adapted foundation models that learn not just from more data but from the structure of the data itself.
3 Dataset and Preprocessing
MIMIC-CXR, CheXpert, PadChest, and Chest X-ray as multimodal resources. We conduct the experiments on four large-scale publicly available chest radiograph archives: MIMIC-CRR (John- son et al.) 2019b), CheXpert Plus (Chambon al.) 2024), PadChest (Bustos et al. 2020 and Chest X-ray (Wang et al., 2017). MIMIC-CXR contains over 377,000 images from 227,000 studies linked to 65,000 patients. CheXpert Plus is extension of the original CheXpert datasct, which includes more reliable labels, reports, and additional metadata, comprising over 224,000 images from 65,000 patients. PadChest includes approximately 110,000 studies comprising around 160,000 im- ages from roughly 67,000 patients. For a detailed overview of the datasots, please refer to TTall 5) in Appendix A. All inchude routine chest radiographs and reports acquired critical-care settings.
Frental
Lateral
Unknown
Study structure. These three corpora are organized at the study level, where each study groups together one or more ra- diographic projections associated with a single radiology re-
port. Image quality varies substantially across datasets due to differences in patient positioning, acquisition devices, and clinical conditions, providing a realistic testbed for robust pre- training. Finally, the Chest X-ray dataset comprises roughly 109,000 frontal-view images from approximately 31,000 pa- tients. Since the original text reports are not publicly released and the label ontology differs from those of the other datasets, we use it only for unsupervised pretraining, rather than for downstream classification or evaluation. This setup broadens visual diversity during pretraining while preserving label con- sistency across evaluation datasets.
Text Reports
Label
Edema
Jobs obelectania iiblly. The right othorvine claar. ws uppascialle peounthooes is owen. Biopiced muzgin....
Fracture
Labeler
Lung Lesion
Preumonia
Figure 2: Study-level instance in MIMIC-CXR. The first row shows three different views in the same study, with their corresponding report and final labels.
We treat a study, the complete set of projections acquired during a patient encounter, as the fundamental unit of analysis. Projections are grouped inlo two main families: fronial (posterior anterior PA or ante- rior- poo-poottiior AP), and lateral lleft-lateral LL or generic lateral), with a subset labeled as unknoun, i.e., undefined in the metadata or view categories of low prevalence. To capture their multiview nature, we adopt a unified FLU (Frontal- valid projection, indexing each available view as E L. Formally, each dataset yields collection x()) = { (1) Jveyao' with yit) (uI,...,PA,0) denoting the set of view types present in study n0) 2 1 being the number of views per study (i). Please note that we allow multiple instances of the same view
policy that includes all studies containing at least one
4

---

Published in Transactions on Machine Learning Research (03/2026)
type This ensures inclusivity: single-view studies are retained, multi-view studies contribute all available projections, and uncertain view types are incorporated rather than discarded. When available, reports are also associated with each study and used to derive the final disease prediction labels. An illustrative example of what one study unit comprises, including its multiple projections, corresponding report, and final label set, is shown in Figure 2 Extended information about the dataset distribution, illustrations, and study nature can be found in Appendix A
Preprocessing pipeline and data split. All radiographs are resized to 224 x 224. For MIMIC-CXR
and CheXpert, study-level labels are inherited from the CheXpert labeler (Irvin et al. 2019) covering 14 diagnostic categories. Following Haque et al. (2023), the three non-positive states ("negative", "not mentioned", "uncertain") are collapsed into single O-class, treating only explicit positives as 1. In MIMIC- CXR and CheXpert, we use the official training, validation, and test splits. Since PadChest and Chest X-ray do not provide predefined splits, we create them manually, stratifying by patient to prevent leakage and to obtain data-split ratio comparable to MIMIC-CXR and CheXpert. For PadChest, we allocate 97% of studies to training and split the remaining 3% evenly between validation and lest. For Chest X-ray, we combine the original training and validation sets and use 98% for training and 2% for validation, while keeping the official test set unchanged. For PadChest, additional preprocessing was required. The dataset originally includes 193 labels, spanning radiographic findings, differential diagnoses, and anatomical locations. To ensure comparability, we mapped these labels into the same 14-category space defined by the CheXpert labeler. This mapping was constructed automatically with the assistance of Claude Code 2.0.35 (Anthropic, 2025) to identify the closest and most clinically consistent correspondences between PadChest labels and the CheXpen label set. This harmonization enables a unified evaluation across all three datasets. Additional details on the datasets and preprocessing, as well as statisties on the labels and other relevant information, are provided in Appendix A.
4 Methods
We study chest X-ray representation learning across multiple pretraining regimes, with focus on the Multi- view Masked Autoencoders, MVMAEE which combines masked reconstruction with a cross-view alignment loss to exploit study-level structure, and can be extended with text supervision, resulting in MVMAE- V2T. An overview of our approach shown in Figure I For evaluation purposes, we investigate ablated variants of MVMAE and state-of-tho-art vision-language models pretrained on broader medical data.
4.1 MVMAE: Multiview MAE
MAFs He et al. 2022 are self-supervised learning approach designed to learn high-quality representations by reconstructing missing portions of the input. MAEs randomly mask a large fraction of the input data and train an encoder-decode architecture to recover the masked content from the visible subset. In this work, we assume the encoder and decoder to be vision transformers (ViTs, Dosovitskiy et al. 2021).
M() applies a random mask to the input, where a is the masking ratio, and defines the number of patches removed from the input. We define the input as a sequence of patch tokens, where each token corresponds to fixed-size image patch that is linearly embedded into vector representation. We define the unmasked input tokens as Dovi = M
(6)
Let Tvia S {L...., T) denote the set of indices corresponding the visible (unmasked) tokens after applying M(.. The number of unmasked tokens is therefore given by Tvis = |Tvisl = (1 - aa) T, where T is the total number of input tokens. The encoder fo processes only the unmasked input tokens (1) producing latent representations z fo f= These are passed to lightweight decoder 9s, which adds mask token placeholders and attempts to reconstruct the original input æ., including the masked parts. The model is trained by minimizing a reconstruction loss LRee over only the masked positions:
1
Σ VED
Σ 2 Te
2000 1
where
a!00 (z()
Roz
D
The code available at: https://githhnn http://hwww...co.ooonii/33//////
5

---

Published in Transactions on Machine Learning Research (03/2026)
Here, Xe, denoles the original input view v al masked position i, and &e. the reconstruction of the same imput position t. Please note that the reconstruction loss is only computed over masked imput patches, but we feed only the visible or unmasked patches x M(x!")) For more details, we refer to He al. (2022).
(
In addition to the reconstruction loss, introduce a study-level alignment term to encourage consistent representations across all views in the same study (Agostini et al,, 2024 Sutter et al. 2024). Let p
(Ui...., y denote the set of available views for study i, and let 2u be the encoder output (CLS and visible tokens only) for view u. We define the alignment objoctive as the average pairwise distance between encoder outputs for all ordered view pairs:
( Align
1 I-
Σ
Σ
dMSE (1) z0 >!!))
(1)
[P()]
(N.))E(()) TETvin
where PD() v(,,v) | u E vio, u * v), and dMse(::) is the mean squared error (MSE). Importantly, alignment is computed only over visible encoder tokens; masked tokens are introduced later in the decoder. The full objective for MVMAE over study i is then:
c =
( Rec
+ B Allign:
(2)
where B is the weighting between reconstruction and alignment loss. We additionally apply beta annealing schedule, progressively increasing the alignment weight 3 from zero to its target value over training to stabilize carly-stage learning.
As introdeeed in the previous section, clinical studies may contain multiple radiographic projections, in- cluding repeated acquisitions of the same view type (e.g., two frontal views or a frontal lateral frontal sequence). To make the encoder aware of the projection type without assigning a separate head per view instance, we introduce learnable modality embedding em RD for each projection category m E (frontal, lateral, unknown]. Thus, all occurrences of frontal view share the same embedding efrontal: even if multiple frontal images appear within the same study.
Let x denote the w-th view of study i with associated modality index m(v). After patchification, we add the regular positional encoding, and additionally shift every visible patch token by the corresponding modality embedding:
h = p pos + em(o)) E Twiss
where p. is the regular VIT positional embedding. The same em(v) is also added to decoder tokens, ensuring that reconstructions remain conditioned the projection type. This design is analogous to segment enbeddings in language models (Devlin et al., 2019), allowing the model to represent projection-specife acquisition bias while keeping a shared encoder across all views. In practice, we use M = 3 modality embeddings and learn the table [= jointly with all other parameters.
4.2 MVMAE-V2T: Text as Additional Learning Signal
In addition to purely imago-based pretraining, we investigate the use of radiology reports as an auxiliary supervision signal, rather than as an input modality, and we introduce MVMAE-Vision to Text (MVMAE- V2T). Each report is paired with its corresponding imaging study and serves only to guide visual repre- sentation learning during training. The text is never used as input inference time during downstream evaluation, ensuring fair comparison with purely vision-based models. To incorporate this weak multimodal supervision, we adopt captioning-style objective inspired by the CapPa. framework (Tschannen et al. 2023)
Specifically, for each view (.(0) of a study and its study report y.(0 (11
(2)
(8) KG) with K() tokens, we
(1) conditioned on
extend MVMAE with a transformer decoder by that causally predicts each report token
all previous report tokens y (1)
<kk as well the encodings of the unmasked visual embeddings of the current
6

---

Published in Transactions on Machine Learning Research (03/2026)
view (0) via cross-altention. To train hw, apply a cross-entropy loss in auloregressive fashion as follows:
K(
C
Σ VEVO
Lce
r))
(4)
KO
where F(1) hw (1 (x
E IR M are the logits of the k-th token given the previous tokens rzk and the unmasked view embeddings =! W the token vocabulary, To is padding token and
exp( lce (x:))) llog Ewew exp(xw))
(5)
We add this additional loss on top of the standard MVMAE loss and train MVMAE-V22 using
C - Llt B. Align + 7 P-LCE
LCE-
(6)
Where y the weighting term for the cross-entropy loss. This joint optimization ensures that complementary information from multiple projections contributes coherently to the text generation pretext, reinforcing the study-level structure inherent in clinical data. Our approach leverages radiology reports solely structured, domain-specific supervision to guide the visual encoder toward disease-relevant semantics, without relying text for downstream reasoning prediction.
Related vision language pretraining methods such as SigLIP 2 (Tschanne et al. 2025 also employ captioning-style objectives, but rely on symmetric image- text alignment and additional mechanisms such as distillation and global local consistency. In contrast, MVMAF-V2T treats text solely as an auxiliary supervision signal and deliberately avoids explicit image text alignment, reflecting the clinical setting in which reports narrative descriptions of images rather than an independent modality.
5 Experiments
Our aim is to investigate how multi-view pretraining with MVMAE influences downstream disease classifica- tion across different datasets and under varying constraints. These settings reflect realistic climical conditions, where annotations are scarce, imaging protocols differ across institutions, and studies may contain variable numbers of views. We study four complementary questions:
(i) How effective are the representations learned by pretroined encoder when the model fully finetuned on downstream classification task? In particular, we assess whether leveraging the multiview and multimodal data structure during pretraining leads to measurable improvements. We compare our approach against supervised baselines, struclure-agnostie pretraining, and recent medical VLMs.
(ii) How robust are MVMAE representations under limited supervision? To simulate annotation scarcity, we finetune the pretrained encoder on progressively larger labeled subsets and compare against baselines, quantifying label efficiency.
(iii) Does leveraging the structure at pretraining help with model colibration? We evaluate whether multi-view pretraining leads to catter-calibrated predictions compared to baselines.
(iv) How does performance vary with the number of views per study? We ablate the maximum number of views available during evaluation to test the robustness of multi-view pretraining when only single view is provided at inference time, and the change when multiple views are available. To ensure a controlled comparison, we restrict this analysis to studies with exactly two available views.
Collectively, these experiments evaluate performance, label efficiency, and sensitivity to study structure.
5.1 Benchmark Models: From Unimodal Variants Foundation Models
To contextualize our results, we benchmark MVMAE and MVMAE-VE V2T against a spectrum of pretrained encoders that. represent different training regimes and levels of domain specialization. These baselines range

---

Published in Transactions on Machine Learning Research (03/2026)
MIMIC-CXR
CheXpert
PadChest
Atclortasis
Atelectanis
Support Devices
Cardiomegglly
Supgort Devicess
Cardionnegaly
Preumathorax
Consolidation
Pnetmochorax
Consolidation
3U7
p-7
Pnetmoría
Edema
Paeumooti
Eelemn
Pleural Othor
Enlarged C.
Pleura Other
Enlarged
Pleuzal Effusion
Fracttrre
Pleural Elfusson
Fracture
No Finding
Lang Lesion
Finding
Lung Lesion
Lang Oparity
Lung Opacity
(a) MVMAE
(b) MVMAE per label Atelectacis
Atelectasis
Support Devicus
Canlionegaly
Support Devices
Cardionegaly
Preumochorax
0.8
Consolidation
Parumeshurax
Consolidation
Paruitionia
Edemma
Paeuinsiia
Ederma
Pleura Other
Entluged
Pleural Other
Enlarged
Pleural Eifusion
Fracture
Pleural Echasion
Practame
No Finding
Lang Lesion
No Finding
Long Lesion
Lang Ouacity
Lung Opacity
(c) CheXagent
(d) CheXagent per label
Figure 3: Per-label AUROC comparison across datascts (MIMIC-CXR, CheXpert Plus, PadChest).. Left: overall performance with model selection done based the best joint dataset. Right: model selection done to optimize per label results.
from variations of our own approach that are purely unimodal or do not enforce cross-view regularization, to state-of-the-as assion-language models trained on large-scale biomedical corpora or chest-X-ray speccfic data. This progression allows us to test three hypotheses: (i) whether multi-view regularization is necessary beyond unimodal masked reconstruction, (ii) whether large but non X-ray-specific biomedical pretraining confers an advantage, and (iii) how our approach compares to foundation models explicitly optimized for chest radiography interpretation. The evaluated benchmarks described below.
Supervised. A fully supervised baseline that trains the backbone end-to-end on the multi-label classi- fication task directly, without any pretraining, masking, or alignment. This represents the conventional approach used in many radiology benchmarks and serves as control assess how much benefit comes from our self-supervised pretraining. The same vision encoder architecture as in MVMAE employed ensure fair and controlled comparison.
Independent. This ablation removes the cross-view aligmment regularizer by setting 3 = 0. Each projec- tion within a study is reconstructed independently, without explicitly encouraging shared latent representa- tions between frontal, lateral, or unknown views. This method tests whether performance gains stem from the alignment mechanism itself, or simply from masked reconstruction. The same vision encoder architecture in MVMAF is employed to ensure fair and controlled comparison.
8

---

Published in Transactions on Machine Learning Research (03/2026)
MVMAR MVMAF.V2
Independent
BiomedC.LIP
Supervised
0.86
0.86
0.82
0.82
AUROC
0.80
AUROC
0.80
0.78
0.78
0.76
0.78
0.74
0.72
0.72
x
10'
10
2 10
8x 10'
10º
5xx 10
10
2 101
3 10
10'
[4]
[4]
(a) Linear Probing
(b) Finetuning
Figure 4: Label efficiency under finetuning. Performance curves of the Combined datasct macro-average AUROC over 14 pathology labels, as function of the number of labeled studies used for finetuning.
CLIP Zhang et al. (2025) propose a large-scale biomedical VLM. It follows the CLIP (Radford ct al. 2021) framework, jointly training a ViT-based image encoder and a PubMedBERT (Gu et al.) 2021 text encoder on PMC-15M (Zhang el al.) 2025 & dataset of over 15 million biomedical image caption pairs mined from PubMed Central. This enables the model to generalize across a wide range of biomedical modalities, including radiology, pathology, microscopy, and illustrations. Compared to prior medical CLIP variants, it benefits from much larger and diverse pretraining corpus, higher input resolution, and domain- specific adaptations.
Cagentt Chen et al. 2024b) introduce a VLM for chest radiography. It is trained on CheXinstruct, triplets curated from 32 public sources, cov-
largo-scale dataset of 8.5 million instuutioo-imaa-rrsppon ering 35 distinct chest-X-ray interpretation tasks. The model combines a SigLIP-based image encoder with decoder-only language model. This design allows to perform wide spectrum of tasks, including view classification, disease identification, temporal reasoning, VQA, phrase grounding, or radiology report gen- eration. For fairness, we evaluate only the CheXagent vision encoder, not the full VLM, which itself was pretrained with self-supervised objectives before being integrated into the decoder-only architecture.
5.2 Implementation Details
For downstream classification, we use late-fusion ensemble that averages prediction scores from unimodal single-view models for both the proposed MVMAE and baselines. A single multi-label classifier is trained to predict the fourteen diagnostic labels. The classifier trained on the Combined dataset, obtained by merging studies from MIMIC-CXR, CheXpert, and PadChest. We report performance both on the Combined valida- tion set and separately each individual dataset. This setup enables a unified evaluation while still high- lighting gross-institutionn generalization. All models were trained with data augmentations common in self- supervised vision sctups, including random resized cropping, horizontal flipping, and color jitter, to improve representation quality. Our MVMAE backbone is ViT-Base (Dosovitskiy el. al. 2021) with 12 layers, 12 al- tention heads, and fi8-dimenliionl hidden embeddings, operating on 16x x 16 image patches. During pretrain- ing, we apply high masking ratio of 90%. The decoder lightweight transformer appended only during reconstruction. All experiments were conducted on an internal cluster using NVIDIA Grace Hopper Super- chips with 40GB memory per GPU. Further implementation and trainig details are included in Appendix B.

---

Published in Transactions on Machine Learning Research (03/2026)
5.3 Results
Overall performance across datasets. The first experiment evaluates the downstream classification performance of the proposed MVMAE compared to the baseline methods described in [Section 5.1 when trained on the full labeled Combined dataset. The MVMAE encoder is finetuned end-to-end on the Combined data, and the same finetuning procedure is applied to BiomedCLIP and to the independent baseline to enable a direct comparison under identical conditions. In contrast, CheXagent is evaluated under a linear-probing setting, where only a classification head is trained while the pretrained encoder remains frozen, as its encoder has already been optimized for chest X-ray interpretation during large-scale pretraining. Model selection is performed using the macro-average AUROC over the fourteen labels on the Combined evaluation set. Results are reported in [Tallss 1 and 2 showing that MVMAE achieves the highest performance overall, with strong generalization across institutions and outperforming both supervised and VLM basclines.
Table 1: Classification performance across datasets. Comparison of MVMAE and baseline methods on the full labeled datasets. The table reports AUROC and F1 (mean + std in three scods) averaged over the fourteen labels per dataset, along with combined metric averaging performance across all three. ViT-B/10 16 stands for VaTe architecture with 16 patch size, and ViT-L/14 for ViT-Large with 14 patch size.
MIMIC-CXR
CheXpert
PadChest
Combined
Model
Backbone
AUROC
F1
AUROC
P1
AUROC
AUROC
F1
Supervised
VIT-8/16
79.25
0.42
25.30
+ 0.36 76.39 : 0.85
23.77
0.66 85.72 + 0.28 27.16
+ 1.04
84.83 + 0.25 29.33
0.66
Independent
VIT-B/16
81.28
0.15
29.16
+ 1.19
79.65
: 1.67
30.58
3.05
88.17
1 0.29
33.22
: 0.22
86.60 : 0.04
35.48
0.18
BiomedCLIP
VIT-B/1G
78.88
0.88
23.03
#
76.72
+ 411
24.30
1.87
84.87
27.53
+ 3.51
83.97 + 0.78
29.02
CheXagent
VIT-L/14
50.85
0.13
63.02
0.22
33.54
4.81
1
28.39
3.69
86.25 1
33.50
MVMAE-V2T (ours)
VIT-B/16
$1.20
0.18
80.87
0.35
35.96
9.35
86.27
37.00
6.74
86.74 4
38.90
MVMAE (ouns)
VIT-B/16
62.48
5.67
83.24
0.81
37.56
0.21
39.69
5.86
87.41
41.27
5.67
Table 2: Top-5 classification performance across datasets. Comparison of MVMAE and baseline methods the full labeled datasets. The table reports Top-5 (labels) average AUROC and F1 (mean + std over three seeds) per dataset, along with a combined metric averaging performance across all three.
MIMIC-CXR
CheXport
PadChest
Combined
Model
Backbone
AUROC
F1
AUROC
F1
AUROC
F1
AUROC
F1
Supervised
VIT-B/16
82.20
3.18
38.64
5.87
78.32 4
40.47
# 1.05
84.10
5.47
29.45
5.21
86.16
3.14
38.09
6.28
Independent
VST-B/16
66.27
1.66
50.18
3.68
46.17
1 4.82
89.55
3.13
3.29
86.99
1.90
50.60
3.61
Chesagent
VIT-L.114
87.30
0.98
52.28
1.28
88.14
0.52
65.30
3.72
92.10
0.50
40.74
2.14
00.64
0.48
53.37
0.92
MVMAE-VIN V2T (ours)
VII-B/16
89.10
0.15
07.83
4 1.61
86,76
0.41
54.14
# 1.63
91.53
0.58
43.38
0.34
91:65
0.05
57.47
# 0.94
MVMAE (ours)
VII-B/16
89.10
0.08 56.00
8.84
86.87
52.34 + 3.30
92.47 0.95
6.96
91.78
0.10
55.32
7.53
To better understand the distribution of performance across different conditions, Figure 3 visualizes the AUROC achieved per label in each dataset. The radar chart highlights substantial variation across discase categories and across datasets, reflecting both differences in label prevalence and the relative difficulty of certain findings (e.g., "Pneumonia" or "Pleural Other"). In our findings, labels corresponding to frequent and visually localized conditions yield higher scores, whereas labels for rare or textually ambiguous conditions remain challenging. We further observe that the relative difficulty of specific labels varies across datasets and models, suggesting that institutional biases, labeling practices, and acquisition protocols influence how well models generalize per pathology. In Figure 3 we show the MVMAE variant achieving the best overall classification performance across the Combined dataset, alongside the strongest baseline (CheXagent), and we include the remaining baselines in Appendix C Notably, even when using CheXpert, where CheXagent performs better in terms of Top-5 AUROC, MVMAE achieves comparable or superior results for several individual labels, e.g., pneumonia, atelactasis, and fracture. To isolate label-wise performance, we also report results using lab-l model selection, selecting the checkpoint that maximizes macro-average AUROC for each label independently (Figure 3. right). We note that part of the apparent per-label variation CheXpert arises from the strong imbalance of its official validation split. Certain conditions (e.g., Pneumonia and Pleural Other) are represented by only a single positive study in this subset, which artificially lowers
10

---

Published in Transactions on Machine Learning Research (03/2026)
AUROC values when using joint model selection. When model selection is instead performed on each label individually, as in Figure 3 b and d, these differences vanish, confirming that the observed gap is due to data scarcity rather than model limitations. Overall, the radar plots further reveal that MVMAE matches or exceeds CheXagent on difficult conditions.
Label efficiency. In Figure 11 we assess how well representations transfer under limited super- vision and label availability. Models are finet on increasing subsets of labeled studies (IL) E (5K, 10K, 20K, 50K, 100K3). This experiment evaluates how effectively each method benefits from increasing supervision, indicating how rapidly performance scales with available labels. Note that the labeled subsets are used only for downstream evaluation, i.e., linear probing or supervised finetuning, while all pretraining, including MVMAF, MVMAF-V2T, and baselines, remains strictly self-superviser ieeppeeeviedddeeeee with all samples. Be- cause our focus is on label-efficient transfer, we restrict baselines to methods that did not use labels during protraining, thereby excluding CheXagent. We benchmark two settings: full finctuning and linear probing. Under linear probing, MVMAE consistently outperforms all baselines, demonstrating more linearly separable representations and superior label efficiency. When finetuned, MVMAF-V2. further highlights the benefit of incorporating reports, especially in low-label regimes, although this vdvatagge diminishes as more labeled data becomes available. In practical terms, achieving approximately 80% AUROC requires 5k labeled sam- ples for MVMAF variants, 10k for the Independent baseline (2x more), and 50k for the Supervised baseline (10x more). These results indicate that MVMAE yiclds label-efficient representations, while MVMAE-V2T provides additional gains when finetuning under scarce label conditions.
Table 3: Brier score under linear probing on 20K samples on the same encoder-type models, lower is better.
Model
MIMIC-CXR
CheXpert
PadChest
Combined
Supervised
0,0885 + 0.0005
0.0955 + 0.0010
0.0718 + 0.0003
0.0815 + 0.0004
Independent
0.0915 = 0.0005
0.0903
0.0016
0.0750 + 0.0005
0.0845
0.0003
BioMedCLIP
0.0924 + 0.0052
0.0980 + 0.0061
0.0769 1 0.0053
0.0859 + 0.0053
MVMAE- V2T (ours)
0.0964 = 0.0015
0.1017 + 0.0014
0.0825 + 0.0022
0.0905
0.0018
MVMAE (ours)
0.0903 = 0.0006
0.0974
0.0008
0.0743 + 0.0008
0.0835
0.0007
Calibration. We evaluate probabilistic calibration using the Brier score, which measures the mean squared gap between predicted probabilities and outcomes, i.e. lower is better. As expected, the supervised classifier baseline reports the lowest Brier score, given its specialized setup with direct end-to-end classification. Under linear probing with 20K labeled samples and matched backboncs, MVMAE reports the lowest Brier score among the pretraining baselines on MIMIC-CXR and PadChest, as well as the best combined result (see Tabll 3). Improvements over Independent and BiomedCLIP are modest but consistent, indicating that multi-view pretraining not only improves discrinination but also produces better calibrated confidence estimates across datasets.
Effect of the number of views per study. We fur- ther investigate how the number of available projections per study influences model performance. To ensure a fair comparison when assessing the effect of adding an addi- tional view, we restrict the analysis to studies that contain exactly two views and finetune the models only on those. We compare the proposed MVMAE with its independent variant (trained without the cross-view alignment term) while varying the number of views included during eval- nation, from single-view to two-views configurations. As shown in Figure 5] where we report the average AAUROC over three random seeds, computed as the AUROC of MVMAE minus that of the independent; MVMAE con- sistently demonstrates a performance advantage over the independent variant. across both settings. On the Com-
1.47
view
1.4
1.2
AUROC
0.8
0.75
0.6
0.42
0.4
0.331
0.26
0.2
0.12
0000
0.0
MIMIC-CII
CheXpert
PodChest
Combined
Figure 5: Average AUROC performance gain of MVMAE on Independent over three soeds.
11

---

Published in Transactions on Machine Learning Research (03/2026)
bined dataset, the improvement is more pronounced under single-view evaluation, suggesting that multiview pretraining provides transferable benefits even when only one view is available. A similar trend is observed for CheXpert and PadChest, whereas for MIMIC-XRR, the improvement is noticeable only under two views.
Extended quantitative results from all our experiments, dataset-specific analyses, and additional visualiza- tions are provided in Appendix C for completeness.
6 Discussion
Our findings highlight the importance of exploiting the inherent structure medical imaging data for representation learning. Multi-view pretraining consistently improves downstream classification compared to training from scratch or using independent single-view objectives, confirming that structural coherence across projections provides strong supervisryy signal even without explicit labels or text.
Leveraging study-level structure during pretraining proves markedly more effective than enforcing consistency only at the supervision stage. Models that align views through soft regularization learn richer and more transferable representations than those combining predictions only at inference.
MVMAE also demonstrates superior label efficioncy, achieving high performance with significantly fewer annotations, crucial property in clinical domains where expert labeling is costly and often incomplete. Compared to large-scale rision-language models such as BiomedCT.IP and CheXagent, MVMAR achieves competitive better results without relying text during pretraining.
A particularly interesting finding of our study is that MVMAE-V2T does not consistently outperform the vision-only MVMAF, despite leveraging additional textual supervision during pretraining. In several fine- tuning settings with full supervision, MVMAE without text matches or exceeds the performance of its V2T counterpart. This outcome is counterintuitive, as incorporating text is often assumed to improve repre- sentation quality. Our results suggest that when downstream labels are abundant, vision modality alone may already provide sufficient semantic structure, leaving limited room for auxiliary textual guidance and potentially introducing additional optimization complexity.
In contrast, MVMAE-V21 exhibits advantages in label-scarce regimes, which are highly representative of clinical practice. After full finc-tuning, it reliably outperforms MVMAE in these settings, suggesting that radiology reports function as a semantic bridge that structures the latent space around clinically meaningful concepts and improves transfer under limited supervision.
These findings indicate that MVMAE-V2T should not be interpreted as universally superior alternative, but rather an extension of MVMAE, oplimized for data-scarce scenarios where textual supervision is available during pretraining.
We further find that MVMAE yiclds better-calibrated predictions among pretraining baselines, suggesting that cross-view regularization encourages more reliable confidence estimates alongside improved discrimi- nation. Finally, training and evaluating across multiple datasets reveal that multi-view structural learning scales with data diversity and supports cross-institude cosssinntitutional generalization.
Overall, these results highlight the importance of aligning pretraining objectives and evaluation protocols with the real-world organization of clinical data. Beyond chest X-rays, this saanework-encompassee both MVMAE and MVMAE-V2T naturally extends to temporal, multi-sequence, or multimodal studies, offering a scalable path toward clinically grounded foundation models built on structural and semantic supervision.
7 Conclusion
We introduced MVMAE, the first multi-view masked autoencoder that leverages the study-level structure of clinical imaging data to learn robust and transferable representations without relying on textual inputs during pretraining. By combining masked reconstruction with cross-view alignment, MVMAE effectively exploits the inherent multiview nature of radiology studies, yielding consistent improvements in performance and label efficiency across MIMIC-XR,, CheXpert, and PadChest. Building upon this foundation, we
12

---

Published in Transactions on Machine Learning Research (03/2026)
further proposed MVMAE-V2T, which integrates radiology reports as an auxiliary text-based learning signal. This extension enhances semantic grounding and boosts performance in low-label regimes after full fine- tuning,, while preserving fully vision-based inference. Together, MVMAE and MVMAE-V2T demonstrate that structural and textual supervision can act as complementary forces: structural alignment provides robustness and transferability, and text guidance enriches semantic understanding. Overall, these results establish a scalable framework for representation learning in medical imaging that leverages the inherent organization of clinical data. The same principles naturally extend other structured modalities, such temporal follow-ups multi-sequence imaging, paving the way toward clinically grounded foundation models built on structural and semantic coherence rather than annotation scale. More broadly, the framework generalizes beyond medicine to any multimodal domain where structured relationships between views or modalities can serve as sell-supervision for learning robust and transferable representations.
Acknowledgments
This work was supported under project ID a135 as part of the Swiss AI Initiative, through a grant from the ETH Domain and compulational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure, and by the LOOP Zurich part of the application driver project supporting the LOOP Zurich Biomedical Informatics Platform (BMIP). TS and AA are supported by the grant #2021-911 I of the Strategic Focal Area "Personalized Health and Related Technologies (PHRT)) of the ETH Domain (Swiss Federal Institutes of Technology). MV and SL. are supported by the Swiss State Secretariat for Education, Research, and Innovation (SERI) under contract number 11822.00047. AR is supported by the StimuLoop grant #1-007811-002 and the Vontobel Foundation. SM acknowledges support from NSF, TARPA, HPI, CZI, Qualcomm, and Disney. ND) and FN received research support from the Digitalization Initiative of the Zurich Higher Education Institutions (DIZH)- Rapid Action Call under TRUST-RAD project. MK is supported by the UZH Global Strategy and Partnerships Funding Scheme and a Research Partnership Grant with China, Japan, South Korea and the ASEAN region (RPG 072023 18).
References
Andrea Agostini, Daphné Chopard, Yang Meng, Norbert Fortin, Babak Shahbaba, Stephan Mandt, Thomas M Sutter, and Julia E Vogt. Weakly-Supervised Multimodal Learning MIMIC-XR. Ma- chine Learning for Health symposium (ML4H), 2024.
Andrea Agostini, Sonia Laguna, Alain Ryser, Samuel Camppilo,,,,,,lo,, Moritz Vandenhirtz, Nicolas De- perrois, Farhad Nooralahzadeh,, Michael Krauthammer, Thomas M Sutter, and Julia E Vogt. Leveraging the structure of medical data for improved representation learning. International Conference on Machine Learning Workshop on FM4LS, 2025.
Anthropic. Claude (version opus 4.1) [large language model). https://www.atthrppicc.co 2025. Accessed: 2025-10-03.
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, and others. Big self-supervised models advance medical image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3478-34488 2021.
Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valli al. Making the most of text semantics to improve biomedical vision languag processing. In European conference compuler vision, pp. 1-21. Springer, 2022.
Aurelia Bustos, Antonio Pertusa, Joso-Maria Salinas, and Maria De La Valesia-Vaya Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis, 66:101797, 2020.
Pierre Chambon, Jean-Benoit Delbrouck, Thomas Sounack, Shih- Chih-Cheng Huang, Zhihong Chen, Maya Varma, Steven QH Truong, Chu The Chuong, and Curtis P Langlotz. Chexpert plus: Augmenting a large chest
13

---

Published in Transactions on Machine Learning Research (03/2026)
x-ray dataset with text radiology reports, patient demographics and additional image formats. arXiv preprint or Kiv:24405. 19538, 2024.
Juan Manuel Zambrano Chaves Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang. Yujia Xie, Mahmoud Khademi, Ziyi Yang, and others. Towards a clinically acces- sible radiology foundation model: open-access and lightweight, with automated evaluation. arXiv preprint arXiv:2403.0800 2024.
Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general-purpose foundation model for computational pathology. Nature medicine, 30(3):850 862, 2024a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Gcoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597 1607. PmLR, 2020.
Zhihong Chen, Maya Varma, Jean-Benot Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, al. Chexagent: Towards foundation model for chest x-ray interpretation. In AAAI 2024 Spring Symposium on Clinical Foundation Models, 2024b.
Nicolas Deperrois, Hidetoshi Matsuo, Samuel Ruipérez- Campélez-Campilo, Moritz Vandenhirtz, Sonia Laguna, Alain Ryser, Koji Fujimoto, Mizuho Nishio, Thomas M Sutter, Julia E Vogt, et al. Radvlm: A multitask conversational vision-language model for radiology. arXiv preprint ar Xiiv:2502.03333 2025.
Jacob Devlin, Ming-Wae Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidircctional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: haman language technologies, volume 1 (long and short popers), pp. 4171 4186, 2019.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, and others. An image is worth 16x16 words: Transformers for image recognition at scale. Proceedings of the International Confer- ence on Learning Representations, 2021.
Florin C Ghesu, Bogdan Georgescu, Awais Mansoor, Youngjin Yoo, Dominik Neumann, Pragneshkumar Patel, Reddappagari Suryanarayana Vishwanath, James M Balter, Yue Cao, Sasa Grbic, et al. Costrastive supervidd learning from 100 million medical images with optional supervision. Journal of Medical Imaging, (66:006453-004555 2022.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23 2021.
Sebastian Gündel, Arnaud AA Setio, Florin C Ghesu, Sasa Grbie, Bogdan Georgescu, Andreas Maier, and Dorin Comaniciu. Robust classification from noisy labels: Integrating additional knowledge for chest radiography abnormality assessment. Medical Image Analysis, 72:102087, 2021.
Md Inzamam UI Haque, Abhishck K Dubey, Ioana Danciu, Amy C Justico, Olga S Ovchinníkova, and Jacob D Hinkle. Effect of image resolution on aulomated classification chest X-rays. Journal of Medical Imaging, 1044):044503 -444003, 2023. Publisher: Society of Photo- Pooto-Optical Instrumentation Engineers.
Hamed Hassanzadeh, Mahnoosh Kholghi, Anthony Nguyen, and Kevin Chu. Clinical document, classification using labeled and unlabeled data across hospitals. In AMIA annual symposium proceedings, volume 2018, pp. 545, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, et al. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVT /CV conference computer vision and pattern recognition, pp. 100000160000 2022.
Jeremy Trvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Tleus, Chris Chute, Henrik Mark- lund, Behzad Haghgoo, Robyn Ball, Katic Shpanskaya, and others. CheXpert: A large chest radiograph dataset with uncerlainty labels and expert comparison. In Proceedings of the AAAI conference urlificial intelligence, volume 33, pp. 590-50 390-597, 2019.
Alistair Johnson, Matt Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. XIMIC-CXR-JPG-chett radiographs with structured labels. PhysioNee, 2019a.
Alistair EW Johnson, Tom Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Roger Mark, and Steven Horng. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019b. Publisher: Nature Publishing Group UK London.
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Tgor Odintsov, Long Phi Le, Georg Gerber, al. A visual-language foundation model for computational pathology. Nature medicine, 30(3):863-874 2024.
Shentong Mo and Paul Pu Liang. Multimed: Massively multimodal and multitask medical understanding. arXiv preprint arx:::2408. 12682, 2024.
Michael Moor, Oishi Banerjee, Zahra Shakeri Hosscin Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist. medical artificial intelligence. Nalure, 616(7956):259 265, 2023.
Thanh Nguyen et al. Self-supervised contrastive learning for multi-view chest x-ray representation. In MICCAI, 2022.
Suraj Pai, Dennis Bontempi, Ibrahim Hadzic, Vasco Prudente, Mateo Sokač, Tafadzwa L Chaunzwa, Simon Bernatz, Ahmed Hosny, Raymond H Mak, Nicolai J Birkbak, al. Foundation model for cancer imaging biomarkers. Nature machine intelligence, 6(3):354-367, 2024.
Chantal Pellegrini, Ege Özsoy, Benjamin Busam, Benedikt Wiestler, Nassir Navab, and Matthias Keicher. RaDialog: Large Vision-Language Models for X-Ray Reporting and Dialog-Driver Assistance. In Medical Imaging with Deep Learning, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763. PmLR, 2021.
Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland S Zimmermann, and Wieland Brendel. Infonce: Identifying the gap between theory and practice. arXiv preprint arXiv:2407.000 2024.
Thomas Sutter, Yang Meng, Andrea Agostini, Daphné Chopard, Norbert Fortin, Julia Vogt, Babak Shah- baba, and Stephan Mandt. Unity by Diversily: Improved Representation Learning for Multinodal VAEs. Advances in Newral Information Processing Systems, 37:74262 -749777 2024.
Ekin Tin, Ellie Talius, Pujan Patel, Curtis Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert- level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature biomedical engineering, 6(12):1399 1406, 2022a.
Ekin Tiu, Ellie Talius, Pujan Patel, Curtis Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert- level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nature biomedical engineering, 6(12):1399-144 1406, 2022b. Publisher: Nature Publishing Group UK London.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. Advances in Neural Informalion. Processing Systems, 36:46830-
46855, 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Tbrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. ar preprint XrXe:22502. 14786, 2025.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale chest x-ray database and benchmarks weakly-supervised classification and local- ization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
Junfei Xiao, Yutong Bai, Alan Yuille, and Zongwei Zhou. Delving into masked auloencoders for multi-label thorax disease classification. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 3588 3600, 2023.
John R Zech, Marcus Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of deep learning model detect pneumonia in chest radiographs: cross-sectional study. PLoS medicine, 15(11):1102263, 2018.
Haoran Zeng et al. Learning cross-view anatomical consistency for radiographic representation. 1EEE Transactions on Medicall Imaging, 2023.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. A multimodal biomedical foundation model trained from fifteen million image-text pairs. NEJM AI, 2(1):A1624006640, 2025.
Bowen Zhou et al. Multi-view self-supervised learning for chest x-ray representation. Medical Image Analysis, 2023.
Xiahai Zhuang et al. Self-supervised feature learning for 3d medical images by playing a rubik's cube. In MICCAI, 2019.
16

---

Published in Transactions on Machine Learning Research (03/2026)
A Datasets and Study Structure
This section provides additional details on the four datasets used in our study: MIMIC-CXR, CheXpert, PadChest, and Chest. X-ray. An illustration of the study-level structure of the combination of datasets is depicted in Figure 6) and an example of the study acquisition for one instance is illustrated in Figure 7
A single study may contain multiple radiographic views, which we group into two macro-categories, frontal and lateral. Views for which this information is unavailable views retained during pretraining to avoid selection bias and to reflect the heterogencity of clinical radiology data, where view metadata may be incomplete or inconsistently recorded. Moreover, retaining all views increases the effective training data volume, which we found to be beneficial despite the presence of potentially noisy signals. Views distribution for each dataset are described in Table
categorized as unknown. All available
Frontal
Lateral
Unknown
Text Reports
Labal
Study
ceptal thickining. Chaze La no cooal consotiaccoon plezeat
Ediema
Dataset
Fracture
((R)
Lung Opacity
la nomall the cardiacy
Filar, nedleatino) oor
Procmonia
:
Eolema
X
cadiographa dacosatzace roreat cardhowed acrina' ellhweere,, nuneroos cunglest clips
CheXPart
Fratcfure
Labeler
Lang Opacity
CheXPert
missing
Chest X-Ray
Ealema
infectar proyecction lateral sin clar teaducaion proyaction podzi relacion proces context
Fracture
PadChest
Claude
Lang Opacity
Przumonii
Figure 6: Study-level structure reprosentation. Each illustrates representative study with three distinct views (frontal, lateral, and unknown), accompanied by corresponding report excerpis (lexi) and the final diagnostic labels (e.g., Edema, Lung Opacity, Fracture, Pneumonia) for the different datasets (MIMIC CXR, CheXPert, Chest X-Ray and PadChest). We treat cach study as the fundamental unit of analysis, and various combinations of views are possible (e.g., including various frontal images (rows p and 4), or missing- ness (row p)). For Chest X-Ray dataset, no lateral or unknown views were available, nor text reports or labels.
Table 4: Distribution of radiographic views across datasets.
Dataset
Frontal (F)
Lateral (L)
Unknown (U)
MIMIC-CXI
239,931
116,555
15,465
CheXpert+
191,071
32,391
PadChest
62,901
27,526
68,095
ChestX-raye
86,524
17

---

Published in Transactions on Machine Learning Research (03/2026)
8888888
Figure 7: Illustration of study acquisition. For a single study, various views could be acquired at different frequencies. This figure illustrates an example of a study composed of three lateral view images and two frontal ones. With these, the radiologist writes a report from which labels are extractod.
Table 5) summarizes the key characteristies of each dataset, including the distribution of studies across the training, validation, and test splits, the total number of patients and imagos, and the cardinality of their original, unprocessed label sets.
Table Dataset Specifications. A detailed breakdown of each dataset, including study counts for training, validation, and test splits, along with patient and image totals, label counts, and original image dimensions.
# Studies
#
#
# Original
Original
Train
Val
Test
Patients
Images
Labels
Image Size
MIMIC-CXR
222,758
1,808
3,269
65,086
371,951
14
2500 x 3056
CheXpert 187,511
200
500
64,725
223,462
14
320 x 320
PadChest
106,677
1,653
1,601
66,610
158,522
193
256 x 256
Chest X-ray 84,774
1,750
25,596
30,805
86,524
15
1024 x 1024
Figure 8 illustrates the label distribution in the training and validation splits for each dataset used in the fine-tuning phase (i.e., MIMIC-CRR, CheXpert, and PadChest), as well as for the Combined dataset. Note that in PadChest, the only available labels are and (indicating the absence or presence of a condition, respectively), whereas the other datasets include additional label values. This variation reflects the inherent diversity among the datasets; however, after preprocessing, all labels were binarized (0 1), ensuring consistent and aligned representation across datasets. In Figure Owe report the statistics after the binarization procedure. Finally, the following paragraphs describe the specific preprocessing steps applied to the labels to create unified evaluation framework.
Label Harmonization and Processing To enable a unified analysis across all four datasets, we harmo- nized their diverse label sets into the 14 diagnostic categories defined by the CheXpert labeler: Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged Cardiomediastinum, Fracture, Lung Lesion, Lung Opacity, No Finding, Pleural Fffusion, Pleural Other, Pneumonia, Pneumothoruz, and Support Devices. The harmo- nization procedure adapted to the characteristics and complexity of each datasct. For MIMIC-CXR and CheXpert, directly utilized the binarized labels provided by the official CheXpert labeler, where explicit positive mentions were coded as 1, and all other states (negative, uncertain, or not mentioned) were coded as O. Finally, the PadChest dataset required the most extensive harmonization due to its large and hetero- geneous label set. It initially contained 433 labels, which were consolidated into 193 unique labels following text normalization (e.g., removal of extraneous spaces). Tn this case, we adopted Claude Code 2.0.35 generate mapping from the 193 PadChest labels to the CheXpert categories using the following prompt "I have these labels fläst of 193 unique PadChest labels] and have to map them to these other 14 labels: flist of 14 unique CheXperl labels), can you provide ensured complete and consistent coverage of all labels. The final mapping file is included in our repository for transparency and reprodueibility.
file with the mopping? This automated approach
18

---

Published in Transactions on Machine Learning Research (03/2026)
Negative
mentioned
Uncertain
MIMIC-CXR
1.0
0.8
0.6
0.4
0.2
CheXpert
1.0
0.8
0.6
0.4
0.2
PadChest
1.0
0.8
0.6
0.4
0.2
All the datasets
1.0
0.8
0.6
0.4
0.2
Atelectaois
Cardiomegaly
Consolidation
Enfarged Cardlomedaastiuum
Edema
Fracture
Lung Lesion
Lung Opacity
No Annding
Pleural Effusion
Pleural Other
Pneumonia
Pneumothorax
Support Devices
Figure 8: Label Distribution. Statistics for the MIMIC-CXR, CheXpert, and PadChest datasets, as well as for the Combined dataset in the training and validation splits. This figure illustrates how the frequency of each diagnostic label varies across individual datasets and the overall collection.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Positive
Negative
MIMIC-CXR
1.0
0.8
0.6
0.4
0.2
CheXpert
1.0
0.8
0.6
0.4
0.2
PadChest
1.0
0.8
0.6
0.4
0.2
All the datasets
1.0
0.8
0.6
0.4
0.2
Atelectasis
Cardiomegaly
Consolidation
Enlarged Cardioeadlasinum
Edema
Fracture
Lung Leston
Lung Opacity
No Andiig
Pleural Effusion
Pleural other
Pneumonia
Pneumothorax
Support Devices
Figure 9: Binary Label Distribution. Statistics for the MIMIC-CXR, CheXpert, and PadChest datasets, as well as for the Combined dataset in the training and validation splits, after label binarization. This figure il- rates how the frequency of each diagnostic label varies across individual datasets and the overall collection.
20

---

Published in Transactions on Machine Learning Research (03/2026)
B
Implementation and Training Details
Timing Analysis
We compare the Independent bascling and MVMAE in terms of computational efficiency. Although MVMAE aligns multiple views, the additional cross view alignment. objective incurs minimal cost relative to image encoding. We measure wall clock training time on identical HPC infrastructure, training both models for 500 epochs using 16 nodes. The Independent bascline requires 19, 19,555 scconds, while MVMAE requires 19, 601 seconds, indicating meaningful difference in training time. These results show that MVMAE improves label efficiency without introducing significant computational penalty. The optional vision to text objective introduces the expected overhead of text decoders and can be omitted when computational efficiency is primary concern.
Alignment Objective Implementation
For cross-view alignment, and effectiveness, enabling representation aligmment. with minimal computational cost. Tn preliminary exper- iments, we observe no substantial difference between MSE-based alignment and contrastive alignment using InfoNCE (Rusak et al. 2024). more systematic exploration of alternative alignment objectives represents an interesting direction for future work.
mean squared error (MSE) loss. This choice is motivated by its simplicity
Training Parameters
Pretraining of the MVMAF model reported in Tables 1 and 2 is performed for 1,200 epochs. We use a linear learning rate warmup for the first 5 epochs, followed by cosine annealing for the remaining training. The base learning rate is set to lr = 2 x 10-4 with a batch size of bs = 64. This learning rate schedule adopted following prior work on masked autoencoders (Xiao et al., 2023).
The weighting between the reconstruction and alignment losses, denoted by 3, follows sigmoid anncaling schedule, increasing from 0 to 1 with steepness parameter of 0.02. This design choice aims to stabilize early-stage training by gradually introducing the alignment objective. For the MVMAE- V2T variant, we fix the weighting cocfficient of the cross-entropy loss to = as a default setting. Duc to resource limitations, we did not conduct a sensitivity analysis of 3 and 7, although we expect that further tuning of these hyperparameters could lead to additional performance improvements.
21

---

Published in Transactions on Machine Learning Research (03/2026)
C
Experiments
This section provides extended experimental details and complementary analyses to those presented in the main text. We include additional visualizations, dataset-specifie label efficiency curves, and per-label classification tables to further examine how performance varies across pathologies, datasets, and model variants. These supplementary results offer a more granular view the trends discussed in Section 5.3 reinforcing the main findings and illustrating the consistency and robustness of MVMAE and MVMAE- V2T across different evaluation settings.
Overall performance across datasets: Results per label and dataset for studied baselines
To complement the aggregate results presented in Section 5.3 we report detailed per-label AUROC scores for each dataset and model variant Table 6 These provide fine-grained view of model behavior across the fourteen diagnostic categorics MIMIC-CXR, CheXpert, and PadChest, across studied baselines. These results allow for closer inspection of pathology-specie achieves competitive or superior performance compared to both unimodal and vision-language basclines. The accompanying radar plots in Figure 10 visualize these results across datasets, illustrating inter-dataset variability and highlighting how model selection based on specific pathologies or disease categories can yield targeted performance gains, an aspect particularly relevant for medical applications.
trends and confirm that MVMAE consistently
22

---

Published in Transactions on Machine Learning Research (03/2026)
Table Per-label classification AUROC on MIMIC-CXR, Results are shown for all compared models.
Label
Supervised
Independent
BiomedCLIP
CheXagent
MVMAE (ours)
Backbone
VIT-B/16
VIT-B/16
VIT-B/16
VIT-L/14
VIT-B/16
Atelectasis
0.81
0.82
0.81
0.81
0.82
Cardiomegaly
0.80
0.81
0,80
0.81
0.82
Consolidation
0.81
0.82
0.80
0.81
0.83
Filema
0.88
0.89
0.88
0.89
0.89
Enlarged
0.68
0.73
0.74
0.71
0.74
Fracture
0.63
0.68
0.65
0.70
0.73
Lung Lesion
0.77
0.78
0.74
0.78
0.83
Lung Opacity
0.74
0.75
0.74
0.73
0.76
Fiuding
0.86
0.85
0.86
0.87
Pleural Effusion
0.91
0.91
0.91
0.91
0.92
Pleural Other
0.74
0.83
0.80
0.79
0.84
Pneumonia
0.72
0.76
0.73
0.72
0.76
Pneumothorax
0.82
0.85
0.81
0.86
0.86
Support Devices
0.88
0.90
0.90
0.90
0.92
Table Per-label classification AUROC on CheXpert. Results are shown for all compared models.
Label
Supervised
Independent
BiomedCLIP
CheXagent
MVMAE (ours)
Backbone
VTT-B/16
VIT-B/16
VIT-B/16
VIT-L/14
VIT-B/16
Atelectasis
0.76
0.81
0.74
0.73
0.78
Cardiomegaly
0.91
0.92
0.94
0.89
0.90
Consolidation
0.76
0.75
0.82
0.79
0.77
Edema
0.86
0.86
0.83
0.88
0.87
Enlarged
0.88
0.87
0.85
0.75
Fracture
0.84
0.86
0.75
0.86
0.90
Lung Lesion
0.82
0.82
0.78
0.85
0.78
Lung Opacity
0.72
0.78
0.74
0.73
0.77
No Finding
0,80
0.82
0.84
0.86
0.83
Pleural Effusion
0.85
0.86
0.85
0.87
0.86
Pleural Other
0.66
0.73
0.70
0.91
0.59
Pueumonia
0.29
0.58
0.82
0.57
0.90
Pneumothorax
0.82
0.90
0.86
0.89
0.93
Support Devices
0.82
0.87
0.88
0.92
0.93
Table 8: Per-label classification AUROC on PadChest. Results shown for all compared models.
Label
Supervised
Independent
BiomedCLIP CheXagent
MVMAE (ours)
Backbone
ViT-B/16
ViT-B/16
VIT-B/16
VIT-L/14
VIT-B/16
Atelectasis
0.85
0.83
0.88
0.89
Cardiomegaly
0.92
0.91
0.92
0.92
Consolidation
0.88
0.91
0.88
0.91
0.91
Edema
0.91
0.94
0.93
0.93
0.95
Eularged C.
0.81
0.84
0.81
0.80
0.84
Fracture
0.84
0.78
Lung Lesion
0.78
0.82
0.79
0.83
Lung Opacity
0.80
0.82
0.80
0.79
0.82
No Finding
0.86
0.89
0.87
0.88
0.89
Pleural Effusion
0.94
0.95
0.94
0.93
0.95
Pleural Other
0.83
0.82
Pneumonia
0.89
0.91
0.82
0.91
0.93
Pneumothorax
0.86
0.95
0.82
0.93
0.91
Support Devices
0.91
0.95
0.80
0.95
0.95
23

---

Published in Transactions on Machine Learning Research (03/2026)
Label Efficiency per Dataset under Finetuning
Figure presents the dataset-specific label efficiency curves for MIMIC-CXR, CheXpert, and PadChest, respectively. These plots correspond to the same experiment described in [Section 5.3 but show the trends separately for cach datasct. For both lincar probing and full finetuning, MVMAE and MVMAE-V2T demon- strate higher label efficiency compared to baselines, most significantly the low-label regime, confirming the consistency of the results across datasets and highlighting the robustness of multi-view pretraining.
Effect of Number of Views per Study: Detailed Results
Table g expands on the analysis presented in Effect of number of views per study in Section 5.3 by reporting the absolute AUROC values across datasets. While the main lext locuses the relative performance gains (A) when transitioning from single-view to dual-view evaluation, this table provides more detailed view of the underlying scores. The results show that both models benefit from incorporating additional views; however, MVMAE with two views consistently achieves the highest performance.
Table 9: Comparison of MVMAE and Independent models under single- and dual-view evaluation using 5k-sample linear probing. Both models were evaluated only on studies containing two available views to ensure a consistent comparison. Results show the mean macro-average AUROC across three random seeds.
AUROC
Model
Setting
MIMIC-CXR
CheXpert
PadChest
Combined
One view Two views (all)
72.43
53.74
71.49
76.63
MVMAE
75.22
57.33
77.38
79.50
One view
72.43
52.28
70.92
76.32
Independent
Two views (all)
74.46
57.21
76.96
79.23
24

---

Published in Transactions on Machine Learning Research (03/2026)
MIMIC-CXR
CheXpert
PadChest
Atclecansis
Atelogracis
Support Devios
Cactionmggay
Support Devioss
Cantionnggaly
Preumathorex
Corsoldaation
Preumathoras
Consdiilation
Pueumomia
Bieet
Premmocaia
Ediema
Plenma Other
Exilanged C
Plennal Other
Enlargod C.
Pleural Elftation
Fraetane
Pleural Efftssion
Proctuze
No Findligg
Lung Lesion
No Finling
Lung Lesion
Lung Opacity
Lang Oparity
(a) MVMAE-221 V2T
(b) MVMAE-V2T per label
Ateloctwii
Suppect Devices
Cardiomegaly
Support Devices
Cardiomegaly
Pneunollxcax
Conodidaatin
Preummohoons
Concolidation
Pneamannia
Eieraa
Puetmonia
12demn
a.s.
Othor
lepped
Pinural Other
Ealarged C.
Pleural Eifucion
Fracture
Piennal Eifisioo
Practure
Findlnng
Lang Leston
Finding
Lung
Lang Opacity
Lung Ouunity
(c) Independent
(d) Independent per label
Audeppoai
Abeley goin
Support Devices
Cardiomogaly
Support Devicess
Cardiomegely
Paremeshooax
Cousolidation:
Preumothoras
Consdiiaation
Preemmonin
Edema
Puemizntis
Edlen
Pheural Otber
Enlarged C.
Plemaa Other
Ealorged C.
Pleund Ellision
Fouturre
Plonral Ellisio
Practure
No Finding
Lung Lesions
No Fincing
Lung Lesion
Lung Opocity
Lung Opecity
(e) BioMedCLIP
(f) BiomedCLIP per label
Atoloctaais
Atrlorradi
Support Devices
Carliomegaay
Support Devices
Cardiomegaly
0.78
Puourzolhorax
J01
Coneellitation
Presmothoruo
0.83
Consolidation
.00
Paeumocin
Edecaa
Puetmonia
Edemaa
Pound Other
Enlarged C.
Pleural Other
Eolarpol C.
Pleural Pifiuton
Precture
Pleural Eliuzion
Fracture
No Faddcg
Long, Losion
No Fincing
Lang, Losion
Lang Opecity
Lung Opacity
(g) Supervised
(h) Supervised per label
Figure 10: Per-label AUROC comparison across dalasets for all models not shown in the main text.
25

---

Published in Transactions on Machine Learning Research (03/2026)
MVMAR
MVMAF.V2T
Independent
BlomedCT.IP
Supervised
0.84
0.84
0.50
0.50
AUROC
0.76
AUROC
0.70
0.72
0.88
0.68
0.88
0.64
0.60
10a
2xx
10'
5x
0x
[4]
[A]
(a) Mimic- CXt:: Linear Probing
(b) Mimic- Mimic-CXRR. Finetuning
0.88
0.94
0.94
0.80
0.80
AUROC
0.76
AUROC
0.76
0.88
0.68
0.84
0.64
0.80
10
10
10
10
5
2 10
[4]
[A]
(c) Chexperl Plus: Linear Probing
(d) Chexpert Plus: Finetuning
0.88
0.88
0.84
0.84
0.80
0.50
AUROC
AUROC
0.77
0.72
0.77
0.68
0.68
0.881
10
10*
2x 10 [A]
5xx 10*
107
5x
10
10F
2 10
5xx 10*
10P
(e) Padchest: Linear Probing
(f) Padchest: Finetuning
Figure 11: Label efficiency curves on Mimic-CXR, Chexpert Plus, and PadChest showing macro-AUROC across varying labeled data sizes.
26


Reviews:
[{"review_id": "9fV4GVlwx2", "reviewer": "Reviewer_DcLe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Multiview Masked Autoencoder (MVMAE), a self-supervised pretraining framework designed specifically for the structured nature of radiology studies. Rather than treating radiographs as independent images, the authors propose a \"study-centric\" approach that leverages the natural redundancy found in multiple projections (e.g., frontal and lateral views) of the same patient encounter.\n\nThe methodology combines two primary objectives:\n\n- Masked Image Reconstruction: Capturing local visual details by recovering masked patches within individual views.\n\n\n\n- Cross-View Alignment: Encouraging view-invariant representations by regularizing embeddings across different projections of the same study.\n\n\nAdditionally, the authors extend this into MVMAE-V2T, which incorporates an auxiliary vision-to-text objective. This uses radiology reports during pretraining to enhance semantic grounding while ensuring the model remains a vision-only encoder during inference, avoiding dependence on text at test time.\n\n\n# Key Strengths\n\n- Structural Alignment with Clinical Reality: The method exploits the inherent organization of clinical data (multiple views per study), which is a powerful yet often underutilized signal in medical AI.\n\n\n\n\n- High Label Efficiency: The framework demonstrates significant gains in low-label regimes. For instance, it achieves approximately 80% AUROC with only 5,000 labeled samples, whereas a supervised baseline requires 50,000 samples to reach a similar performance level.\n\n\n\n\n- Robust Benchmarking: The authors evaluate their model across three large-scale public datasets (MIMIC-CXR, CheXpert, and PadChest), demonstrating consistent performance and cross-institutional generalization.\n\n\n\n- Improved Calibration: Beyond simple classification accuracy, MVMAE yields better-calibrated confidence estimates (measured by Brier score) compared to independent or vision-language baselines.\n\n# Key Weaknesses\n\n- Incremental Technical Novelty: While the application is highly specialized and effective, the underlying technical components are largely adapted from existing methodologies. The novelty lies in the synthesis and domain adaptation—combining standard Masked Autoencoders (MAE) with standard alignment techniques and captioning objectives (CapPa) —rather than the invention of new architectural primitives. Overall, the introduced framework aligns with existing wokrs quiet a lot, such as SigLIP2 etc.\n\n**Tschannen M, Gritsenko A, Wang X, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features[J]. arXiv preprint arXiv:2502.14786, 2025.**\n\n- Dependency on Pre-existing Labelers: For dataset harmonization, the work relies heavily on the CheXpert labeler and automated tools (Claude) to map disparate label sets (e.g., mapping PadChest’s 193 labels to CheXpert’s 14). Any inherent bias or noise in these labeling tools could propagate into the evaluation metrics.\n\n\n\n- Computational Overhead: Jointly processing multiple views with high masking ratios (90%) and additional text decoders increases the complexity of the pretraining phase compared to simple unimodal SSL approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The evidence provided is robust and directly addresses the paper's claims:\n\n\nEmpirical Gains: MVMAE consistently outperforms supervised baselines and domain-specific VLMs like BiomedCLIP.\n\n\n\nLabel Efficiency: The model demonstrates a \"10x gain\" in low-label regimes; achieving 80% AUROC with 5k samples compared to the 50k samples required by a supervised baseline.\n\n\nCalibration: The use of the Brier score proves that multi-view pretraining leads to better-calibrated confidence estimates, which is critical for clinical reliability.", "audience_yn": "Yes", "audience_explanation": "While the technical components are familiar, the application of structural constraints to medical foundation models is of high interest to researchers in medical AI and self-supervised learning. The findings regarding how multiview alignment compensates for limited labels provide a valuable blueprint for other structured modalities like MRI or CT.", "requested_changes": "The following adjustments are essential to support the paper's claims regarding the necessity and robustness of the specific multiview framework:\n\n* \n**Quantitative Justification of latent MSE vs. Contrastive Objectives**: The paper uses a MSE loss for . Since contrastive losses (e.g., InfoNCE) are more standard in multiview SSL, the authors must provide a brief comparative analysis or stronger theoretical justification for why a simple distance-based alignment is sufficient for radiographs.\n\n\n* \n**Ablation on \"Unknown\" View Noise**: The framework incorporates \"Unknown\" views into the training pipeline. It is critical to demonstrate whether these undefined views act as a regularizer or introduce noise that could degrade the alignment of high-quality Frontal-Lateral pairs.\n\n\n* \n**Clarification of Label Mapping Accuracy**: The use of an LLM (Claude) to map 193 PadChest labels to 14 CheXpert categories is a practical choice but introduces potential systematic error. A small-scale validation by a human expert on this mapping is necessary to ensure the \"Combined\" dataset metrics are not artifactually inflated by incorrect label grouping.\n\n\n\n### Strengthening Suggestions\n\nThe following changes would improve the depth and transparency of the work but are not strictly required for acceptance:\n\n* \n**Visual Interpretation of Cross-View Information**: While the paper proves the method works via AUROC , providing attention maps (e.g., Grad-CAM) would strengthen the work by showing if  forces the model to focus on anatomically consistent regions across views (e.g., the same nodule in both frontal and lateral projections).\n\n\n* \n**In-depth Complexity Analysis**: Given that the novelty is primarily in the assembly of existing tools (MAE + Alignment + Captioning) , a more detailed discussion on the training wall-clock time and VRAM overhead compared to the \"Independent\" baseline would help practitioners assess the cost-benefit ratio of the structural supervision.\n\n* \n**Expanded Discussion on Technical Modesty**: The authors should more explicitly acknowledge that the primary contribution is the \"domain-specific synthesis\" of existing SSL primitives rather than the invention of new architectural components. This transparency aligns better with the TMLR criterion regarding modest but clear contributions.", "broader_impact_concerns": "No, The paper uses de-identified public datasets and aims to improve the robustness of medical diagnosis.", "additional_comments": "While the proposed method is undeniably effective and highly aligned with the specific nature of radiological data, a critical reviewer would note that the **intrinsic technical novelty is constrained**, as it primarily re-assembles existing SOTA components:\n\n## The framework is a combination of three well-established paradigms:\n\n1. \n**Masked Autoencoders (MAE)**: The reconstruction objective () and the high masking ratio (90%) are directly adopted from the original MAE work.\n\n\n2. \n**Multiview Alignment**: Using MSE or contrastive-like losses to align different views of the same object is a standard technique in multiview SSL (e.g., SimCLR/MoCo adaptations).\n\n\n3. \n**CapPa/Image Captioning**: The V2T extension uses a standard transformer decoder to causally predict report tokens, a method already explored in frameworks like CapPa.\n\n\n\n## Domain-Driven Adaptation over Algorithmic Innovation\n\nThe \"novelty\" here lies not in a new mathematical operator or a novel transformer architecture, but in the **domain-specific engineering** of the objective functions:\n\n* \n**Modality Embeddings**: The use of learnable embeddings () for frontal/lateral views is analogous to BERT’s segment embeddings.\n\n\n* \n**Beta Annealing**: The schedule for weighting reconstruction vs. alignment is a common optimization heuristic in VAEs and multimodal learning.", "extra_scores": {}, "extra_text": {}}, {"review_id": "N6mzRKN8Cg", "reviewer": "Reviewer_czM9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis paper introduces MVMAE, a self-supervised pretraining framework for chest radiography that exploits the inherent multi-view structure of radiology studies. MVMAE combines masked image reconstruction with cross-view alignment. The authors extend this with MVMAE-V2T, which incorporates radiology reports as auxiliary signals during pretraining while maintaining vision-only inference. Experiments across MIMIC-CXR, CheXpert, and PadChest demonstrate consistent improvements over supervised and vision-language baselines, particularly in low-label regimes.\n\n**Strengths**\n1. Well Written: The paper is written in a good manner making it easy to follow,\n2. Clinically motivated: The chosen problem statement and approach of leveraging the structured nature of radiology studies with multiple projections, is relevant and well motivated.\n3. Comprehensive Evaluation: The experimental design is good. They evaluate their methods on 3 datasets, and analyze the per-label behavior of their models. \n4. Strong Baselines: The authors compare their models, to CheXagent (VLM) and BiomedCLIP.\n\n**Weakness**\n1. MVMAE-V2T under performs MVMAE nearly across all comparisons. I feel authors should address this surprising result in the main text. What could be the reasons for this? Does text introduce noise? \n2. Statistical Significance of Tables-1,2,3 : Were the results shown in Tables 1,2 averaged over random seeds, if so please report them in the main text. If not, I would recommend the authors to report the mean performance and std. dev instead of just performance.\n3. Brier Scores : The values in table 3 seem very closeby (with the differences being < 0.01). It would be good if authors could do a significance testing to support these results.\n4. Is it a fair comparison to use ViT-B/16 to compare against CheXagent (which uses ViT-L/14) , as it has 3x more parameters?", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper presents convincing, accurate, clear evidence in forms of the tabular results and result plots. The experimental design makes sense and is well thought of.", "audience_yn": "Yes", "audience_explanation": "Yes, this work would be relevant to the medical AI community and is an incremental step towards improving AI's performance on medical topics.", "requested_changes": "The following additions would strengthen your work:\n\n- Statistical Significance of the results (Table 1,2,3)\n- Address why MVMAE-V2T under performs as compared to MVMAE in main text.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LdfleJxmGP", "reviewer": "Reviewer_ZDv9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes MVMAE, a study-centric self-supervised pretraining method for chest X-rays that leverages the natural multi-view structure of radiology studies. The method combines (1) masked image reconstruction and (2) a cross-view alignment loss to encourage view-invariant representations across multiple projections within a study. The paper also introduces MVMAE-V2T, which adds a vision-to-text objective using radiology reports during pretraining, while keeping inference vision-only. Experiments on MIMIC-CXR, CheXpert, and PadChest show MVMAE improves over supervised training and several baselines, while the V2T variant is presented as improving semantic grounding (especially in low-label regimes). \n\n## STRENGTHS:\n\n1. Radiology studies have inherent structure (multiple views + report), and exploiting that structure for SSL pretraining is well motivated.\n\n2. Reconstruction + alignment is straightforward to implement and ablate, which has simple and clean objectives.\n\n3. Evaluation tasks are broad, across multiple datasets and settings.\n\n## Weaknesses:\n\n1. MVMAE-V2T is not a symmetric image-text alignment objective (e.g., contrastive alignment like CLIP). Instead, it is a captioning objective: a transformer decoder predicts report tokens conditioned on visual embeddings via cross-attention, optimized with cross-entropy loss. This introduces an **inductive bias** toward report-predictive features, which could dilute fine-grained visual signals that matter for certain downstream tasks. This concern is consistent with Table 1: V2T is not consistently better than MVMAE in the full-label results (and MVMAE without text can be stronger on some metrics/datasets). The paper should more clearly justify why captioning is the best way to use text, and provide diagnostics showing when and why V2T helps versus hurts. At minimum, align the main claims with the mixed outcomes in the main table, and ideally provide variance/statistical significance. \n\n2. The method introduces beta to balance reconstruction and alignment, and mentions a beta-annealing schedule. However, key details are unclear: what is the target beta value and the precise schedule, and how is the cross-entropy term weighted (it appears added with coefficient 1)? With multiple losses, results can be sensitive to scaling, and tuning can be expensive, especially across multi-dataset training. The paper would benefit from (i) explicitly reporting the chosen beta and schedule, (ii) optionally adding a gamma weight for the CE loss (even if gamma=1), and (iii) a small sensitivity analysis over beta (and gamma if used).\n\n3. The paper states that reports are not used at inference to ensure fair comparison with purely vision-based models and to avoid relying on report availability or quality. This is reasonable for deployment settings where reports may be missing or noisy. However, it limits the scope: in many clinical workflows, text reports may be available at test time, and a multimodal inference model (image + text) could plausibly improve performance, robustness, or calibration. It would help to clarify whether the “vision-only inference” constraint is a core design goal or mostly a benchmarking choice. A small-scale experiment where reports are available during downstream training/inference (e.g., simple fusion or report-conditioned classifier) would clarify the trade-off.", "claims_and_evidence_yn": "No", "claims_explanation": "I am not fully convinced that we should incorporate text into learning to provide better downstream performance. At least, the current design seems to be suboptimal. However, I could misunderstand the performance reported in the table. Please correct me.", "audience_yn": "Yes", "audience_explanation": "This paper targets self-supervised pretraining for chest X-rays and introduces a simple, study-structure-driven objective (multi-view masked autoencoding + cross-view alignment) that shows consistent improvements across widely used medical imaging benchmarks. Even for readers outside radiology, the core idea—treating “structure” (multi-view/grouped observations) as supervision for representation learning—is broadly relevant and potentially transferable to other domains with grouped or multi-sensor data. The mixed outcome of the vision-to-text extension is also informative for the community, as it clarifies when adding language supervision may or may not help.", "requested_changes": "1. Explain more explicitly that V2T is a captioning-style loss (vision-conditioned report generation), not a symmetric image–text alignment objective. Discuss what representation properties this is expected to encourage, and why this inductive bias should help downstream classification/transfer.\n\n2. Table 1 suggests MVMAE-V2T is not consistently better than MVMAE (vision-only). Please (i) align the narrative/abstract with these outcomes, (ii) report mean±std over multiple seeds, and (iii) indicate which improvements are statistically significant.\n\n3. Specify the exact beta value(s) used for the alignment loss, the annealing schedule, and how the cross-entropy term is weighted (if coefficient=1, justify). Add a small sensitivity study over beta (and gamma for CE if applicable) to show robustness and reduce concerns about expensive tuning.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)