Advancing Radiograph Representation Learning with Masked Record Modeling
Authors: Hong-Yu Zhou
Chenyu Lian
Liansheng Wang
Yizhou Yu
Venue: ICLR 2023 poster (2023)
Keywords: Representation Learning
Radiograph
Self-supervised Learning
Medical Imaging

Abstract: Modern studies in radiograph representation learning (R$^2$L) rely on either self-supervision to encode invariant semantics or associated radiology reports to incorporate medical expertise, while the complementarity between them is barely noticed. To explore this, we formulate the self- and report-completion as two complementary objectives and present a unified framework based on masked record modeling (MRM). In practice, MRM reconstructs masked image patches and masked report tokens following a multi-task scheme to learn knowledge-enhanced semantic representations. With MRM pre-training, we obtain pre-trained models that can be well transferred to various radiography tasks. Specifically, we find that MRM offers superior performance in label-efficient fine-tuning. For instance, MRM achieves 88.5% mean AUC on CheXpert using 1% labeled data, outperforming previous R$^2$L methods with 100% labels. On NIH ChestX-ray, MRM outperforms the best performing counterpart by about 3% under small labeling ratios. Besides, MRM surpasses self- and report-supervised pre-training in identifying the pneumonia type and the pneumothorax area, sometimes by large margins.

Published as a conference paper at ICLR 2023
ADVANCING RADIOGRAPH REPRESENTATION LEARN- ING WITH MASKED RECORD MODELING
Hong- Yu Zhou 1.2 Chenyu Lian Liansheng Wang"
Yizhou Yu
School of Informatics, Xiamen University
Department of Computer Science, The University of Hong Kong 3 AI Lab, Deepwise Healthcare whuzhouhongyu@gm
com, cylian@stu. xmu, edu. 1 sswagg@ xmu .edu.cn cn, yizhouyčacm. org
ABSTRACT
Modern studies in radiograph representation learning (R²L) rely on either self- supervision to encode invariant semantics or associated radiology reports to in- corporate medical expertise, while the complementarity between them barely noticed. To explore this, we formulate the self- and report-completio as two com- plementary objectives and present a unified framework based on masked record modeling (MRM). In practice, MRM reconstructs masked image patches and masked report tokens following a multi-task scheme learn knowledge-enhancee semantic representations. With MRM pre-training, obtain pre-trained mod- els that can be well transferred to various radiography tasks. Specifically, we find that MRM offers superior performance in label-efficient fine-tuning. For in- stance, MRM achieves 88.5% mean AUC on CheXpert using 1% labeled data, outperforming previous L methods with 100% labels. On NIH ChestX-ray, MRM outperforms the best performing counterpart by about 3% under small la- beling ratios. Besides, MRM surpasses self- and report-supervieed pre- pre-trrining identifying the pneumonia type and the pneumothorax area, sometimes by large margins. Code and models are available at hntpp: / /github. com/ RM4M/
MRM- pytorch
INTRODUCTION
Radiograph representation learning (R²L) has been among the core problems of medical image analysis. Previously, downstream radiograph analysis tasks
Inputs
Representations
Outputs
counts on pre-trained models ImageNet (Deng al.
2009) or large X-ray datasets (Wang et al.
(anael Ivioid sornal
2017
Irvin et al. 2019) Johnson et al.) 2019 Bustos
ctt al.
2020) to alleviate the shortage of expert label- The emergence of self self-supervised representa-
Palieel
ing.
tion learning (Doersch et al. 2015] Agrawal et al.
Wang & Gupta, 2015: Zhou et al.] 2021a 2023 provides a choice to conduct pre-training with negligible human intervention by exploiting self- supervision. However, the self- supervvedd paradigm ignores the introduction of medical expertise (e.g.. anatomy), reducing its transfcrability to downstream tasks with limited label information.
2015
Figure 1: Illustration. MRM learns trans-
ferable radiograph representations via recon- structing masked records, i.e., masked radio- graph patches and masked reports tokens.
On the other hand, free-text radiology reports written by experienced radiologists often contain rich do-
main knowledge. To leverage this, researchers developed automated rule-based labelers (Wang
Work done while visiting Xiamen University. First two authors contributed equally. Corresponding authors.

---

Published as a conference paper at ICLR 2023
et al. 2017: Irvin et al., 2019) to extract structured labels from unstructured texts. Nevertheless, these labelers have several practical limitations. First, some procedures the label extraction work- flow, such rulemaking and natural language processing, still require the intensive involvement of experts and engineers. Besides, the developed labelers can hardly adapt to new scenarios due to the fixed rules and lexicons.
Against this background, report-supervised L was proposed (Zhang et al. 2020) to acquire super- vision from radiology reports. In practice, this paradigm leverages words and sentences free-text reports as supervision to guide deep neural networks to learn radiograph representations, outper- forming the archetypical label- and superviide pre-training by observable margins various downstream tasks (Zhang et al., 2020, Zhou et al. 2022). The super--superviied R²L highlights the incorporation of domain knowledge. This differs from the self-supervised paradigm, which focuses on learning invariant semantic representations. Nonetheless, current stud- ies view the self- and report-superviser R as separate, discrete choices, preventing their combina- tions.
the importance
Driven by this analysis, we present a unified framework based on masked record modeling (MRM), where the self- and report-completion tasks are modeled two complementary objectives. Specif- ically, masked image reconstruction integrates semantics into pre-trained models, while masked report restoration facilitates the incorporation of medical expertise. As result, MRM learns knowledge-enhancee semantic representations that generalize well. In practice, MRM masks ran- dom patches and tokens from the input radiograph and associated radiology report with high mask- ing ratios. Following a multi-task scheme, MRM asks the radiography pre-trained model to learn visual representations that can not only reconstruct the missing patches but also restore the missing tokens from the mon-masked token embeddings along with mask tokens.
With MRM pre-training, we can train radiography models on MIMIC-CXR (Johnson et al. 2019 with improved generalization performance. With pre-trained ViT-B/16 model, we achieve 88.5% mean AUC when fine-tuned on CheXpert (Irvin et al., 2019) with only 1% labels. This outperforms all previous counterparts with 100% labeled data. On NIH ChestX-ray (Wang & Gupta, 2015), MRM surpasses the report-supervised paradigm by about 3% when the labeling ratios
Jare are 1% and 10%. On pneumonia identification tasks, MRM outperforms self- and report-supervised baselines, sometimes by substantial margins. These observations help verify the effectiveness of MRM in learning more transferable radiograph representations.
RELATED WORK
2. REPORT-SUPPERVISE] RADIOGRAPH REPRESENTATIN LEARNING
Recently, rewort--upeervvieeeeee supervsed learning Zhhang et al. 2020 Liao et al. 202 Huang et al. 2021 Zhou
al., 2022 Boecking et al. 2022 emerges as new R paradigm gm that automatically acquires supervision from free-text radiology reports. /Zhan et al. (2020) proposed CIRT to contrast the radiograph features with latent embeddings sentences in radiology reports. [iao al, (202T) and Huang et al. (2021) explored the alignment between local patches and words in the report. Zhou al. (022) presented Transformer-baser RLL framework that conducts autoregressive report modeling and study-report matching. Report-superrised RLL takes the advantage of label-supervieed learning, which is the incorporation of domain knowledge. Compared to the self-supervised paradigm, report- supervised R lays no emphasis learning semantically invariant representations. To address the discrepancy between them, we formalize self- and report-completion as two complementary objectives, based on which propose encode both semantics and medical expertise into latent representations following a multi-task scheme.
2.2
VISUAL. REPRESENTATION LEARNING VIA LANGUALA PRE-TRAINING
Learning visual representations from image- langugg pairs has achieved tremendous success in nat-
ural image tasks (Sariyildiz et al. 2020 Desai & Johnson, 2021 Radford et al.) 2021 Mu et al.
2021
Zhao et al. 2022 Li et al. 2021 Geng et al. 2022 Wang et al.) 2022 Chen et al. 2022
'The labeling ratio X% means that X% of the training set from fully annotated downstream dataset used for supervised fine-tuning.
2

---

Published as a conference paper at ICLR 2023
Input report
Masked report generation
Hybrid representations
Masked roport restoration
masked token embed.
maskedd embed.
Non masked hybri token embed. toven embed.
Mosted lingunge modeting (MMM) Inss
Telertizer
Text
Ho HAAE an (4LD))
Report decoder
High extad
Sind mas
Image deceder
Irato
Lew wwwoosrrinngge
Mosked tmoge modeting (MM!)) lass
Input radiograph
Masked LR ganeration
Radiograph representations
Macked restoration
Figure 2: OVERVIEW, During the pre-training stage, MRM requires the image encoder provide radiograph representations to simultaneously support the restoration of masked radiograph patches and masked associated radiology report tokens. The masked language and image modeling losses are only calculated on image and report tokens highlighted in pink Embed., LR, and HR stand for embeddings, low-resolution, and high-resolution, respectively.
Singh et al. 2022 Yu et al., 2022; Dou et al. 2022 Arici et al. 2021 Kwon et al. 2022). Simi-
lar to Con V IRT [Zhang al. (220), image- language contrast has been widely adopted to conduct pre-training (Radford et al. [221]] Mu et al. 2021 Li et al. 2021 Yu et al. 2022 Dou et al.) [022]
Gao et al. 2022). Nowadays, efforts have been made to train unified encoder for vision and lan-
guage data (Geng et al.] 20022] Wang et al. 2022 Chen et al. 2022 Geng et al. 2022). Akin to our approach, SLIP (Mu et al. 2021) combines SimCLR (Chen et 2020) and CLIP RRafford iet al., (222)) to train a vision encoder using image-language pairs. However, SLIP only slightly outperforms SimCLR in fine-tuning, while requiring large batch sizes and tens of millions image- language pairs for pre-training. In contrast, our MRM surpasses various self- ceff-suprvvied methodolo- gies by large margins and can be pre-trained using only hundreds of thousands of radiograph-repon pairs, enabling effective medical visual representation learning with limited annotations and com- puting resources.
MASKED RECORD MODELING
We propose MRM (i.e., Masked Record Modeling) to learn radiograph representations using record- level supervision. the name implies, MRM acquires supervision signals from both radiographs and associated radiology reports. The motivation behind is to learn knowledge-enhaneed semantic latent representations by reconstructing masked radiograph patches and masked radiology report tokens in medical records.
Fig. presents an overview of MRM. We first apply random masking each low-resolution radio- graph and its associated radiology report (with different high masking ratios). Then, we forward the obtained non-masked image patches to the image encoder to acquire non-masked image patch embeddings. These embeddings serve two purposes: (i) assist non-masked report tokens to restore the masked report tokens; (ii) restore the high-resolution masked radiograph patches. To achieve the first goal, we add the globally averaged radiograph representation each non-masked report token embedding and pass the resulting hybrid representations to the report decoder for masked re- port restoration. As for the second purpose, conduct novel patch restoration task to explicitly encode more local details into radiograph representations by reconstructing high-resolution patches from now-resolution inputs.

---

Published as a conference paper at ICLR 2023
3.1 REPORT COMPREHENSION
Masked report generation. In our scenario, each radiology report is associated with a radiograph. To convert the free-text report into tokens, we use WordPiece (Wu et al. (016) as the default to- kenizer, whose vocabulary has approximately 30k tokens. After tokenization, a number of report tokens with [MASK]- Compared to BERT (Devlin et al.] (018) that randomly masks 15% tokens, we
randomly mask
50% probability of masking each token in the report. The insight
behind the use of a higher masking ratio is that we want the model to lean more upon the image embeddings to finish the report-completion task.
Hybrid representations for storing multi-modal information. We then transform non-masked report tokens into token emheddings using simple lookup table] which stores randomly initialized embeddings of a fixed dictionary and size. In practice, we retrieve embeddings using indices. Then, the global embedding of the associated radiograph is added each non-masked token embedding. The resulting non-masked hybrid embeddings are supposed to include the multi-modal information from the radiograph and associated radiology report, which ought to be helpful for restoring the masked tokens.
Masked report restoration. To reconstruct the masked tokens, we forward latent embeddings of both hybrid tokens and mask tokens to the report decoder (a light-weight transformer model), where fixed (i.e., unlcarnable) positional embeddings are added to encode the position information. We train the report decoder using the masked language modeling objective.
3.2 RADIOGRAPH UNDERSTANDING
Masked image generation with low resolution. We propose to learn radiograph representations by reconstructing high-resolution radiograph patches from low-resolution inputs. The motivation behind is to encode more local information into latent embeddings via super-resolution imaging. As shown Fig. [ we sub-sample each high-resolution radiograph by factor of two to generate a low-resolution input. Following He et al. (0022), split low-resolution radiograph into non- overlapping image patches, where 75% patches are randomly masked.
Radiograph representations. We add fixed unlearnable positional embeddings to linearly trans- formed non- non-masked image patches. Next, we forward the resulting patch embeddings to the transformer- baasedd image encoder, which produces global average pooling (GAP) is applied to all non-masked embeddings, whereby a global feature obtained. Here, hypothesize that the image-level information brought by the global feature is helpful to the restoration of masked report tokens. Based on this hypothesis, we duplicate and add the global feature to each non-masked report token embedding, producing the hybrid token embeddings that encode the multi-modal information.
masked image patch embeddings. Then, the
Masked image restoration with high resolution. Non-masked image and mask token represen- tations with added positional embeddings are passed to the image decoder for the restoration of masked radiograph patches. Specifically, the image decoder is required restore a high- regouuesolution (2x the input resolution) patch from each input token via a shared fully-connected (FC) layer (across all tokens). In practice, the proposed restoration procedure explicitly requires the learned image rep- resentations to include more local details that often matter lot in medical diagnosis.
3.3 MULTI-TASK MODELING
Suppose each input radiograph consists of two set IM and IN- The masked set Im=(xx,... } (ground truths) contains high-resolution image patches that serve as reconstruction targets. The non-masked set [v=(si,.... , } comprises & low-resolution patches that are treated as model in- puts. Likewise, we denote the associated radiology report using the masked set RM= up) (ground truths) and the mon-masked set R ..... va} (inputs). Here, x, u, and v stand for the masked image patch, non-masked image patch, masked report token, and non-masked report token, respectively. For model parameters, use θg, θp. and θ₂ to denote the parameters of the image encoder, image decoder, and report decoder, respectively.
Rmbeddi .
htm"

---

Published as a conference paper at ICLR 2023
For the restoration of masked report tokens, we forward hybrid representations to the report decoder and minimize the negative log- ligeliill function. During the training stage, the objective function CR (i.c., the MLM loss in Fig. ( of the above optimization procedure be summarized as follows:
(RM, RN.
Σ log P (u; Ving: $1:k: Og. On);
where P stands for the conditional probability. We ignore the mask tokens for simplicity.
Similarly, we can formalize the objective function of the high-resolution masked radiograph restora- tion (ef. the MIM loss in Fig. 2 as follows:
(IM. = MSE (ʃe, (feg X1sh). (1.1).
(2)
In practice, we adopt the mean squared error (MSE) to measure the differences between the predicted and ground-truth image patches with high resolution, where all pixel values normalized to [0, 1].
The total multi-task training objective function (to be minimized) of the multi-modal restoration is as follows:
(3) where a parameerr that controls the relative impacts of two objective funetions. After pre-training, we can transfer the weight parameters of the image encoder (i.e., O to various down- stream tasks for fine-tuning.
M(RM: Rr Ix) = (R(RM, Rw; IN) ACI(IM,
4 EXPERIMENTS
In this section, we mainly compare MRM against report- and self-supervised R methodologies on well-established public datasets. Average results are reported over three training runs.
4.1 MIMIC-CXR FOR PRE-TRAINING
We conduct pre-training MIMIC-CXR (Johnson al. 2019). of the largest X-ray datasets, that contains more than 370,000 radiograph images from over 220,000 patient studies. Each radio- graph paired with one associated radiology report.
4.2 DATASETS FOR FINE-TUNING
We validate the transfcrability of learned radiograph representations on X-ray based classification and segmentation tasks via end-to-end fine-tuning. Specifically, we evaluate the pre-trained model 4 X-ray datasets in the classification tasks, which are NIH ChestX-ray (Wang al.) (017), CheXpert (Irvin et al.] 2019), RSNA Pneumonia (Shih et al. 2019), and COVID-19 Image Data Collection (Cohen al. 2020)). For the segmentation task, we fine-tune the pre-trained model on SIIM-ACR Pneumothorax Segmentation
CheXpert introduces multi-label classification problem chest X-rays. We follow the official guideline (Irvin et al.) 2019) and report the model performance on 5 selected pathologies, i.c., at- electasis, cardiomegaly, consolidation, edema. and pleural effusion. Considering the official test set of CheXpert is not available to the public, we follow CIRVIRT (Zhang et al. 2020) to regard the official validation set as the test set. Meanwhile, we randomly sample 5,000 images from the official training set to build the validation set. The training/validationnneet split each constitutes 18.41145,,000/34 images of the whole dataset.
RSNA Pneumonia defines binary classification problem, where each chest radiograph is cat- egorized as either pneumonia or normal. We adopt the official data split, where the train- ing/validation/test set comprises 15,184/1,500/3,00 images, respectively.
NIH ChestX-ray consists of about 112,120 frontal-view chest radiograph images, where a multi- label classification problem 14 chest pathologies is introduced. The craining/vaiiattion/ee split each constitutes 70%/10%//202 of the whole dataset.
https:: www. www.waaagga kaggle com/ s/ per-ppeumttoorax sewmentttttttttiinnnnnn

---

Published as a conference paper at ICLR 2023
Methods
Input
Pre-train.
CheXpert
RSNA Pneumonia
SIIM
Size
Data
10%
100%
10US:
10%.
100%:
Our MRM
224
MI-CXR
88.5
88.5
88.72
91.3
92.7
93.3
73.2
91.4
CNN bused
COnV
CheXpert
85.9
86.8
87.3
77.4
80.1
81.3
43.2
59.9
GLoRIA
224
CheXpent
86.6
87.8
88.1
86.1
88.0
88.6
46.9
63.4
COnVIRT
224
MI-XXR
87.0
88.1
88.1
88.8
91.5
92.7
MedKLIPI
224
MI-CXR
87.3
88.0
89.3
72.1
79.4
BioVil.
PuhMed
88.1
89.1
MI-II/XXR
Tranformer-based
GLORIA
224
MI-CXR
86.5 ba
87.5
as
8 us
91.2 as
71.8
90.9
REFERS
224
MI-CXR
87.2+ 8x
88. a3
88,2+
89 az
91.6=00
92.7+
12.
89,7+
MBAE
224
MI-CXR
86.2+
87.3+
87.9=
89
90.8
92.3+
72.0+
90.4
MGCA!
224
MI-CXR
88.8
89.1
89.7
89.1
89.9
90.8
59.3
64.2
Table 1: COMPARISONS ON CHEXPERT, RSNA PNEUMONIA, AND SIIM. We report AUC scores of different labeling ratios when fine-tuning on CheXpert and RSNA Pneumonia. In compari- son, dice scores are presented on SIIM. The best results are bolded. MI- stands for the MIMIC dataset series. Note that ResNet-50 and ViT-B/16 are treated as the default backbones for CNN- and Transformer-base methods, respectively. denotes our implementation of GLoRIA using ViT- B/16. Approaches with leverage disease-level annotations for pre-training. Specifically, numbers of MGCA on CheXpert and RSNA Pneumonia are linear classification results.
Pleuna Thick Thickening
Labeling Ratios
Powumatbura
Cardoonggaly
Consolidationn
Atelectsii
Emphyssma
Infiltratinn
Avenage
Edema
Eifusion
Fibroti
Hemia
Nodule
Mass
Methods
MRM
0.
78.8
90.3
80.0
86.5
86.9
82.0
71.9
90.0
82.3
69.6
72.3
84.0
McdKLIP
77.2
REFERS
76.7
77.5
85.6
78.6
84.9
85.4
79.5
72.3
77.1
67.5
76.2
66.5
71.6
69.3
81.7
1%
67.1
75.8
64.88
62.2
16.2
CIL
71.1
75.1
67.1
77.6
75.1
83.4
71.5
66.8
70.0
63.8
70.1
66.2
68.1
65.7
74.4
Context Restoration
67.8
69.1
64.4
73.2
73.8
78.1
70.0
62.1
70.2
65.2
62.4
59.1
65.0
62.2
73.8
TransWW
71.3
74.5
68.9
76.7
67.9
68.7
66.8
66.5
66.2
68.5
68.8
75.0
ImageNet Pre-training
69.8
73.3
69.6
76.0
81.7
80.5
67.1
64.9
64.8
65.8
67.0
62.3
65.7
65.0
74.0
Our MRM
84
82.3
90.9
81.1
89.0
88.8
92.2
84.8
94.00
70.1
86.6
75.1
78.6
74.3
88.4
McJKLIP
REFERS
80.9
80.1
89.8
79.5
87.8
87.5
88.2
77.2
86.1
69.6
82.0
72.8
74.2
72.2
85.6
10%
Model Genesis
76.0
77.2
72.8
77.5
85.7
85.2
81.0
75.3
78.0
68
73.1
09.5
72.2
67.7
80.4
75.5
71.5
81.2
75.7
77.5
71.7
Context Restoration
73.8
75.5
70.6
77.1
$4.5
84.2
79.4
73.1
67.5
68.1
70.9
66.9
71.7
65.2
79.1
TransWW VW
74.4
76.5
70.8
77.6
83.0
84.8
79.7
69.9
74.7
68.5
72.1
68.3
72.4
63.2
79.6
ImageNet Pre-training
79.8
75.9
80.4
72.1
71.7
65.6
69.6
79.7
Our MRM
85,9 01
84.2
93.0
82.2
91.0
89.6
94..1
N6.7
94.4
71.8
N8..
78..8
81.4
77.3
90.2
McdKLIP REFERS
83.2
92.3
82.1
91.4
76.7
89.1
100%
Midel Genesis
81.0
78.8
84.5
79.2
87.8
56.6
89.7
81.0
85.2
71.1
81.9
73.2
15.8
73.0
85.6
C2L
82.2
81.1
90.2
81.0
88.1
88.0
88.3
80.8
86.8
72.0
82.7
74.1
76.2
75.3
85.9
Context Restoration
82.9
76.4
88.2
78.6
79.6
69.5
Trans
81.7
79.8
85.0
80.0
88.2
87.1
90.1
81.8
85.9
72.3
82.5
74.4
76.6
74.0
86.1
ImageNet Pre-trainnng
80.0
78.3
89.3
77.6
87.9
85.9
87.
78.5
88.8
65.9
79.9
70.7
74.5
71.0
84.7
Table 2: COMPARISONS ON NIH CHESTX-RAY. Besides self-supervised and transfer learning baselines, we also present the performance of REFERS and MedKLIP (with competitive perfor- mance on CheXpert, RSNA Pneumonia, and SIIM) as references. AUC scores are displayed. The highest AUC scores in each labeling ratio are bolded.
COVID-19 Image Data Collection is a relatively small dataset, which involves 900 chest radio- graphs. We follow Zhou et al. (2022) to conduct fine-tuning on this small-scale dataset to investigate the effectiveness of various pre-training methodologies when the amount annotations limited.
There are two tasks included. The first task requires the model to distinguish COVID-19 cases from non-COVID-19 pncumonia cases, where the training/validationnees set comprises 356/54/99 radiographs, respectively. The second task is to distinguish viral pneumonia cases from bacterial pneumonia ones, where the trining/validation/ees set contains 297/43/86 cases, respectively.
SIIM-ACR Pneumothorax Segmentation (SIM) aims to facilitate the development of segmen- tation models to identify pneumothorax disease chest radiographs. SIIM contains over 120,000 frontal-view chest X-rays with precise manual segmentation of pneumothorax. We follow Huang

---

Published as a conference paper at ICLR 2023
al. 2021) to construct the training/validattionnnss split, where each constitutes 70%/15%/15% of the whole dataset.
4.3 BASELINES
4.3.1
REPORT-SUPPERVIED METHODOLOGIES
We first compare MRM against range of pre-training approaches, which use radiology reports as supervision learn radiograph representations. There are report-supervised approaches volved in the baseline comparisons, which are ConVIRT (Zhag et al. 2020). GLoRIA (Huang cet al. 202T). Bil (Boecking et al. 2022), and REFERS (Zhou et al. 2022]. Specifically, Con-
VIRT (Zhang et al. 2020) proposed to learn medical visual representations by contrasting paired radiographs and sentences from radiology reports. CLoRIA (Huang et al. 2021) improved Con VIRT by contrasting radiograph sub-regions and words in the reports. BioVit. (Boecking al. 2022 and
REFERS (Zhou et al.) 2022) incorporated masked language modeling loss into contrastive learn- ing. Moreover, REFERS introduced multi-view fusion attention to better align the representations of each radiograph and its associated report. In addition, MGCA (Wang et al. (002)) and Med- KLIP (Wu et al. 2023) were included as two recent basclines Apart from above baselines, also include M3AE (Geng et al. 2022), recent masked multi-modal pre-traine pre-training method aside from the application to medical data, for comparison.
In experiments, we fine-tune pre-trained models of MRM and other report-superviset methods on Xperp (classification), RSNA Pneumonia (classification), and SIIM (segmentation).
4.3.2 SELF-SUPERVISED AND TRANSFER LEARNING METHODS
Besides report-supervised approaches, we also include self-supervised and transfer learning ap- proaches in our comparisons. Specifically, Context Restoration (Chen et al. 2019), Model Gen- esis (Zhou et al.] 2021b), and VWW (Haghighi et al. 202T) are based on predictive SSL., while C2L (Zhou et al. (020) developed on top of contrastive learning. In addition, MRM is also compared to ImageNet pre-training (Wang et al. (017).. In practice, we conduct the comparisons with self-supervised and transfer learning approaches NIH ChestX-ray and COVID-19 Image
Data Collection.
v.
Methols
1%
10/%
100%
MRM REFERS
MRM
79.4
84.0
85.9
80.4
Masked modkling Supee-resolution restoration
Mode: Genesis
Contest Restoration
C2L.
TrancWW
74.6
76.
77.8
76.0
69.5
71.3
73.0
71.5
Supur-neooution restoration
Masked radsograph modeling (Cg)
Masked repoet modeling (fx)
76.7
78.8
24.7
83.7
82.2
81.3
84.7
85.7
85.1
ImogoNet Presenining
74.1
70.3
Hybrid
image
78.9
Table COMPARISONS ON COVID-
Table 4: ABLATIONS ON NIH CHESTX-RAY AUC scores of three different labeling ratios are reported. The best results bolded.
19 IMAGE DATA COLLECTION. The best results are bolded.
4.4 RESULTS
4.4.1
COMPARISONS WITH REPORT-SUPERVISEE BASELINES
In T we present the comparative results with report-supervisee methodologies on CheXpert, RSNA Pneumonia, and SIIM (segmentation). Specifically, we investigate the performance when fine-tuning with limited and full supervision. We provide reconstruction examples and segmentation results in the appendix.
From Table I we observe no obvious performance gap between CNN- and Transformer-basee report-supervised pre-training methods. For instance, after implementing GLoRIA on MIMIC-CXR with ViT-B/16, we observe performance drops and improvements on CheXpert and RSNA Pneumo- nia, respectively. This contrast demonstrates that replacing CNN with Transformer may not bring
'MGCA Wang al. 2023 and McdKL.IP Ww et al. 2023) were released after the submission deadline of ICLR 2023. We added results the camera ready for better comparisons.
7

---

Published as a conference paper at ICLR 2023
Pi
1%
10%
100%
1%
10%
100%
Fusion
1%
10%
100%
75%
79.4
84.0
85.9
1
79.4
84.0
85.9
GAP 79.4
84.0
85.9
50%
78.8
84.4
86.1
78.6
83.5
85,4
GMP
77.2
83.8
86.0
0%
75.4
82.0
84.4
0.5
79.0
83.7
85.9
(a) Multi-modal fusion strategies
(b) Masking ratios for radiographs
(c) Loss controlling factor À
Table 5: Ablation studies on choices of multi- modall fusion and paree--arameters GAP and GMP stand for global average and maximum pooling, respectively. Experiments are performed on NIH ChestX ray under various labeling ratios. The best results bolded.
performance gains report-superviser pre-training. Among all baselines, REFERS the best per- forming approach.
Nonetheless, MRM consistently outperforms various baselines on all three datasets under different labeling ratios. Specifically, MRM maintains more advantages over previous pre-training method- ologies when fine-tuning with limited annotations, which is quite meaningful for medical image analysis as large amounts of specialist annotations (from radiologists clinicians) usually hard to access. It is worth noting that MRM achieves 88.5% when using only I % labeled data on CheX- pert, better than previous counterparts with 100% annotations.
We see that M3AE generally performs worse than our MRM in all labeling ratios, especially under extremely limited data. The underperformance of M3AE may be attributed to the fact that requires a large amount of multi-modal data to learn transferable joint representations for images and texts.
4.4.2 COMPARISONS WITH SELF-SUPERVISED AND TRANSFER LEARNING BASELINES
Table 2 presents the average and per-class classification results on NIH ChestX-ray. Compared to self-supervised learning baselines, MRM achieves large improvements in almost every chest pathol- ogy. On average, MRM outperforms C2L (Zhou al. 2020) and VWW (Haghighi al. 2021 the best two self-supervised pre-training methodologies, by about 8% when the amount of available labeled data is extremely limited (i.e., the labeling ratio is 1%). Similarly, also observe remark- able improvements when comparing MRM to ImageNet Pre-training (Wang et al. 2017). These phenomena demonstrate that the radiograph representations learned by MRM are more transferable than those from previous self-supervised and transfer learning methods.
Compared to REFERS (i.e., the best performing sepert-sup baseline in Table D. MRM still provides notable improvements in most chest pathologies. Specifically, MRM is more ad adantageuus when the amount of labeled data limited. For instance, MRM surpasses REFERS by 2.7% and 3.1 3.1% on average when the labeling ratios are 1% and 10%, respectively. These comparisons again verify the effectiveness MRM over previous repert-supervised pre-training counterparts.
We also investigate the impacts of pre-training on the real-world scenario with extremely limited specialist supervision. In Table 3 we compare the performance of range of pre-training method- ologies on two binary classification tasks, which are distinguishing COVID-19 from non-COVID-19 pneumonia, and differentiating between viral pneumonia and bacterial pneumonia. Again, MRM outperforms supervider, transfer learning, and suport-supervised pre-training methodologies by substantial margins. Compared to REFERS, MRM brings nearly 4% and 1 11% improvements to two tasks, respectively, further enhancing the practicability of the diagnosis system trained with extremely limited supervision.
4.5 ABLATION ANALYSIS
Advantages over single-task pre-training paradigm. First of all, we remove the two masked mod- eling objectives and super-resolution restoration task, resulting in substantial performance drops. These results verify the necessity of using masked modeling and super-resolution restoration in MRM. After removing the masked report modeling objective, MRM only acquires supervision signals from self-supervision. Thus, the whole framework degenerates into a self-supervised pre- training methodology. From Table 4 we observe that removing Lg leads to dramatic performance degradation in different labeling ratios. Moreover, we find that introducing the masked report mod-
8

---

Published as a conference paper at ICLR 2023
eling is greatly helpful to the fine-tuning performance with limited labeling resources % and 10%).. For instance, adding LR brings about 5-percent improvements to MRM. Removing the masked ra- diograph modeling also leads to notable performance drops in all labeling ratios.
Is super- restoration helpful? As Table 4 shows, the proposed super-resoution restora- tion provides consistent performance gains in different labeling ratios. The underlying reason may be that the low to high resolution restoration process helps preserve more local information into latent representations, which enhances the transferable ability to downstream tasks.
Would it be beneficial to introduce multi-modal information to image restoration? We investi- gate this question by adding non-masked report token embeddings image patch embeddings and passing the resulting hybrid features to the image decoder. As Table 4 shows, introducing multi- modal information to masked image restoration does not improve the fine-tuning performance. We leave the exploration of the reason behind to future work.
Ablations on multi-modal fusion and pyper-parameters. In Table Sa we find that global average pooling (GAP) outperforms global maximum pooling (GMP) by 2.2% when access to labeled data is quite limited while achieving comparable results as the labeling ratio increases. We also investigate the impact of applying different masking ratios to input radiographs (cf. Table 5b). Specifically, we find that a ratio of 75% performs the best on the extremely small labeling ratio (i.e.. 1%), while a ratio of 50% achieves slightly better results when the labeling ratio becomes larger. In MRM, we set the default masking ratio for radiographs to 75% because this operation leads to fewer input patches, accelerating the pre-training process and reducing the memory cost. The insight behind applying high masking ratio (i.e., 75%) is that it addresses the heavy spatial redundancy of radiographs. We also perform ablative experiments to investigate the influence of À in Eq. 3 whose results are displayed in Table Sc We see that À = 1 is an optimal choice, while a smaller À value (i.e, 0.5) performs better than larger value (i.e., 2.0), indicating that the MLM objective may play a more important role than the MIM objective during the pre-training stage.
4.6 IMPLEMENTATION DETAILS.
Our code implemented using PyTorch 1.8.2 Passke et al. 2019). The pre-training experi- ments were conducted on 4 GeForce RTX 3080Ti GPUs, and the training time is about days for 200 epochs. requiring 12GB memory from each GPU, The training batch size 256. We use AdamW (Loshchilov & Hutter 2017) as the default optimizer, where the initial learning rate is 1.5c -4, weight decay is 0.05, 3n 0.9, and 33 0.95. The MSE and cross-entropy losses are used for masked image and language modeling, respectively. In practice, we set À in Eq. to 1.
For fine-tuning on SIIM, we train the segmentation network on GeForce RTX 3080Ti GPUs. AdamW W is the default optimizer, where the initial learning rate is 2 weight decay is 0.05, 31 is 0.9, and 82 0.999. For fine-tuning on other datasets, we train the classification network on a single GeForce RTX 3080Ti GPU, where the default optimizer is SGD with momentum 0.9. The training cost is a mix of focal and dice losses.
For fine-tuning on CheXpert, RSNA Pneumonia, NIH ChestX-ray, and COVID-19 Image Data Col- lection, we adopt the cross-entropy loss. We search the best initial learning rate from 3e-2, 3e-3, and 5e-4 to get the best performance validation set.
For both the pre-training and the fine-tuning of image classification task, the network is "warmed up" by increasing the learning rate linearly to the set value, and then learning rate is decreased using the cosine decay schedule.
5 CONCLUSION
We present a unified yet simple framework based on masked record modeling (MRM) for radiograph representation learning. MRM formalizes the radiograph understanding and radiology report com- prehension as two complementary masked modeling objectives. In practice, MRM learns pre- traitra models that generalize well. With MRM pre- mr--trannig, achieve better results on well-established datasets. Specifically, MRM outperforms previous self- and report-superviser counterparts by large margins when the labeled data is extremely limited.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was supported by Hong Kong Research Grants Council through General Research Fund (Grant 17207722) and the National Key Research and Development Program of China (Grant STio030-Major #roject220212DD0219000
REFERENCES
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. Proceedings of the IEEE International Conference on Computer Vision, pp. 37-45, 2015.
Tarik Arici, Mehmet Saygin Seyfioglu, Tal Neiman, Yi Xu, Son Train, Trishul Chilimbi, Belinda Zeng, and Ismail Tutar. Mlim: Vision-and-languuae model pre-training with masked language and image modeling. arXiv preprint arXiv:2109.12178, 2021.
Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- Valle, et al. Making the most of text semantics to improve biomedical vision-language processing. arXiv preprint ar arXiv:2204.098:
2022.
Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayee Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical Image Analysis, 66: 101797, 2020.
Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueck- ert. supe-vupe learning for medical image analysis using image context restoration. Medical Image Analysis, 18:10,, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06799 067994 2022.
Joseph Paul Cohen, Paul Morrison, Lan Dao, Karsten Roth, Tim Q Duong, and Marzyeh Ghas- semi. Covid- 19 image data collection: Prospective predictions are the future. arXiv preprint arXiv:2006. 11988, 2020.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, pp. 248-258822 Icce, 2009.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11162-111 11173, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Associa- fion for Computational Linguistics, 2018.
Carl Doersch, Abhinay Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 1422-1430, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
10

---

Published as a conference paper at ICLR 2023
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al. Coarse-to- to-eettooeeeeeeee lisouallagguage pre-training with fusion in the backbone. arXiv preprint 2206:22000.77433 2022.
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, and Chunhua Shen. Pyramidelip: Hierarchical feature alignment for vision-language model pretraining. arXiv preprint arXiv:2204. 14092, 2022.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and Picter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205. 2tt.:2205.12244 2022.
Fatemeh Haghighi, Mohammad Reza Hosseinzadch Taher, Zongwei Zhou, Michael B Gotway, and Jianming Liang. Transferable visual words: Exploiting the semantics of anatomical patterns for self-supervised learning. IEEE Transactions on Medical Imaging, 40(10))2857-2860 2021.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Shih-Chema Huang. Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3942-3951, 2021.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik
Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 590-597, 2019.
Alistair EW Johnson, Tom Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth Berkowitz, and Steven Horng, Mimic-cxr-jpg, large publicly available database of labeled chest radiographs. arXiv preprint arXiv:190.000422, 2019.
Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. Masked vision and language modeling for multi-modal representation learning. arXiv preprint arXiv:2208 02131, 2022.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language- image pre- training paradigm. In International Conference on Learning Representations, 2021.
Ruizhi Liao, Daniel Moyer, Miriam Cha, Keegan Quigley, Seth Berkowitz, Steven Horng, Polina Golland, and William M Wells. Multimodal representation learning via maximization of local mutual information. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 273 273-283. Springer, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.051000 2017.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training, arXiv preprint arXiv:2112.12753 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga. et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8036-8037, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. $748-8763, PMLR, 2021.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus, Learning visual representations with caption annotations. In European Conference Computer Vision, pp. 153-170. Springer, 2020.

---

Published as a conference paper at ICLR 2023
George Shih, Carol C Wu, Safwan S Halabi, Mare D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya ing the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology. Artificial intelligence, 1(1) 2019.
et al. Augment-
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp-
15638-15650, 2022.
Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu. Multi-granularity cross-modal alignment for generalized medical visual representation learning. In Advances in Neural Information Processing Systems, 2023.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image a foreign language: Beit pretraining for all vision and vision-langen lssonnlanguage tasks. arXiv preprint arXiv:2208. 100222208.044,,,.. 2022.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794 2802, 2015.
Xiaosong Wang. Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: -ospital-scale chest x-ray database and benchmarks weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEÉE/CVF con- ference on Computer Vision and Pattern Recognition, pp. 2097-2106, 2017.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowl- edge enhanced language-image pre-training. medRxiv, pp. 2023-01, 2023.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint arXiv:2205.019177 2022.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis Langlotz. Con- trastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.007. 007777 2020.
Gangming Zhao, Quanlong Feng, Chaoqi Chen, Zhen Zhou, and Yizhou Yu. Diagnose like radi- ologist: Hybrid neuro- probbb-pbbaiiiiticc reasoning for attribute- medical image diagnosis. JEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 44(11):7400) (:710004477766 2022.
Hong- Yu Zhou, Shuang Yu, Cheng Bian, Yifan Hu, Kai Ma, and Yefeng Zheng. Comparing to
learn: Surpassing imagenet pretraining on radiographs by comparing image representations. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 398-407. Springer, 2020.
Hong- Yu Zhou, Chixiang Lu, Sibei Yang. Xiaoguang Han, and Yizhou Yu. Preservational learning improves self-supervised medical image models by reconstructing diverse contexts. In Proceed- ings of the IEEE International Conference on Computer Vision, pp. 3499-3509, 2021a.
Hong- Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, and Yizhou Yu. Generalized radiograph representation learning via coosssuuperiiiion between images and free- text radiology reports. Nature Machine Intelligence, 4(1):32 40, 2022.
Hong-Yu Yu Zhou, Chixiang Lu, Chaoqi Chen, Sibei Yang, and Yizhou Yu. A unified visual infor- mation preservation framework for self-supervised pre-training in medical image analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B Gotway, and Jianming Liang. Models genesis. Medical Image Analysis, 67:101840, 2021b.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DISCUSSION: DIFFERENCES FROM MASKED VISION-AND-LANGGUAGE TRE-TRAINING BESIDES MEDICAL DATA
To our knowledge, M3AE [Geng et al. 2022, MLIM Ariii et al. 2021 and MaskVLM wKn al. 2022) are three recent efforts that are most closely related to ours, all of which use masked modeling for vision and language pre-training. In the following, we clarify the differences between our MRM and these approaches from two perspectives: motivation and implementation.
Motivation. M3AE (Geng al. 2022) and MLIM (Arici et al. 2021) aim to learn joint image- text representations. MaskVLM Kwon et al., 2022) is developed for improving vision+language tasks, such as image-text retrieval, visual question answering, and natural language for visual rea- soning. These characteristics differ from our methodology, which aims to learn a representation for radiographs only for disease diagnosis even though both radiographs and reports are used during training.
Implementation. As aforementioned, M3AE (Geng et al. 2022 and MLIM (Arici al.) 2021) implement unified multi-modal transformers that take imaging and textual as inputs. As a result, they require much more pre-training data, which is often an order of magnitude larger than ours. This is not practical and applicable in the medical field, where access to the data is quite limited. MaskVLM (Kwon et al.) 2022) applies masked modeling and contrastive learning pairs, where a binary matching task is employed to tell whether an image and a text are aligned or not. Besides, MaskVLM builds cross-modality encoders incorporate multi-modee information. In contrast, our MRM is much simpler. MRM uses masking modeling as the only training objective, and simple GAP-addition workflow proposed for fusing multi-modal information.
image-text
A.2 CONFIGURATIONS OF NETWORKS
The image encoder ViT-like encoder (Dosovitskiy al. 2020) that includes patch embedding layer followed by twelve transformer blocks. The architecture of the image decoder is very similar to that of the encoder. The decoder architecture consists of decoder embedding layer, four trans- former blocks, and one fully-connected layer to predict masked patches. The numbers of attention heads in the image encoder and decoder are twelve and six. respectively. We add 2D sin-cos posi- tional embedding to input patches. The report decoder is weight transformer that includes an embedding layer and six transformer blocks, followed by a one-layer fully-connected predictor. The number of attention heads in the report decoder is six. We adopt a learnable positional embedding the report decoder.
A.3 RECONSTRUCTION ANALYSIS
Fig.1 3 presents example results MIMIC-CXR. We find that with high masking ratios, MRM can still produce satisfactory reconstructions, though some details are missing. Specifically, the re- constructed reports are surprisingly close to the ground truths, which we attribute to the introduction of hybrid multi-modaa representations. For instance, MRM can tell the position of rib fractures based on the radiograph input. Besides, we obtain some interesting mistakes. In the first example, MRM recognizes the clips over the left lung calcifications (potentially in nipple shadow). This observa- tion again shows that the report reconstructions rely on the image input as the clips are masked in the input radiograph.
A.4 SEGMENTATION ANALYSIS
In this section, we visualize the segmentation results of GLoRIA (Huang et al. REFERS (Zhou et al. 2022). and our MRM.
2021
13

---

Published as a conference paper at ICLR 2023
[CLS] (MaSE)] [MA54] (MAG) (MAS4] and (MAGK] indication with [MASS]
for infection
[MACKK]
Inputs
[Mass] (MACC) (MAC)) none |MASS]: i [MASS] (ooss] lozs (MAS)] [MAS))] pleurs (MAS) preumethens [MAS)] (asasa] [MASS]] (MaAC)] than (Monaa) kirly reprosent in pale chacnuus [MPCC JMASS, hosseee (MAAK]] (NoSS]]. [[ASS]] [MACI]] moe the left [MAc)]] patemnally [MA]O]] (MAC)) (aessa the (aoAsa upper
abdomen (MASK] [MASA]] [MASK] [MASI (MSSS] (MASS loft (MASSK) soverth (MASS]] ar IMAG (AASK] (MSSK, no scate carciopu monury procesa.
[tSS] final import formintion ctaed
indication Twith taae urset confisson for infection jechnsque gtaed leternl compen
Recons.
fistngg thene locel inmoliilation pleurall Affieen preumathores Bilateral heslidar npserties that mad Molyy reprisett nipple shadtnon. li
cartooeediectne shnnatte hurmel Eniiificciin Bmoee the potentisll stople shaeiing the imagedd widommn internriinnle besied
frachuei
left deth seventh
cortopulmmnnryy
[Cus] finll isport isamisation cheat so and Ri indication Twith arset ise If exal inrinfccionn techin quit: thest and aseral comparison
GT
findings there local consolidation plleural iffusan proumethoras bitateral nodider usucttis that maad Haaly reprisent nipple shadows. the
candomediestnne whourtte nurmel Eizee over the wit lung. patertially within the trmedd the imagad upper abdomen is intemarisable Shromd
delormby the postencr left voth and seowith tibs nseed impresui scute cediouulmmrrry
[CLS] [MASK] report (MASS [MASS] [MASK] [MASK] (MAS lot (MASK)] [MASK] (MASK] jM with shortness (MASK] (MASK] [MASR] [MASK] (MASK) and [MASK) comparioon I (MASE)] teading: (MWSS] rantor, mediediral and is (BAC0K] are normal. (MASK]] [MoSs] (MAX noraal langs are (NNNS]] [MAS]] p pleari (NANS))
Inputs
[MASS] pneumothones (MAX (MASK] multall cliss (MASR) again soon [MASS]] (MAS) the left (MASS) remote (MASS] sided fractures are aluo re [MASK]
inp esioon [MASE (MASS) candiopu Imonary abnormaliy Rty..
[a.s] Bod report commination thed 18 ind be Indicstion hidoor with sheetnees Smoath bechnique Steed and lnterdl comparison: finding:
Recons.
the cerliee medivttraa and tile pontoues are noemal Eutmonary insoulatinn normal lungs Best pleursl pitusin pre pre-mothorss present
nurtipe clps Bid rigiin seen Brojecting bvnl the balli remote lift sied fractures also Bemonstrated imprestoo: na sone droooimmoaayy abnormality
report
bechniquee
GT
mipieee clpe rgain projecting piomonary
lings pleurst
Impression cortopulmenaryy
abnormalty.
Figure 3: Example results on MIMIC-CXR. For each triplet, we show the masked radiograph and report (Inputs), our MRM reconstruction (Recons.), and the ground truth (GT). The masking ratios
are 75% (radiograph) and 50% (report). Predicted and corresponding ground truth words are high- lighted in pink and green respectively.
14

---

Published as a conference paper at ICLR 2023
Radiograph
GLoRIA"
REFERS
Our MRM
GT
15

---

Published as a conference paper at ICLR 2023
Radiograph
GLoRIA
REFERS
Our MRM
GT
Segmentation results of MRM, GLoRIA, and REFERS. GT stands for the ground truth masks. means the GLoRIA implementation is based ViT-B/16, the same backbone used in REFERS and MRM.
Figure 4:
16


Reviews:
[{"review_id": "nP36vnUHPmf", "reviewer": "Reviewer_Xf1P", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel approach for self-supervised learning from radiographs and associated medical records. The proposed multimodal method learns to predict masked inputs from both record (test) and radiograph (image) data.\n", "summary_of_the_review": "The paper seems like a straightforward combination of existing techniques, but generates compelling results on a variety of benchmarks. Novelty of combining MAE with BERT is already proposed by some papers, e.g. M3AE, and there is no clear and demonstrated improvement over these works, aside from the novel application to medical data. However, ablation experiments, discussion, and writing is clear and complete. \n", "strength_and_weaknesses": "Strengths\n- Paper is well written and easy to understand \n- A thorough review of literature is provided\n- Extensive experiments validate the proposed approach on the MIMIC-CXR (pre-training) and CheXPert, RSNA Pneumonia, NIH-Chest X-ray and COVID-19 Image Data Collection datasets (fine-tuning).\n- Compelling results are presented showing the computational improvements in AUC with the proposed method\n\nWeaknesses\n- Not clear why masking is necessary for images, or whether the task could be super-resolution only\n- Ablation studies with non-masking, non-super resolution pre-text tasks on records are missing\n- Experimental comparisons to other multimodal (language+image) methods, such as M3AE (Geng et al. 2022) are missing\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FgJYfY90fHC", "reviewer": "Reviewer_JZho", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses a novel method to enhance the pre-train chest x-ray image embeddings using the associated report. They train two masked auto-encoders (one for reports and the other for images). They then combine the image patch embeddings with report embeddings to decode the masked report tokens. At the same, they also decode a high-resolution image using the same image patch embeddings. They compare their method with existing methods and show that they outperform other methods by a considerable margin on multiple datasets.", "summary_of_the_review": "In summary, I found the method introduced in the paper to be interesting, novel, and elegant. The results across multiple datasets and downstream tasks demonstrate that the method is better than other methods at learning insights from reports to improve image embeddings. There are some important ablation experiments missing from the paper which would only strengthen the paper. I, therefore, give a recommendation of marginally above the acceptance threshold. ", "strength_and_weaknesses": "Strength(s):\n1. The method is novel and explained well. \n2. The method outperforms other methods by a considerable margin on multiple datasets thereby signifying the method efficiently utilizes the signals from reports to learn better image embeddings.\n3. The ablation experiments of different components nicely demonstrate how each component contributes to the performance.\n\nWeakness(es)/Suggestion(s):\n1. It would be nice to have confidence intervals/ SD around the metrics used to compare different methods.\n2. Why GAP was used to share image information with the masked record decoder? What happens if we use other alternatives like Max pooling, or have a transfer module that takes in embedded patches and returns a global embedding for the image?\n3. How is the lambda parameter in equation 3 decided? \n4. Some ablation studies to investigate how well the model learns for different masking ratios and masking patterns would be interesting for the readers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CrfsOkO9DOo", "reviewer": "Reviewer_UVnZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on building better models for unsupervised pretraining of deep neural networks for chest xray analysis.  This is achieved combining multi-modal information, i.e. radiological reports and chest xray images, and defining an unsupervised training procedure that predicts simultaneously masked parts of the images/reports given the non-masked part (masked record modelling). \n\n \n\nThe learned representations can be used to initialize networks for downstream tasks (classification, segmentation), and allow to outperform competing methods even when using a much lower amount of labelled data. \n\n ", "summary_of_the_review": "Overall an interesting paper with practical applications. I believe it can have an impact in the medical imaging community. ", "strength_and_weaknesses": " \nOne of the biggest issues when developing algorithms for chest xray analysis is access to labelled data, since it is expensive for radiologists to label tens of thousands of images. Most often, labels are  then generated with rule-based NLP algorithms. These labels are however very noisy, both because of the complexity of the text data to analyse, but also because in many cases radiologists might not write in the reports about diseases that are present in the image but are not clinically relevant for the patient. \n\nThe method discussed in this paper allows to consider an alternative approach: learn transferable radiography representations using image+report data, and fine-tune the networks for downstream tasks using a low amount of labels. \n\nThe experimental section is extensive, and shows increased performances compared to related methods. Importantly for practical purposes, thanks to the pretrained model, good classifiers can be built only using 10% of the labels. \n\nAll the ablation studies I had in mind while reading the MRM section were done by the authors, which show that all components of the proposed model are necessary.  \nIt would be interesting for me to see how the model performs with different amounts of masking probabilities. In particular I am surprised that the model performs well even when removing 75% of the patches. \n\nThe structure of the experimental section is a bit confusing the way it is split now in \"Baselines\" vs \"Results\". For example, after reading about the tasks and baselines in section 4.3.1, it would make more sense for the reader to directly see their results (section 4.5.1). \n\n\n_Typos_: \n* Second last paragraph of introduction: base->based \n* First sentence of section 3: learns -> learn ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8kyg3zdB4OL", "reviewer": "Reviewer_35Lo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new approach to learning radiograph representation through a joint masked image and language modeling. Specifically, for each pair of radiograph and radiology report pair, both the image and the report are masked for restoration tasks. The masked image patches are encoded and used for both restorations, while the masked text embedding is only used for report restoration. Besides, the input image is downsampled which combines a super-resolution task. The model is pre-trained on the large MIMIC-CXR dataset and then fine-tuned for downstream tasks: classification and segmentation on CheXpert, NIH ChestX-ray, RSNA Pneumonia, COVID-19 Image Data Collection, and SIIM-ACR Pneumonia Segmentation datasets. Experiment results show the proposed method outperforms all comparison methods, including report-supervised, self-supervised, and transfer learning methods. Ablation study experiments also provide an analysis of contributions from individual components in the proposed method.", "summary_of_the_review": "This paper combined masked image and language modeling for radiograph representation learning and showed significant improvement over existing methods for downstream tasks. The weaknesses of this paper could be addressed with some more effort.", "strength_and_weaknesses": "Strength\n\n- The paper is well-written and easy to follow. The proposed method is explained well with text descriptions and figure illustrations.\n- The proposed method, Masked Record Modeling (MRM), combining masked image and language modeling, is effective for learning radiograph representations. MRM outperforms state-of-the-art both report-supervised and self-supervised / transfer learning methods by large margins in some cases.\n- The ablation study shows the contributions of individual components in the proposed method. Specifically, it shows masked language modeling has a significant impact on improving radiograph representation learning.\n\nWeakness\n\n- Some technical details need more clarification. What is the lookup table for report token embedding? What are the specific configurations of the image encoder, image decoder, and report decoder?\n- In Table 2, MRM outperforms REFERS in most categories, except for “Infiltration”. Why? It is worth conducting a more in-depth analysis.\n- In Section 4.4, “The cross-entropy and MSE losses are used for masked image and language modeling, respectively.” The order of two losses should be swapped.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster