A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis
Authors: Yue Yang
Mona Gandhi
Yufei Wang
Yifan Wu
Michael S Yao
Chris Callison-Burch
James Gee
Mark Yatskar
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Robustness
Interpretability
Domain Generalization
Knowledge Prior
Medical Images
Primary area: interpretability_and_explainability
TL;DR: We tackle robustness to domain shifts in medicine by incorporating knowledge priors from documents via inherently interpretable models.

Abstract: While deep networks have achieved broad success in analyzing natural images, when applied to medical scans, they often fail in unexcepted situations. We investigate this challenge and focus on model sensitivity to domain shifts, such as data sampled from different hospitals or data confounded by demographic variables such as sex, race, etc, in the context of chest X-rays and skin lesion images. A key finding we show empirically is that existing visual backbones lack an appropriate prior from the architecture for reliable generalization in these settings. Taking inspiration from medical training, we propose giving deep networks a prior grounded in explicit medical knowledge communicated in natural language. To this end, we introduce Knowledge-enhanced Bottlenecks (KnoBo), a class of concept bottleneck models that incorporates knowledge priors that constrain it to reason with clinically relevant factors found in medical textbooks or PubMed. KnoBo uses retrieval-augmented language models to design an appropriate concept space paired with an automatic training procedure for recognizing the concept. We evaluate different resources of knowledge and recognition architectures on a broad range of domain shifts across 20 datasets. In our comprehensive evaluation with two imaging modalities, KnoBo outperforms fine-tuned models on confounded datasets by 32.4% on average. Finally, evaluations reveal that PubMed is a promising resource for making medical models less sensitive to domain shift, outperforming other resources on both diversity of information and final prediction performance.

A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis
Yue Yang, Mona Gandhi, Yufei Wang, Yifan Wu, Michael S. Yao, Chris Dallion--Burch, James C. Gee, Mark Yatskar
University of Pennsylvania yueyang1996. .it..gi............ .ocibiio//o iorikoo//oooo
Abstract
While deep networks have achieved broad success in analyzing natural images, when applied medical scans, they often fail in unexpected situations. We investigate this challenge and focus on model sensitivity to domain shifts, such as data sampled from different hospitals or data confounded by demographic variables such as sex, race, etc, in the context of chest X-rays and skin lesion images. A key finding we show empirically that existing visual backbones lack an appropriate prior from the architecture for reliable generalization in these settings. Taking inspiration from medical training, we propose giving deep networks a prior grounded in explicit medical knowledge communicated in natural language. To this end, we introduce Knowledge-enhee eenanned Bottlenecks (KnoBo), a class of concept bottleneck models that incorporates knowledge priors that constrain it to reason with clinically relevant factors found in medical textbooks or PubMed. KnoBo uses surieval-augmenteed language models to design appropriate concept space and an automatic training procedure for recognizing the concept. We evaluate different resources of knowledge and recognition architectures broad range domain shifts across 20 datasets. In our comprehensive evaluation with two imaging modalities, KnoBo outperforms fine-tuned models on confounded datasets by 32.4 % on average. Finally, evaluations reveal that PubMed is a promising resource for making medical models less sensitive domain shift, outperforming other resources on both diversity of information and final prediction performance.
Introduction
Robustness domain shifts is key property for models operating medical images because transfer scenarios arise widely. Deep networks have achieved broad success in analyzing natural images (everyday human contexts), but when applied medical scans, they often substantially degrade under distribution shift [5). Medical datasets are small, and unidentified confounds in them combined with model misspecification can dramatically degrade performance (46] 24) Such failure erodes confidence as models do not learn the right information from training data, hampering adoption by medical professionals. We study such problems by investigating the performance of systems in the presence of confounded data and address a main shortcoming we discover.
Model sensitivity to domain shift can be measured by introducing synthetic confounds into data and evaluating on samples where the confound misleads the model. For example, in Figure T we introduce confounded datasets for chest X-ray and skin lesion images where, during training, positive data is sampled from one group and negative from another. This association reversed at testing time, creating adversarial out-of-distribution (OOD) evaluation. In 5 such constructed confounds per modality, covering scenarios of race, sex, age, scan position, and hospital, we find models unable generalize well, dropping over 63% on average over an in-distribution (ID) evaluation.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Medical Images
Confounded Datasets
A
OOD
Av Skin Lesion
X-ray
ID
OOD
ID
OOD
100
100
Accuracy (%)
75
80
Chest X-ray
50
141.8
122.9
60
Skin Lesion
Race
Hospital
VIT
KnoBo
40
VIT
KnoBo
Figure I: In-domain (ID), out-of-domain (OOD), and average of ID and OOD (Avg) performance on confounded medical image datasets. Our interpretable Knowledge-enhancee Bottlenecks (KnoBo) are more robust to domain shifts (e.g., race, hospital, etc) than fine-tuned vision transformers 17
Priors are an important signal allowing models to adopt appropriate hypotheses in low or misleading data regimes. We hypothesize that existing visual backbones lack an appropriate prior for robust generalization in medicine. Like previous work identifying that vision backbones have deep image prior even when entirely untrained [72 78
we compare the quality image representations produced by untrained networks on natural versus medical images. Given the output from frozen untrained visual backbone, we train a linear classifier for predicting a diversity of labels (see Figure 2). Across architecture, these untrained models are higher quality featurizers of natural images than directly using pixels as features. In contrast, across multiple medical modalities, the deep image prior in current major visual backbones is more effective than using pixels (and often worse).
To address the lack of an effective deep image prior for medical images, propose using an inherently interpretable model design. We draw inspiration from medical education, where students first learn from textbooks and later more practical setting during the residency with an attending doctor. Our models mimic this pattern: first, documents are used to identify important knowledge, and then they learn by example from data. We employ concept bottleneck models (CBMs) [1 and enrich them with information derived from resources broadly accessible to medical students. CBMs are a class of inherently interpretable models that factor model decisions into human-readable concepts that combined linearly. Our methods build on recent approaches for language model (LM) guided bottleneck construction where LMS are prompted for discriminative attributes [90].
We introduce Knowledge-enhance encourage reasoning with factors found medical documents. KnoBo extends CBMs to medical imaging and employs retrieval- augmented generation into concept design. For example, we extract concepts from medical textbooks natural language questions like Is there ground-glat opacity? to help the model classify whether X-ray is positive for respiratory infection. As illustrated in Figure KnoBo factors learning into three parts: (1) an interpretable bottleneck predictor, (2) a prior over the structure of the bottleneck, and (3) a prior over predictor parameters. This factorization allows us to guide the model with a prior rooted in medical documents. The approach relies on an iterative retrieval process where LM summarizes documents to propose concepts, forming medical image prior (Sec 4..2. Given the concepts, a pretraining corpus of reports and images is used to construct classifier for concept (Sec 4.3 Finally, CBM is learned using predictions from classifiers on data while regularized by a prior formed from LM generations. (Sec 4.4)
Bottlenecks (KnoBo) to incorporate knowledge priors that
We evaluate KnoBo on our benchmark of confounded tasks. Averaged over confounds. KnoBo increases OOD performance by 41.8% and 22.9% on X-ray and skin lesion datasets, respectively. KnoBo's success in OOD performance comes at little sacrifice in ID settings, providing better model overall when averaging the two data settings. We also explore different sources of knowledge and reveal that PubMed outperforms other resources in terms of both diversity of information and final prediction performance. Overall, our work demonstrates that a key missing ingredient for robustness to distribution shift in medical imaging models is prior rooted in knowledge.
Natural Images
X-rays
Skin Lesion Images
Accuracy (%)
40
30.7
70
53.2
47.6
50.6
75
61.6
55.8
61.5
20.5
37.9
20
16.0
35
27.8
VIT Classification performance on natural and medical images through linear probing using features extracted from untrained and frozen models versus pixels features (See Sec for details).
Random
Pixel
CNN
VIT
Random Pixel
CNN
VIT
Random
Pixel CNN
Figure 2:

---

2 Related Work
The rapid advancement in medical foundation models offers the opportunity to develop healthcare
AI [55 49 However, their lack of transparency presents risks in real-world applications [5].
Interprefability is crucial for using models in high-stakes domains [30 79 87 Previous work
has primarily focused on post-hoc interpretability [76] 95 29 75), which may not provide faithful
explanations [69]. As an alternative, inherently interpretable methods produce explanations that align with the model's reasoning processes [ 3 this work, we build upon Concept Bottleneck Models (CBMs) 41), which predict by linearly combining suman-designed concepts. Recent work 191] 60 90] scales the applications of CBMs by aligning concepts and images with CLIP [64] and prompting language models to generate concept bottlenecks automatically. In our work, we treat CBMs as the architecture to incorporate knowledge priors for mitigating medical domain shift problems. Our bottlenecks, built from the medical corpus, are attributable and more trustworthy.
Domain Generalization and Robustness are critical in medical domains where the distribution of imaging protocols, devices, and patient populations can significantly vary [25]. A line of work
studies various domain-shiff shift problems 42 21 proposing algorithms learn invariant representations
93 44 (96.. However, many studies show those methods do not improve over standard Empirical Risk Minimization (ERM)
57
[22
80
63 and employing domain/group information for reweighting [71
67
26
3
[77] Fine-tuning the last layer [68] 40 or selectively fine-tuning few layers 47 is
sufficient for robustness against spurious correlations in those datasets. We address domain shifts in medical imaging from a novel perspective by employing interpretable models to integrate knowledge priors. Our approach encourages models to adhere to diagnostic rules similar to those doctors use rather than relying spurious correlations. Concurrent work shows bottleneck models can perform well on out-of-domain X-ray data but severely reduced in-domain performance as a consequence (89). In contrast, we demonstrate significantly better compromise between OOD and ID performance, using a broader set of modalities and constructing our bottlenecks from medical documents.
Knowledge Rich Multimodal Reasoning. Knowledge plays an important role in clinical diagnosis 6 Some multimodal tasks 82 74 require models to use explicit outside knowledge to make correct predictions. Previous methods [52
50 31 retrieve documents for each example from the external knowledge base as context for models to generate the answer. Our work focuses on leveraging knowledge in medical image classification. Retrieval-Augmenter Generation [48] 23 has been shown to be beneficial for mnowledge-intensin retrieved medical documents either used context during inference [88] data for pretraining
tasks 38), including biomedicine
20 84 The
[94]. In contrast, we treat documents as background knowledge for large language models to build concept bottlenecks. Instead of retrieving documents for every imput, we build global knowledge prior from a medical document corpus, which is shared across all examples.
3 Deep Image Priors for Medical Images
This section revisits the concept of deep image priors 72 78 i.e., some data-agnostic assumptions from model structure, in the context of image classification across various domains. By comparing linear prohing using features extracted by untrained deep networks against pixel-based features, we observe that existing vision backhones lack suitable priors for medical domains. This observation motivates our wwwwledge-enhancee bottlenecks (Sec 4 to integrate more robust priors into models.
Setup. Consider a dataset of imaee-label pairs, D = { {1, v) where is an image and E y denotes the label from one of N classes. The model learns to predict P (v]T, 0).. where 0 is the model parameters. We employ a frozen, untrained vision backbone V to extract features from I, producing a feature vector (f), where E Rdd A linear mapping function fo Ed - 3) is then trained to classify these features into label spaces. this case, the model parameters will inherit the implicit architectural priors of V. As bascline, we extract subset of d pixels directly from the image as the feature without any model-based priors, represented as EP E Rdd We compare the classification performance using versus Tp to probe the efficacy of the vision backbone's 's priors.
Experiments. We evaluate two state-of-the-art vision backbones, VIT-L/14 [17 and Lext
51
on three categories of images: natural photos (e.g.. ImageNet (70), X-rays (e.g.. NIH-CXR
83 ). and skin lesion images (e.g., HAM10000 (77)). Each image category has datasets, and we
report their average performance in Figure (see Table II in the Appendix C. for full results).
3

---

Query: How diagnose COVID from X-rays?
there ground-glass opacity?
retrieve relevant documents
retrieval augmented
Clinical Report
concept generation
Yes
No
Medical Corpus
Influctun HINI reapinatary infection:
posilive
negative
Pub Med
frqquett rudiological
jonerts were ground spacities and peiteoncccral murkings
STATPEARLS
WIKPEEDA
Learnod
Is there collapse of the lung?
W
COVID
Grounding
the trachea in the midline?
B
0.7
Align
Is there ground-glass opacity?
95
0.9
Is there a gas bubble present?
04
Predefined
Input Medical Image
Structure Prior
Bottleneck Predictor
Parameter Prior
Figure 3: Overview of Knowledge-enhanced Bottlenecks (KnoBo) for medical image classification,
comprising three main components: (1) Structure Prior (Sec 4.2) constructs the trustworthy knowl-
edge bottleneck by leveraging medical documents; (2) Bottleneck Predictor (Sec (.3) grounds the images onto concepts which are used as input for the linear layer and (3) Parameter Prior (Sec
4.4) constrains the learning of linear layer with parameters predefined by doctors or LLMs.
Figure 2 (left) shows vision backbones have effective priors for natural images, with VIT notably outperforming pixel by 14.7%. However, pixel features surpass those extracted by vision backhones for specialized domains such as X-ray and skin lesion images. This underscores these deep networks' lack of image priors appropriate for these domains, which can hamper model learning and hurt generalizability, Without guidance from appropriate priors, models can overly rely on data, risking catastrophic failures. aim to overcome this by injecting additional priors into models.
4 Knowledge-enhaneed Bottlenecks
In this section, we present Knowledge-enhancee Bottlenecks (KnoBo), a class of CBMs that incorporate knowledge priors that address the failures we identified in Section 3 Figure presents an overview of our method. From left to right, we optimize three terms: (1) Structure Prior (Sec 4.2)
induces bottleneck structures from medical corpus to incorporate human knowledge as concepts, (2) Bottleneck Predictor (Sec4.) projects input image onto bottleneck concepts and then feed concept predictions into the linear layer for label prediction, and (3) Parameter Prior (Sec (Sec|4.4) aligns the learned parameters with known associative information to further enhance priors.
4.1 Problem Formulation
Preliminary on Concept Bottleneck Model. Given bottleneck C with No concepts, CBMs optimize two functions for predictions: ÿ = (G(z)), where g IRd - R-Ne maps image features into concept space, and f Ne - y uses concept predictions for final label predictions.
Formulation. Our goal is incorporate priors over C. the concept structure, into the learning of the joint probability E(r.)) log (y, C, ([1). which can be decomposed into three factors:
log P Cy,C, log P (CC)+ + log (0) + +log P (,),,
structure prior parameter prior
botteneck predictor
where we assume the priors over structure C and parameters 0 are independent. The structure prior P(C)) (Sec 4.2p is formulated as the construction of a bottleneck with No concepts, C
(ei, C2, CNe) }. derived from a background corpus B. Each concept a factor that humans will use when solving the same task. The bottleneck predictor P T, C. (Sec 4.3) is concept bottleneck model that predicts the label conditioned on the input image, bottleneck, and learned parameters. The bottleneck predictor is inherently interpretable, and each parameter in A has semantics, denoting association between concepts and labels. The parameter prior P(0) (Sec[4.4) regularizes learning of model parameters 0 with information derived from human knowledge. Jointly optimizing over structure and model parameters intractable, so we first select high-quality concept space and then optimize the parameters of the bottleneck jointly with the parameter prior.

---

4.2 Structural Prior
Given background corpus B that spans various documents, aim identify bottleneck structure containing con- cepts beneficial for classifying labels y E y. As outlined in Algorithm [1 we use the class names y initial queries to retrieve relevant documents B' C B. Large Language Models (LLMs) are then prompted generate concepts us- ing these retrieved documents as context: C' = LLM These newly generated concepts are added to the bottleneck and used as new queries to retrieve additional documents. This iterative process continues to expand the bottleneck until a predetermined number of concepts Ne is reached. Such concept structures have high likelihood under the lan- guage model, conditioned on the background corpus, and the language model probability serves an implicit prior.
Algorithm Retrieve Augmented Iterative Concept Bottleneck Generation
V. set of target class names
B, set of background documents
Q y, class names as initial queries
C - [] concepts in the bottleneck
while |C| < Ne do
Q + [] set of new queries
for q 2 do
B' Retrieve (B, 9)
LLM (B')
C + C+C 인 + Q'
end for
Q x- Q' update queries #/
end while
4.3 Bottleneck Predictor
With the structure prior from the background corpus, we optimize (1) the grounding function G: _ + Re, which maps the input image to the concept space, and (2) a linear layer f R. + +y that projects concept predictions onto labels. In practice, we implement & as a set of grounding functions: g = fecI where each Ic predicts the probability of an image / having the concept Pccl)) = gr. (z). and æ E Rd is the image feature. Specifically, we derive training examples for grounding functions from pretraining dataset of image-text pairs. We the language model to estimate the presence of a concept in the image based on the information in the accompanying text.
Concept Grounding. Suppose have pretraining dataset Pre of image-text pairs { (L, where t a textual description (such as clinical report) of the image. Based on f, we can infer a concept is present in the image. This can be automated by prompting large language model to generate response indicating whether the text implies the concept. This way, we label our pretraining data as positive and negative examples for each concept c, which can be used to train its grounding function. With those annotated training examples, we implement each grounding function as binary logistic
regression classifier: (x( = x .. where W Rdd is the weights of grounding function and is the sigmoid activation. Finally, we form a collection of grounding functions G {ge)oEc to map an image feature into No probabilities over all bottleneck concepts, with ((x) E Re,
Linear Layer. Using concept probabilities G(x) as input, we train simple linear function f to make the final label prediction: f (G(=z) ((x) W where W R NxN is the linear weight matrix, with the number classes and No the number of concepts.
4.4 Parameter Prior
The bottleneck predictor is inherently interpretable because the parameters the linear layer encode the affinity between labels and concepts. Therefore, can guide the parameters based on prior knowledge, i.e., if the label y is positively related to concept c based on background knowledge, the weight Wy.c W should be high. We hope the learned parameters do not deviate too much from this assumption, otherwise, the model may capture spurious correlations in the data.
To enforce this, we let language models define weight matrix of priors W prior E Nx with each element Wy.c E {-1, +1} indicating the sign of a preferred correlation between the label y and concept The prior loss is calculated the LI distance between between W and W prior:
Cptior
N No tanh (W) W prior
which we apply tanh activation on W to scale the linear weights to (-1,1), matching the seale of the weights in the prior matrix. This adjustment aligns the model's parameters with the expected sign of the correlations based on prior knowledge. The final loss function to train the linear layer the sum of the cross- cross-entropy loss and the prior loss: L Lce + Epror-
5

---

In summary, we search for a structure C' that is consistent with prior knowledge from background corpus B severe as the bottleneck for the predictor (ylI, The parameters 0 are aligned with the predefined correlations between labels and concepts identified by language models.
5 Experimental Setup
This section introduces (1) the confounded and unconfounded medical datasets to evaluate the robustness of our knowledge-enhanced bottlenecks (Sec|5.1) (2) the black-box and interpretable baselines for comparison (S2c)5.2), and (3) the implementation details of our method (Sec 5.3
5.1 Datasets
We evaluate two groups of datasets for each modality: (1) the confounded datasets, which aim to assess the robustness of models by creating splits with spurious correlations; (2) the unconfounded datasets are randomly split to measure the models' performance in natural settings.
Confounded Datasets. As illustrated the left of Figure D formulate the confounded datasets as binary classification tasks, where each class is confounded with one factor. The confounding combinations are reversed for in-domain (train and validation), and out-of-domain (test) splits.
The confounded datasets of chest X-ray are constructed from NIH-CXR 83 and CheXpert (35) with their provided attributes: (1) NIH-sex uses sex (male, female) the confounding factor; (2) -IH-age confounds the data with age (young, old); (3) NIH-pos analyzes the patient's 's position (standing, lying down) during X-ray examinations: (4) CheXpert-race splits the data based patient's 's race (white, black or African American); (5) NIH-CheXppe confounds X-rays across datasets (NIH, CheXpert).
The confounded datasets of skin lesion are derived from the International Skin Imaging Collaboration (ISIC) (1) ISIC-sex and (2) ISIC-age are set up similarly to the X-ray datasets mentioned previously; (3) ISIC-site studies lesions developed on different sites of the body (head, extremities); (4) ISIC- color evaluates examples with different skin colors (light, dark); and (5) ISIC-hospital uses instances sampled from hospitals in different cities (Barcelona, Vienna).
Unconfounded datasets. We evaluate 10 datasets with random splits. 5 for each modality. X-ray: Pneumonia [39]. COVID-QU (9). NIH-CXR (83), Open-i [16], and VinDr-CXX CXR (58). Skin Lesion: HAM10000 77 BCN20000
PAD-UFES-20 [62). Melanoma [3)] and UWaterloo 45
All datasets are split into train/validattinntess and ensure the validation and test set are balanced across classes. Detailed statisties and additional information each dataset are provided in Appendix A
Pretraining Datasets. The training of vision backbones and concept grounding functions utilizes datasets with image-text pairs. For X-rays, we choose MIMIC-CR (37), which contains 377, 10 X-ray images with accompanying clinical reports. Since there is no existing text-annotated dataset for skin lesion images, employ GPT-4V [61] to generate captions (see examples in Figure 9 for subset of 56,590 images from ISIC without overlap the confounded and unconfounded datasets.
5.2 Baselines
We compare KnoBo against both black-box models and interpretable concept bottleneck models.
Black-box Models. We include two end-to-end fine-tuning baselines: (1) VIT-L/14 17 and (2) DenseNet121 [32], both pretrained on the pretraining datasets mentioned earlier. Additionally, (3) Linear Probe extracts visual features with the frozen ViT-L/14 encoder and learns linear layer for classification. (4) Language-shaped Learning (LSL) [56] aims to disentangle the impact of knowledge and interpretable structure. Inspired by LSL via captioning, we finetune ViT-L/14 with the same data used for concept grounding functions and apply a linear layer (see Appendix B.2.
Concept Bottleneck Models. (1) Post-hoc CBM (PCBM-h) 91] ensembles concept bottleneck models with black-box residual predictors. We let PCBM-h use the same bottlenecks as our KnoBo method; (2) LaBo (90) applies language models to generate concepts, followed by the submodular selection to identify subset that enhances performance. Following their original settings, PCBM-h and LaBo use CLIP (fine-tuned on medical pretraining datasets) to align concepts with images.

---

NIH-sex
NIH-
NIH-pos
CheXpert-race
NIH-CheXpert
Method
OOD
Avg
ID
OOD
Avg
ID
OOD
Avg
ID
OOD
Avg
ID
OOD
Avg
ViT-L/14
97.0
30.9
64.0
97.4
3.2
50.3
99.7
2.7
51.2
89.4
48.2
68.8
99.9
0.1
50.0
DenseNet
91.4
32.1
61.8
90.6
15.6
53.1
99.3
1.0
50.2
85.0
55.4
70.2
99.9
0.2
50.1
Linear Probe
94.2
46.7
70.5
95.0
11.4
53.2
99.3
17.0
58.2
87.8
71.4
79.6
99.6
6,8
53.2
LSL
84.0
74.3
79.2
79.8
53.8
66.8
95.3
39.0
67.2
80.4
76.4
78.4
95.0
31.8
63.4
PCBM-h
94.2
45.6
69.9
95.0
10.8
52.9
99.3
17.0
58.2
88.0
71.4
79.7
99.6
8,2
53.9
LaBo
91.4
51.3
71.4
92.8
14.4
53.6
98.0
24.3
61.2
86.8
69.2
78.0
98.4
14,9
56.7
KnoBo (ours)
88.6
78.6 83.6
88.8
38.8
63.8
95.7
45.3 70.5 84.0 79.0 81.5
91.6 52.3
72.0
ISIC-sex
ISIC-age
ISIC-site
ISIC-color
ISIC-hospital
Method
ID
OOD
Avg
ID
OOD
Avg
ID
OOD
Avg
ID
OOD
Avg
ID
OOD
Avg
VIT-L/14
92.0
69.0
80.5
95.0
61.3
78.2
94,8
38.3
66.6
96.9
59.2
78.1
99.2
10.0
54.6
DenseNet
85.3
76.0
80.7
93.7
61.3
77.5
81.7
54.5
68.1
93.9
44.6
69.2
98.4
15.1
56.8
Linear Probe
86.0
69.7
77.8
92.7
60.7
76.7
90.2
37.2
63.7
90.8
65.8
78.3
100.0
27.1
63.6
LSL
82.7
78.3
80.5
90.3
66.0
78.2
84.3
50.2
67.3
87.3
73.1
80.2
99.6
27.9
63.8
PCBM-h
86.7
69.0
77.8
93.0
59.3
76.2
90.0
38.5
64.3
91.2
66.5
78.9
100.0
26.8
63.4
LaBo
83.0
69.3
76.2
91.3
61.0
76.2
88.0
39.3
63.7
86.9
78.9
82.9
100.0
8.6
54.3
KnoBo (ours)
84.0
79.7
81.8
88.0
67.7
77.8
80.7
58.8
69.8
89.2
75.8
82.5
88.2
77.5
82.9
Table 1: Results on 10 confounded datasets two modalities (top-5 are X-ray and bottom-5 are skin lesion). We report in-domain (ID), out-of-domain (OOD), and average of ID and OOD (Avg) accuracy. The best score of each column bold, and the second best is underlined.
All baselines use backbones trained on the same pretraining data as our method to ensure a fair comparison. Appendix B.2 provides additional details about the baselines.
Evaluation Metrics. We use accuracy as the metric since all evaluated datasets are single-label classification tasks with balanced validation and test sets. For confounded datasets, report in- domain (ID, validation), out-of-domain (OOD, test), and domain-average (mean of ID and OOD) accuracies, along with domain gaps (A [ID OOD), where a lower A indicates better robustness. For unconfounded datasets, we report test accuracy. A robust and performant model must achieve good compromise between confounded and unconfounded datasets. For all the baselines and our KnoBo method, the checkpoints with the highest validation accuracy are evaluated on the test set.
5.3
Implementation Details
Pretraining of Medical CLIP. We fine-tune OpenCLIP 34 (ViT-L/14 pretrained on LAION-2B
(73) the pretraining medical data for each modality. Unlike previous work [I8 85 92 that directly pairs medical images with sentences from clinical reports, we preprocess the reports by employing GPT-4 I extract short phrases. Our CLIP models perform the best for both X-ray and skin lesion datasets in zero-shot and linear probing, as shown in Table in the Appendix.
Medical Corpus. We download 5.5 million articles from PubMed and segment them into 156.9 million snippets to serve as documents for retrieval. Alternatively, we take the medical corpus organized by MEDRAG [88), including documents from Wikipedia, SStateaarss] and medical textbooks. We employ BM25
(66) as the ranking function for document retrieval.
as the underlying LLM for concept generation (Sec 4.2). For training concept grounding functions (Sec 4.3), we opt for Flan-T5 XXL [0] to annotate clinical reports for each concept, considering cost-efficiency. Unless otherwise specified, KnoBo uses bottlenecks constructed from PubMed, each with 150 concepts. Figure shows our prompt, and during concept generation, we apply several heuristic filters (Appendix B.3)
KnoBo Details. We select GPT-4
6 Results
In this section, we discuss KnoBo's performance on confounded and unconfounded medical image datasets 6.1 and analyze diffferntt knowledge resources and our model design (Sec
(ec6.2 6.2
7

---

Chest X-ray Datasets
Skin Lesion Datasets
Method
ID
OOD
|
Avg
Unconfd
Overall
ID
OOD
A
Avg
Unconfd
Overall
ViT-L/14
96.7
17.0
79.7
56.8
70.2
63.5
95.6
47.6
48.0
71.6
84.3
77.9
DenseNet
93.2
20.9
72.4
57.1
66.0
61.5
90.6
50.3
40.3
70.4
71.0
70.7
Linear Probe
95.2
30.7
64.5
62.9
73.8
68.4
91.9
52.1
39.8
72.0
82.8
77.4
LSL
86.9
55.1
31.8
71.0
67.0
69.0
88.9
59.1
29.8
74.0
77.2
75.6
PCBM-h
95.2
30.6
64.6
62.9
74.7
68.8
92.2
52.0
40.1
72.1
81.7
76.9
LaBo
93.5
34.8
58.7
64.2
72.1
68.1
89.9
51.4
38.4
70.6
80.0
75.3
KnoBo (ours)
89.7
58.8
30.9
74.3
73.1
73.7
86.0
70.5
14.1
78.3
78.1
78.2
Table 2: Averaged results across all datasets, including in-domain (ID), out-of-domain (OOD), domain-gap gap (A, lower is better), and mean of ID and OOD (Avg) accuracy for confounded datasets. For unconfounded datasets (Unconfd), we report test accuracy. Overall performance is calculated as the mean of the Avg and Unconfd, the overall tradeoff between data conditions.
Knowledge Source
Chest X-ray Datasets
Skin Lesion Datasets
Confd
Unconfd
Overall
Diversity
Confd
Unconfd
Overall
Diversity
PROMPT
72.9
72.8
72.9
0.542
78.4
77.0
77.7
0.332
TEXTBOOKS
72.0
72.9
72.4
0,585
77.5
78.3
77.9
0.350
WIKIPEDIA
72.8
72.7
72.8
0.542
77.6
77.9
77.8
0.356
STATPEARLS
73.4
72.0
72.7
0.598
77.1
79.1
78.1
0.379
PUBMED 74.3
73.1
73.7
0.619
78.3
78.1
78.2
0.341
Table 3: Comparison of concept bottlenecks built from different knowledge sources. PROMPT is our baseline without retrieving documents for concept generation. We report the accuracy of confounded (Confd, average over ID) and OOD), unconfounded (Unconfd) datasets, and the overall performance of all datasets. Diversity measures the difference between the concepts in bottleneck.
6.1 Main Results
KnoBo is more robust to domain shifts. I the results 10 confounded datasets of X-ray and skin lesions. Black-box models excel at in-domain (ID) data but drop significantly on out- of-domain (OOD) data, especially in datasets confounded by hospitals/resoorces (NIH-CheXperr and ISIC-hospital), which can be common when collecting medical datasets [12] 81). KnoBo outperforms baselines in OOD and domain-average accuracy by large margins, ranking top-1 in eight datasets and second-best in the other two. End-to-end models (ViT-L/114 DenseNet) exhibit larger domain gaps than linear probes, as they have more parameters to optimize performance on in-domain data and capture spurious correlations. Shaping the visual representations with knowledge (LSL) improves robustness but underperforms KnoBo, with lower ID, OOD, and average performance across most datasets. PCBM-h combines interpretable and black-box predictions but exhibits behaviors similar to black-box models with severe drops across domains. Unlike KnoBo, which uses medical documents to create one global bottleneck for each modality, LaBo builds a bottleneck for each dataset using the in-domain data, which can be biased and affected by confounding factors and performs more poorly. In summary, KnoBo mitigates the catastrophic failures in domain shifts encountered by black-box and is more robust against various confounding factors across modalities.
KnoBo performs the best across confounded and unconfounded data. Table 2 illustrates the performance averaged across confounded and unconfounded datasets. For both types of medical images, KnoBo achieves the best out-of-domain (OOD) and domain-average performance (Avg) with minimal domain gaps (A), outperforming the strongest end-to-end baseline (ViT-L/14) by 41.8% (X-ray) and 22.9% (skin lesion) in OOD accuracy. KnoBo achieves competitive performance for unconfounded X-ray datasets, trailing the gest-performing black- model (Linear Probe) by only 0.7%. While KnoBo is less competitive on skin lesion datasets due to the lack of targe-scale pretraining data for accurate concept grounding, it still maintains performance comparable to the baselines. By calculating the mean accuracy across both confounded and unconfounded datasets, KnoBo ranks top across all models, confirming that our knowledge-enhance is a promising direction for building more robust and performant systems for medical imaging.
interpretable approach
8

---

Confounded (ID)
Confounded (OOD)
Confounded (Avg)
Unconfounded (Test)
70
80
Accuracy (%)
Accuracy (%)
Accuracy (%)
90
70
60
70
Aesuraa
50-
Knolo
80-
KnoRo (tpeiee)
KunBa (tpeior)
Knofo (tprioo) Lincse Probec
60
Knelto (tprior)
Linear Piile
Lincer Prube
Linzz Puiee
45
75
135
T0
45
105
135 60
75 105 135
15
45
105 135
# of Concepts/ Faatepps/Feattressss
#f Concepts/ eatures
# of Concepts/eeatures
of Concopts/Features
Figure 4: Ablation of bottleneck sizes on X-ray datasets. The x-axis the number of randomly selected concepts (KnoBo) or visual features (Linear Probe). +prior means adding parameter prior.
Chest X-ray Datasets
Skin Lesion Datasets
Method
ID
0OD
A 4
Avg
Unconfd
Overall
ID
OOD
A
+
Avg
Unconfd
Overall
KnoBo
89.7
58.8
30,9
74.3
73.1
73.7
86.0
70.5
14.1
78.3
78.1
78.2
w/o
87.8
51.5
36.3
69.6
70.1
69.9
83.7
69.4
11.5
76.6
70.2
73.4
w/o Epice
91.6
48.1
43.5
69.8
73.6
71.7
86.5
69.1
16.6
77.8
78.4
78.1
Table 5: Ablation studies concept grounding (9; 4.3)
and parameter prior (Pprior Sec 4.4
6.2 Analysis
In this section, compare the bottlenecks constructed from differentt knowledge sources. We evaluate the impact of each KnoBo component on the final performance, including bottleneck size, concept grounding function, and parameter prior. Additional analyses are available in Appendix C
Knowledge Sources. Besides the empirical results on confounded and unconfounded datasets, we i measure the diversity of bottleneck as Diversity (CrssittyCCC= = কলp-جc Eejec Layec (1- (1-iim(c;,,) cy)). where the sim(-) is the cosine similarity of concept features encoded by sentence transformer [65]. The Diversity computes the distance between each concept and every other concept in the bottleneck. Table comppars different knowledge sources. The than those generated by prompting, especially for skin lesions, where more specific knowledge is required because prompting lacks diversity. Across both modalities, PubMed is the best overall, performing better for the X-ray modality than other knowledge sources and among the best for skin lesion modalities. Moreover, shown in Table [ our retrieval- augmented concepts are attributable, which allows doctors to verify the source knowledge.
bottlenecks perform better
Human Evaluation on Bottlenecks. In evalua- tions by two medical students, information from all knowledge sources is rated as highly relevant and groundable. Two medical students evaluated the quality of bottlenecks using two metrics: (1) Relevance measures the concept's 's relevance to scale from (not all
Table 4: Relevance and Groundability of con- cepts in bottlenecks generated from different re- sources, evaluated by student doctors.
Knowledge Source
Relevance
Groundability
X-ray
Skin
X-ray
Skin
diagnosing diseases relevant) to 4 (mostly relevant), and (2) Ground- ability assesses the verifiability of the concept from the image on a scale from 1 to 4. We eval- uated 30 randomly sampled concepts from each bottleneck. [able[[ppreeee these metrics for bot- tlenecks constructed from five different knowledge sources. While all bottlenecks show good relevance, groundability scores are lower, reflecting the challenge of deriving visual concepts from text-only data.
PROMPT
3.83
3.93
3.03
3.00
TEXTBOOKS
3.70
3.80
2.90
3.27
WIKIPEDIA
3.80
3.67
2.83
3.33
STATPEARLS
3.87
3.80
2.70
2.97
PUBMED
3.70
3.83
2.77
3.20
Bottleneck Size. Figure compares KnoBo and linear probes while varying the number of con- cepts/features. KnoBo consistently outperforms linear probes across all metrics when given the same quota of features, and KnoBo can obtain better performance with fewer features. This indicates that interpretable concept scores have effective priors than black-box visual features.
Ablations. Table summarizes experiments ablating major components of our approach. Row 2 shows the performance of using dot-products from prompted CLIP models concepts, which

---

Bottleneck
Concept
Query
Reference Document
X-ray (PubMed)
Is there lung col- Atelectasis lapse?
Atelectasis and pneumonia were diagnosed radiological and clinical criteria. Atelec- tasis was diagnosed when a finding lung collapse was made on chest X-ray, chest CT and/or lung ultrasound. [Source]
X-ray (StatPearls)
Is there a widened Aortic Enlarge- mediastinum ment chest X-ray?
On chest X-ray (CXR), findings that may in- dicate aortic pathology include a widened mediastinum, loss of the aortic knob tour, inferiorly displaced left bronchus, and left pleural effusion. [Source]]
Skin (Textbook)
Does the lesion Seborrheic Ker- have waxy appear- atosis ance?
Lesions have no malignant potential but may be a cosmetic problem. Present as exo- phytic, waxy brown papules and plaques with prominent follicle openings. [In the book: First Aid Step-21
Skin (Wikipedia)
Are there small Basal Cell Car- blood vessels run- cinoma
BCC. also known as basal-cell cancer, the most common type of skin cancer. often appears as painless raised area of skin, which may be shiny with small blood vessels running it. [Source]]
ning over the skin lesion?
Table 6: We show concepts from various bottlenecks by image modality (corpus), with the queries for retrieval and the corresponding reference documents to generate the concept. Every concept is attributable, allowing medical professionals to verify its origin in the supporting documentation.
markedly reduces performance. This shows the importance of knowledge grounding in ensuring KnoBo's effectiveness. However, this step can be simplified more advanced medical foundation models are available. Row 3 shows performance omitting the parameter prior. It is an important mechanism for constraining the final learning phase, resulting in consistent OOD improvements. This is also reflected in Figure where with the parameter prior, KnoBo performs better on OOD splits while decreasing on in-domain as the parameter prior constrains the model rely on data.
7 Conclusion and Limitation
In this paper, we analyze domain-shift problems in medical image analysis and identify a missing medical deep image prior main contributor to poor performance. To address this, we introduce knowledge-enhane enaanneedeeeee bottlenecks (KnoBo) to integrate knowledge priors from medical documents. Across two medical image modalities under various domain shifts, KnoBo significantly improves robustness over black-box baselines.
Limitation. KnoBo assumes the availability of medical multimodal datasets, limiting applications to rare conditions. While our work improves robustness, medical experts do not fail in these ways, and they should be used in conjunction with models. Our preliminary work with PubMed suggests it is valuable resource for developing medical models, and future research can explore how to utilize such fruitful knowledge resources more effectively.
Acknowledgement
This study was conducted at the University of Pennsylvania School of Engineering and Applied Science and supported in part by the Office of the Director of National Intelligence (ODNI), Intel- ligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022- 22072200005, and gifts from the UPenn ASSET center and Ai2. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Govern- ment is authorized to reproduce and distribute reprints for governmental purposes, notwithstanding any copyright annotation therein. Michael S. Yao was supported by the National Institutes of Health (F30 MD020264). James C. Gee was also supported by the NIH (RO1 EB031722).
10

---

References
[1] Achiam, S. Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02899, 2019.
[3] A. Barnett, F. R. Schwartz, C. Tao, C. Chen, Ren, Y. Lo, and Rudin. A based interpretable deep learning model for classification of mass lesions in digital mammography. Nature Machine Intelligence, 3(12):1061- 2021.
[4] J. Bevan and A. Atapour-Abarghouss Detecting melanoma fairly: Skin tone detection and debiasing for skin lesion classification. In MICCAI Workshop on Domain Adaptation and Representation Transfer, pages 1-11. Springer, 2022.
[5] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint wxXi:2208.00255: 2021.
[6] H. P. Boshuizen and H. G. Schmidt. On the role of biomedical knowledge in clinical reasoning by experts, intermediates and novices. Cognitive science, 16(2):153-184, 1992.
[7] L. Bossard, M. Guillaumin, and L. Van Gool. Food-100 101-minng discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[8] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
[9] M. E. Chowdhury, T. Rahman, A. Khandakar, R. Mazhar, M. A. Kadir, Z. B. Mahbub, R. Islam, M. S. Khan, A. Iqbal, N. AI Emadi, et al. Can ai help in screening viral and covid-19 pneumonia? leee Access, 8: 16133666-132676, 2020.
[10] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y.,, X. Wang, M. Dehghani, Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1-53 2024.
[11] A. Coates, A. Ng. and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 213.223. JMLR Workshop and Conference Proceedings, 2011.
[12] J. P. Cohen, P. Morrison, and L. Dao. Covid-19 image data collection. arXiv preprint vrXiv:2003.11591 2020.
[13] J. P. Cohen, J. D. Viviano, P. Bertin, P. Morrison, P. Torabian, M. Guarrera, M. P. Lungren, A. Chaudhari, R. Brooks, M. Hashir, and H. Bertrand. TorchXRay Vision: A library of chest X-ray datasets and models. In Medical Imaging with Deep Learning, 2022. URL https
//github com/ www.coommmmmmmmoocco.ee.ceommmoe htt///mmmmde//////mmmm http://mmmdmmoocccoeeeee torchxayyriiiin
[14] M. Combalia, N. C. Codella, V. Rotemberg, B. Helba, V. Vilaplana, O. Reiter, Carrera, A. Barreiro, A. C. Halper. S. Puig, et al. Ben20000: Dermoscopic lesions in the wild, arXiv preprint arXiv: 1908, 02288, 2019.
[15] A. J. DeGrave, J. D. Janizek. and S.-I. Lee. Ai for radiographic covid-19 19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610-61) 2021.
[16] D. Demner-Fushman S. Antani, M. Simpson, and G. R. Thoma. Design and development of multimodal biomedical information retrieval system. Journal of Computing Science and Engineering, 6(2):168-177 2012.

---

[17] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words; Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[18] S. Eslami, G. de Melo, and C. Meinel. Does CLIP benefit visual question answering in the medical domain as much as it does in the general domain? arXiv e-prints, art. 1rX1v:2112.13066, Dec. 2021.
[19] B. Fitzpatrick. The validity and practicality of sun-reactive skin types i through vi. Archives of dermatology, 124(6):8698871, 1988.
[20] G. Frisoni, M. Mizutani, G. Moro, and L. Valgimigli. BioReader: retrieval-enhancee text-to- text transformer for biomedical literature. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5770-5793, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguisties. doi: 10.1 1665///12022..emmlpmm 2022. emnlp mwin.aii..... 390
URL ltttttsss/aaaatttttteeeeeeeeeeee
.rg//
[21] J. Futoma, M. Simons, T. Panch, F. Doshi- and L. A. Celi. The myth of generalisability in clinical research and machine learning in health care. The Lancet Digital Health, 2(9): (489-e492, 2020.
[22] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky, Domain-adversarall training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
[23] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang. -eugeetal-augmenteed generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2023.
[24] W. Gichoya, I. Banerjee, A. R. Bhimireddy, J. Burns, L. A. Celi, L.-C. Chen, R. Correa, N. Dullerud, M. Ghassemi, S.-C. Huang, et al. Ai recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health, 4(6) 14, 2022.
[25] H. Guan and M. Liu, Domain adaptation for medical image analysis: a survey, IEEE Transac- tions on Biomedical Engineering, 69(3): 11111133111111 1185, 2021.
[26] I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01334 2020.
[27] L. L. Guo. S. R. Pfohl, J. Fries, A. E. Johnson, J. Posada, C. Aftandilian, N. Shah, and L. Sung. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Scientific reports, 12(1): 2022.
[28] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-10344 2015.
[29] L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele, and T. Darrell. Generating visual explanations. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 3-19. Springer, 2016.
[30] A. Holzinger, G. Langs, H. Denk, K. Zatloukal, and H. Müller. Causability and explainability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1312, 2019
[31] Z. Hu, A. Iscen, C. Sun, Z. Wang. K.-W. Chang. Y. Sun, C. Schmid, D. A. Ross. and A. Fathi. Reveal: Retrieval-augeeeee netrieval-augmented visual-language pre-training with multi-source multimodal knowl- edge memory. In Proceedings of the IEEE/CVF conference computer vision and pattern 2023.
recognition, pages
[32] G. Huang, Z. Liu, L.. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
12

---

[33] B. Y. Idrissi, M. Arjovsky, M. Pezeshki, and D. Lopez-Paz. Simple data balancing achieves competitive sanst-rroupaaccurccy In Conference on Causal Learning and Reasoning, pages 336-351, PMLR, 2022.
[34] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openelip, July 2021. URL https: https://ddi.org// org/10. 10. 5281/2enode 5143773 If you use this software, please cite it as below.
[35] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-llcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597, 2019.
[36] M. H. Javid. Melanoma skin cancer dataset of 10000 images, 2022. URL https: www kaggle. com/dsv/3376422
[37] A. E. Johnson. T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng. Mimic-exr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
(38) N. Kandpal, H. Deng, A. Roberts, E. Wallace, and C. Raffel. Large language models struggle to learn long-tail knowledge. International Conference on Machine Learning, pages 156966 15707. PMLR, 2023.
[39] D. S. Kermany, M. Goldbaum, W. Cai, C. C. Valentim, H. Liang, S. L. Baxter, A. McKeown, G. Yang, X. Wu, F. Yan, et al. Identifying medical diagnoses and treatable diseases by image- based deep learning. cell, 172(5): 1125))1122-1111111 2018.
[40] P. Kirichenko, P. Izmailov, and A. G. Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.029:. 029337, 2022.
[41] P. W. Koh, T. Nguyen, Y. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang. Concept bottleneck models. In International conference on machine learning, pages 5338-5348. PMLR, 2020.
[42] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pages 5637-5664. PMLR, 2021.
[43] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[44] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville. -ut-of-distribbtion generalization via risk extrapolation (rex). In International Conference Machine Learning, pages 5815-5826. PMLR, 2021.
[45] V. I. Lab. University of waterloo skin cancer database, 2021. UtL/https://wwaerrooo ca/ vision- Dision-image- www.aiaproceeeeeeeeeeeeeeeeeeeee
cetcddddmmmsssssssee cancer detteerreeeettionn
[46] A. J. Larrazabal, N. Nieto, V. Peterson, D. H. Milone, and E. Ferrante. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences, 1 117(23): 12592-1229222 2020.
[47] Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn. Surgical fine-tuning im- proves adaptation to distribution shifts. International Conference on Learning Representations, 2023.
[48] P. Lewis. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal. H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. Retrieval-augmenn Advances in Neural Information Processing Systems, 33:9459-9447 2020.
generation for mnowledge-intensive nlp tasks.
[49] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao. Llava-med: Training a large language-and-visie vision assistant for biomedicine in one day. arXiv preprint 2306v: 00890, 2023.
13

---

[50] W. Lin and B. Byrne. Retrieval augmented visual question answering with outside knowledge. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1 1 1038-112544 1254, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguisties. doi: 10. 18653/v1/2022. amnlp-main.777 URL wtt.httsss//accaattooolooooooooe
org 2022. emnlp-main. 772
[51] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[52] M. Luo, Y. Zeng. P. Banerjee, and C. Baral. Weakly-supervisedd visual-retriever--eadee for knowledge-based question answering. In M.-F. Moens, X. Huang, Specia, and S. W.-t. Yih. editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6417-6431, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10. (0.18653/v1/202) wetpe//////...mlppppp. main.517. URL https://aclanthoolgg org/ 2021 emnlp-main. 517
[53] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[54] D. Mclnernyy, G. Young, J.-W. van de Meent, and B. Wallace. CHiLL: Zero-shot custom interpretable feature extraction from clinical notes with large language models. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Compulational Linguistics: EMNLP 2023, pages 8477-8494, Singapore, Dec. 2023. Association for Computational Lin- guistics. doi: 10. 18633/11/02223 00066663//2000..0nnn--mmmmmm.mm 2023. findings emm1p. emnlp.568
httpttt/////alaatoolllyy/ org/
[55] M. Moor, O. Banerjee, S. H. Abad, H. M. Krumholz, J. Leskovec, E. Topol, and P. Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 61 616(7956):259-265, 2023.
[56] Mu, Liang, and N. Goodman. Shaping visual representations with language for few-shot classification. arXiv preprint arXiv:1911.02683 2019.
[57] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In International conference muchine learning, pages 10-18. PMLR, 2013.
[58] H. Q. Nguyen, K. Lam, L. Le, H. H. Pham, Q. Tran,
Nguyen, D. Le, C. M. Pham, H. T. T. Tong, D. H. Dinh, C. D. Do, L. T. Doan, C. N. Nguyen, B. T. Nguyen, Q. V. Nguyen, Phan, A. T. Nguyen, P. H. Ho, D. Ngo. N. Nguyen, N. T. Nguyen, M. Dao, and V. Vu. Vindr-exr: An open dataset of chest x-rays with radiologist's annotations, 2020.
A. D. Hoang,
[59] M.-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing. pages 722-729. IEEE, 2008.
[60] T. Oikarinen, S. Das, I. M. Nguyen, and T.-W. Weng. Label-free concept bottleneck models. arXiv preprint 23X4v:2304.0611 2023.
[61] OpenAl. Gpt-4v(ision) technical work and authors., 2023. URL ctttss:/ccdm.ppnnt com/ contributions/gpt -pt pdf.
[62] A. G. Pacheco, G. R. Lima, A. S. Salomao, B. Krohling, I. P. Biral, G. G. de Angelo, F. C. Alves Jr, J. G. Esgario, A. C. Simora, P. B. Castro, et al. Pad-ufes-200 A skin lesion dataset composed of patient data and clinical images collected from smartphones. Data in brief. 32: 106221, 2020.
[63] F. Qiao, L. Zhao, and X. Peng. Learning to learn single domain generalization. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pages 2256-11265. 2020.
14

---

[64] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR, 2021.
[65] N. Reimers and I. Gurevych. Sentence- Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 11 2019, URL URL cattps://arxxt www.:/orgga///bbo 1908 10084
[66] S. Robertson, H. Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-38,, 2009.
[67] E. Rosenfeld, P. Ravikumar, and A. Risteski. The risks of invariant risk minimization. arXiv preprint arX1::2010.05761, 2020.
[68] E. Rosenfeld, Ravikumar, and A. Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.06886 2022.
[69] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, (55::20-2255, 2019.
[70] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Sathcesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-2523 2015.
[71] S. Sagawa, W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 1,0831,1 2019.
[72] A. M. Saxe, P. W. Koh, Z. Chen, M. Bhand, B. Suresh, and A. Y. Ng. On random weights and unsupervised feature learning. In ICML, volume 2, page 6, 2011.
[73] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25294, 2022.
[74] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv, 2022.
[75] R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient- gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618 626. 2017.
[76] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[77] P. Tschandl, C. Rosendahl, and H. Kittler. The ham 10000 dataset, a large collection of multi- source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
[78] D. Ulyanov, A. Vedaldi, and Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446-9454, 2018.
[79] A. Vellido. The importance of interpretability and visualization machine learning for applica- tions in medicine and health care. Neural computing and applications, 32(24): 12(24):18069-111 10189-6868888 18083, 2020.
[80] R. Volpi, H. Namkoong, O. Sener, J. C. Duchi, V. Murino, and S. Savarese. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31, 2018.
15

---

[81] L. Wang, Z. Q. Lin, and A. Wong. Covid-net: A tailored deep convolutional neural network design for detection of covid- 19 cases from chest x-ray images. Scientific reports, 10(1):19549, 2020.
[82] P. Wang, Q. Wu, C. Shen, A. v. d. Hengel, and A. Dick. Explicit knowledge-base reasoning for visual question answering. arXiv preprini arXiv: 1511002500 2015.
[83] X. Wang, Y. Peng, L.. Lu, Z. Lu, M. Bagheri, and R. M. Summers. Chestx-ray -ra8: Hospital-scale chest x-ray database and benchmarks on weakly classification and localization of common thorax diseases. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 2097-2106, 2017.
[84] Y. Wang, X. Ma, and W. Chen. Augmenting black-box Ilms with medical textbooks for clinical question answering. arXiv preprint SxXiv23009002333, 2023.
[85] Z. Wang, Z. Wu, D. Agarwal, and J. Sun. MedCLIP: Contrastive learning from unpaired medical images and text. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3876-3887, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10. 18653//1/022..emmlpp main. URL
emnlp-main main
256
[86] K. Wantlin, C. Wu, S.-C. Huang, O. Banerjee, F. Dadabhoy, V. V. Mehta. R. W. Han, F. Cao, R. R. Narayan, E. Colak, et al. Benchmd: A benchmark for modality-agnosse learning on medical images and sensors. arXiv preprint arXiv:2304.08888 2023.
[87] Y. Wu. Y. Liu, Y. Yang, M. S. Yao, W. Yang. X. Shi, L. Yang. D. Li, Y. Liu, J. C. Gee, et al. A concept-based interpretable model for the diagnosis of choroid neoplasias using multimodal data. arXiv preprint wixv:2403300660 2024.
[88] G. Xiong, Q. Jin, Z. Lu, and A. Zhang. Benchmarking retrieval-augmeente generation for medicine. arXiv preprint 2rX2v:2402. 13178, 2024.
[89] A. Yan, Y. Wang. Y. Zhong, Z. He, P. Karypis, Z. Wang, C. Dong, A. Gentili, C.-N. Hsu, J. Shang, et al. Robust and interpretable medical image classifiers via concept bottleneck models, arXiv preprint arXiv22310.03182 2023.
[90] Y. Yang, A. Panagopoulou, S. Zhou, D. Jin, C. Callison- Burch, and M. Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19187-19197, 2023.
[91] M. Yuksekgonul, M. Wang, and Zou. Post-hoc concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023. URL /ettps://openreiee. met/forum?id-nA5AAZ
[92] S. Zhang, Y. Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, Rao, M. Wei, N. Valluri, et al. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint vrXiv:2303.00913 2023.
[93] X. Zhang. Cui, R. Xu, L. Zhou, He, and Z. Shen. Deep stable learning for out-of- distribution generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5372-5382, 2021.
[94] X. Zhang, C. Wu, Y. Zhang, W. Xie, and Y. Wang. Knowledge-enhanced lanual-language pre-training on chest radiology images. Nature Communications, 14(1):4542, 2023.
[95] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
[96] X. Zhou, Y. Lin, R. Pi, W. Zhang, R. Xu, P. Cui, and T. Zhang. Model agnostic sample reweighting for outtt---iittibuttinn learning. International Conference on Machine Learning, pages 27203-27221 PMLR, 2022.
16

---

A Dataset
Datasct
Confounding of Factor
n. of Images
Class Names
Name
Class
train
val
test
NIH-sex
SCx
2
Atelectasis, Effusion
3000
500
1000
NIH-age
age
2
No finding, Has findings
3000
500
500
NIH-pos
position
2
Atelectasis, Effusion
3000
300
300
CheXpert-race
2
No finding. Has findings
4000
500
500
NIH-CHi AIH-CheXpert
dataset
2
Atelectasis, Effusion
5000
1000
1000
Pneumonia
39
2
Normal, Pneumonia
5216
16
624
COVID, Lung Opacity, Normal, Viral Pneumonia
COVID-QU 9
4
16930
2115
2114
Atelectasis, Cardiomegaly, Effusion Consolidation, Edema, Pneumonia
NIH-CXR 83
6
8066
1140
2317
Open-i 16
3
Cardiomegaly, Lung Opacity, Normal Aortic enlargement, Cardiomegaly, Pulmonary fibrosis, Lung Opacity, Pleural thickening, Nodule, Normal
884
438
890
VinDr-CXR
7
1400
175
175
Table Detailed statistics of the 10 Chest X-ray datasets evaluated in this work.
Dataset
Confounding n. of Factor
n. of Images train val test
Class Names
Name
Class
ISIC-sex
sex
2
Benign, Malignant
2400 300 300
ISIC-age
age
2
Benign, Malignant
2800 300 300
ISIC-site
body site
2
Benign, Malignant
2000 600 600
ISIC-color
skin color
2
Benign, Malignant
1800
260
ISIC-hospital
hospital
2
Benign, Malignant
2400 280 280
Actinic Keratoses, Basal Cell Carcinoma, Benign Keratosis-like Lesions, Dermatofibroma, 8010 Melanocytic Nevi, Melanoma, Vascular Lesions
HAM10000 7
7
1000 1000
Nevus, Basal Cell Carcinoma
BCN20000 44
4
4800 800 800
Melanoma, Actinic/Seborrhcic Keratosis
Basal/Squamous Cell Carcinoma,
PAD-UFES-20 62
2
1602
200 200
Actinic/Schorrhiic Keratosis
Melanoma 36
2
Benign, Malignant
8605 1000
1000
U'aterloo 45
2
melanoma, not melanoma
166 20
20
Table 8: Detailed statistics of the Skin Lesion datasets evaluated in this work.
Table 7 and g show the detailed statistics for all 20 datasets evaluated in this paper, where we list the number classes with class names, the number of images for training, validation, and testing. For confounded datasets, here explain some details about how define the confounding factors:
NIH-sex and ISIC-sex are built based on the patient sex from NIH-CXR and ISIC
NIH-age defines young as patient's age S 30 and old as patient's age
60.
NIH-pos refers the patient's 's position during an X-ray procedure. The standard position, Posterior Anterior (PA), involves the patient standing, while the Anterior-Posterti (AP) view is used when the patient cannot stand and must lie down.
CheXpert-race selects racial subgroups (White, Black African American) from CheXpert 35
NIH-CheXpert confounds classes by sourcing X-rays from either the NIH-CXR or CheXpert datasets. Specifically, data for one disease obtained from one dataset, while data for different discase is sourced from the other dataset.
17

---

Pneumonia
COVID-QU
NIH-CXR
Open-i
VinDr-CXR
Average
Model
LP
zs
LP
ZS
LP
ZS
LP
ZS
LP
ZS
LP
Random
50.0
50.0
25.0
25.0
16.7
16.7
33.0
33.0
14.3
14.3
27.8
27.8
DenseNet
83.8
82.9
53.0
63.8
27.4
62.2
OpenAI-CI.IP
62.5
6.7
91.0
35.0
49.6
21.6
60.3
16.0
33.1
28.4
63.3
OpenCLIP
62.5
77.4
6.3
90.0
7.5
47.9
21.6
58.7
15.4
34.9
22.7
61.8
PubMidCIIIP
63.3
72.9
22.0
87.7
30.4
47.7
26.4
59.6
15.4
26.9
31.5
58.9
BioMedCLIP
74.0
85,4
11.8
90.3
31.4
58.7
56.0
67.6
22.3
36.6
39.1
67.7
PMC-CLIP
57.7
84,9
48.1
94.9
40.4
60.6
S7.6
67.4
16.0
42.3
43.9
70.0
MedCLIP
84.9
89.9
68.6
87.4
25.1
64.8
70.2
71.9
24.6
40.0
54.7
70.8
Ours
77.7
88.6
60.6
94.9
47.4
68.4
67.5
73.3
29.1
44.0
56,5
73.8
Extraction
57.2
88.8
46.0
95.3
41.4
68.2
62.9
72.1
21.1
46.0
45.7
74.3
HAM10000
BCN20000
FFD-UFS-20
Mclanoma
UWaterloo
Average
Model
7S
I.P
7S
1.P
7S
I.P
7S
I.P
7S
I.P
7S
1.P
Random
14.3
14.3
25.0
25.0
50.0
50.0
50.0
50.0
50.0
50.0
37.9
37.9
DonseNet
79.0
69.6
69.5
91.9
45.0
71.0
OpenAI-CLIP
3.6
79.9
28.3
67.9
47.0
84.5
50.9
91.6
50.0
60.0
35.9
76.8
OpenCLIP
5.2
82.2
25.0
67.8
45.0
83.5
50.1
92.6
55.0
80.0
36.1
81.2
PubMedCLIP
76.3
28.9
64.4
50.5
85.5
48.8
92.2
50.0
60.0
36.6
75.7
BioMedCLIP
75.2
27.5
61.8
61.0
84.5
57.3
90.0
50.0
65.0
51.2
PMC-CLIP CLIP
25.1
82,4
24.8
67.6
55.0
86.0
66.1
92.7
55.0
55.0
45.2
76.7
McdCLIP
8.5
71.4
22.6
55.1
50.0
71.0
50.1
89.9
50.0
50.0
36.2
67.5
Ours
61.5
82,9
53.0
71.0
86.5
84.0
93.5
75.0
80.0
66.0
82.8
Extraction
50.9
83.3
46.5
72.0
52.0
86.5
80.1
96.0
70.0
70.0
59.9
81.6
Table 9: Zero-shot (ZS) and Linear Probe (LP) results of different models on five chest X-ray and five skin lesion datasets (not confounded, random split). The best score is bold, and the second best is underlined. Extraction stands for not using LLM to extract findings from clinical reports.
ISIC-age thresholds young as patient's 's age S 30 and old as patient's 's age 2 70.
ISIC-site focuses on lesions located either on the head or an extremity.
ISIC-color organizes images based on the Fitzpatrick scale of skin tones 19 Fitzpatrick is classified as light skin, and III, IV, and V as dark skin, according to the annotations from 4
ISIC-hospital introduces confounds in classes by using lesion images exclusively from either the Hospital Clinic de Barcelona the Medical University of Vienna.
B
Implementation Details
B.1 CLIP Pretraining
We experimented with existing CLIP models in the medical domain and found their performance to be unreliable. Therefore, we decide to train our own CLIP models for X-ray and skin lesion images.
Pretraining Dataset. For X-rays, we utilize the MIMIC-CXR dataset [37 specifically selecting only the
AP X-rays, which results in 243,334 images, each accompanied by a clinical report written by doctors. For Skin Lesion images, employ the ISIC dataset and use GPT-4V [6]] to generate clinical reports for 56,590 images, examples are shown in Figure 9 We preprocess these reports by extracting medically relevant findings, each described in a short and concise term. The example below demonstrates a report alongside the findings captured by GPT-4. In total, we assemble 953K image-text pairs for X-rays and 438K for skin lesion images.
FINAL REPORT HIS ISST Unresponsive. Evaluate pneumonia. COMPARISON:
Chest radiographs and CT thoracic spine the chest. The lung bibasilar atelectasis, left greater than right. Heart size is normal. Mediastinal and hilar structures are unremarkable. The contiguration trachea unchanged from prior croas-sectional imaging.
FINDINGS:
Portable frontal
No pleural effusion pneumothorax. There
IMPRESSION: Low lung volumes without an acule cardiopulmonary process. FINDINGS (GPT-4): low lung volumes, bibasilar atelectasis, left greater than right, normal heart size, trachea unchanged
18

---

Training Details. We utilize the training script from OpenCLIP 13 and select ViT-L/14 as the backbone. Training is performed on 4 RTX A6000 GPUs for 10 epochs with a batch size of 128 and a learning rate of 1e-5. We choose checkpoints based on the lowest contrastive loss on validation sets.
CLIP Baselines. We compare various CLIP models across unconfounded datasets for two modalities, including OpenAI-CLIP 664, OpenCLIP [34], PubMedCLIP (88), BioMedCLIP [92]. PMC-CLIF" and MedCLIP 855] We evaluate these models in both zero-shot and linear probe scenarios. In zero-shot, GPT-4 generates prompts for each class, and we use the ensemble of cosine similarities between the image and prompts as the score for each class. In linear probing, we use the CLIP models as image encoders to extract features for logistic regression. Additionally, we include DenseNet-122 121 [32 (fine-tuned on the pretraining datasets with cross-entropy loss) as a baseline for linear probing.
Results. Figure 9 shows that our CLIP models perform best in both zero-shot and linear probing scenarios for both modalities. We find that preprocessing the text data with an LLM significantly enhances zero-shot performance. Existing medical CLIP models outperform general CLIP models on X-ray datasets but not on skin lesion images, possibly because X-ray data are more prevalent and accessible in the medical domain. While our CLIP models excel with careful data curation, training converges quickly, suggesting the current contrastive objective might not fully exploit the information from the data, potentially taking shortcuts, such as comparing images from different patients instead of focusing on diseases. Future research should explore more suitable objectives and larger-scut scale data collections to develop more robust medical foundation models.
B.2 Baselines
This section outlines the implementation details of all baselines compared with our knowledge bottlenecks for medical image classification. All baselines are run on a single RTX A6000 GPU.
We utilize the visual encoders from the CLIP models we pretrained in Sec B.I and add a classification head for downstream classification datasets. We unfreeze the ViT-L/ backbone and train all parameters with learning rate of 1e-6 and a batch size of 64 for 20 epochs.
VIT-L/14:
DenseNet-122 Similarly to ViT-L/14, we add a classification head to the pretrained DenseNet and train the entire network end-to-end. For X-rays, we use the DenseNet pretrained on MIMIC-XR by TorchXRay Vision [33. For skin lesion images, we pre-train the DenseNet from scratch on ISIC using a cross-entropy loss. When fine-tuned for downstream classification datasets, we train the DenseNet with a learning rate of 1e a batch size of 64, and also for 20 epochs.
Linear Probe: We employ visual encoders (ViT-L/14) from pretrained CLIP models to extract features for images in downstream classification datasets. We train a linear layer to map these features into labels for 200 epochs with learning rate 1e and a batch size of 64.
We fine-tune the pretrained CLIP with contrastive loss on annotated concept data (PubMed bottleneck), using the same data as for concept grounding functions (4.3). Training instances are triplets (V), where I is the image, binary label indicating whether the image contains this concept. Given the visual encoder V and textual encoder T of the CLIP, the cosine similarity between image and concept is (7,
LSL (56)
textual concept, and y {0, 1} is
cos (V(1), T())). The contrastive loss function is defined as Leuntrast y max (0, m (L, (1-y) s (I, c), where for 20 epochs with a learning rate le -6 and a batch size of 64. After obtaining the fine-tuned CLIP, we extract features and train linear probe in the same manner the linear probe baseline.
0.6 is the margin. We fine-tune the CLIP with concept annotations
We use their codebases to implement these baselines. The concept alignment in both models is achieved using CLIP to compute the dot product between image and concept features. PCBM-h uses the same bottleneck as KnoBo, which is generated from PubMed. For LaBo, we employ GPT-4 to generate candidate concepts for submodular selection.
PCBM-h 99 and LaBo [90:
B.3 KnoBo Details
This section provides additional details about the implementation of KnoBo.
Medical Corpus. We utilize a comprehensive medical corpus for retrieval-augmenntdd generation, detailed as follows: (1) PubMed (5.5M docs, 156.9M snippets); (2) StatPearls (9.3K docs, 301.2K
19

---

You are an experienced radiologist (dermatologist) You are summarizing knowlsdge about QUERY from chest X-rays [skin lesion images]. Here the documents retrieved from the corpus:
RETRIEVED DOCUMENTS
want you to filter and summarize the information in these documents and genrerate knowledge in the form of "bi- nary questions", e.g. "Is there lung opacity?" ["Is the lesion asymmetric?"". Please follow instructions below strictly:
to guide diagnosis on chest X-rays [skin lesion images], they must "visually identiliable' from chest X-ray [skin lesion images] only.
Those knowledge
be
The binary questions should concise and not too specific which can reused for different cases.
3. The binary questions must not contain the class(disease) name, e.g. you "must not" generate "Is there cardioreggaly" I"Is the lesion malignant?"] as the knowledge for "Cardiomegaly" ['Malignant Lesion"). there
much information in the some documents, you ignore those documents..
the documents contain
useful information, you can skip this task by typing skip'.
Answer the following format: question document ID reference sentence, e.g., Is there lung opacity? | 1234 | lung 1234| | asymmetric lesion is common finding for ...]
opacity is common finding for [Is the lesion asymmetric?
Please answer without additional information and do not add numbers or bullet points in the answer.
Prompt template for retrieval-augmentee concept bottleneck generation. The text in the square brackets is words that need to be changed when using this prompt for skin lesion images.
Figure 5:
Chest X-ray Datasets
Skin Lesion Datasets
LLM
ID
00D
A +
Avg
Unconfound Overall
ID
OOD
A 4
Avg
Unconfound Overall
Flan-T5
89.7
58.8
30.9
74.3
73.1
73.7
86.0
70.5
14.1
78.3
78.1
78.2
GPT-4
89.9
56.8
33.1
73.3
72.9
73.1
86.0
71.6
14.4
78.8
78.5
78.6
Table 10: Comparison of using different LLM annotating concepts clinical reports.
snippets); (3) Textbooks (18 docs, 125.8K snippets); (4) Wikipedia (6.5M docs, 29.9M snippets). The StatPearls, Textbooks, and Wikipedia sources are obtained from MEDRAG (88). Unlike the abstract-only approach of PubMed in MEDRAG, we utilize full articles from PubMed, including all paragraphs. We employ the retrieval codebase of MEDRAG and select BM25 as the ranking function.
Retrieval-augmmenee Concept Bottleneck Generation. Figure illlstrates the prompt template we use to generate concepts from documents. We retrieve the top 10 documents for each query as context for the large language model (GPT-4) to generate concepts. After generating concepts, validate each concept based on three criteria before inclusion in the bottleneck: (1) the concept must be distinct from existing concepts; (2) it must be visually identifiable from the image; and (3) there must be sufficient positive and negative instances in the pretraining corpus to support training its grounding function. A concept is added the bottleneck only if meets all three criteria, as judged by another language model (GPT-4). We initially target 200 concepts per bottleneck but ultimately select the top 150 with the highest grounding accuracy for inclusion. This selection is due some concepts lacking sufficient reports to effectively train their grounding functions, making 150 the minimum size for all the bottlenecks we construct. Table shows examples of generated concepts.
Concept Grounding. We use a language model to annotate 2,000 clinical reports for each concept from the pretraining corpus. To efficiently label reports and achieve balance of positive and negative examples, we retrieve the top 1,000 reports showing high textual similarity (measured by Sentence Transformer (65)) to the concept as its potential positive examples and randomly sample another 1,000 for potential negatives. We use Flan-T5-2 TS-XXL [10] as the underlying large language model. Specifically, the annotation task is treated as next token prediction, similar to the approach proposed by McInerney et al. [54]. where we compare the probabilities of the next token being Yes or No to determine if the report contains the concept. Table 10 shows no big difference in final classification performance when using Flan-T55 versus GPT-4 for annotating concepts on reports.
C Additional Analysis
This section presents additional analysis and ablation studies on our method.
C.1 Details about Deep Image Priors
We provide further details about the deep image prior experiments in Sec
20

---

Natural Image Datasets
Feature
CIFAR-10
STL-10
ImageNet-10
Food-101
Flower-102
Average
Random
10.0
10.0
10.0
1.0
1.0
6.4
Pixel Value
21.2
24.7
22.4
3.0
8.5
16.0
ConvNext-LL*
25.6
29.1
32.6
3.7
11.3
20.5
ViT-L/14"
33.3
40.6
47.2
8.9
23.6
30.7
X-ray Datasets
Feature
Pneumonia
COVID-QU
NIH-CXR
Open-i
VinDr-CXR
Average
Random
50.0
25.0
16.7
33.0
14.3
27.8
Pixel Value
77.6
66.9
43.1
58.3
20.0
53.2
ConvNext-L."
62.5
59.2
38.9
57.5
20.0
47.6
ViT-L/14"
67.6
68.0
40.4
57.2
20.0
50.6
Skin Lesion Datasets
Feature
HAM10000
BCN20000
PAD-UFS-20
Melanoma
UWaterloo
Average
Random
14.3
25.0
50.0
50.0
50.0
37.9
Pixel Value
65.9
39.4
58.0
74.5
70.0
61.6
ConvNext-L."
66.9
37.1
53.5
68.3
50.0
55.8
ViT-L/14"
67.3
45.9
54.0
84.8
50.0
61.5
Table 11: Linear Probe results of different features on five natural image datasets, five X-ray datasets, and five skin lesion datasets. denotes the network is randomly initialized without any training.
Datasets. We evaluate the deep image prior on three categories of images. The X-ray and skin lesion
images are the same as those described in the unconfounded datasets section (Sec A). For natural images, we select five datasets: (1) CIFAR-10 10 43 (2) STL- I. (3) ImageNet-10 [70
Food-101 7. and (5) Flower-102 102 59
Setup. We employ the vision backbones VIT-L/14 17 and ConvNext-L. [51), both implemented by OpenCLIP [34], and initialize them using Kaiming initialization [28] following Py PoTch''ss '' default setting Both backbones extract feature vectors of size 768. For the pixel value baseline, convert the image to grayscale, resize it to 28 x 28, and then flatten into a vector of 784 dimensions. We the first 768 values of this pixel vector match the size of the deep features. All features passed through linear layer to predict the labels with learning rate of 1e 3 batch size of 64 for 200 epochs. Additionally, we include a random baseline for comparison.
Results. Table displays the full results all methods across the three image categories. ViT-I/14 excels natural datasets with significant gains over pixel baselines. ConyNext-L. performs than ViT-L/14 but is still notably more effective than pixel-based methods. For X-ray images, the pixel baseline clearly surpasses the networks on almost all datasets, indicating that deep models lack priors or even have harmful priors for X-ray imaging. For skin lesion images, which are closer to natural images, the pixel value baseline performs comparably to ViT-L/14 and better than Lex-Ne Overall, the results suggest that deep networks lack sufficient priors for medical domains, potentially affecting their generalizability.
C.2 Full Results on Unconfounded Datasets
Table 12 shows the comprehensive results of all baselines across the 10 unconfounded medical datasets. KnoBo performs competitively in the X-ray category, securing top-1 positions for two datasets and achieving an average ranking of third among all methods. For skin lesion datasets, KnoBo's performance is limited by the smaller scale and lower quality of the pretraining corpus, which is annotated by GPT-4V rather than by human experts. This affects the effectiveness of the grounding functions for lesion concepts. With access to larger-scale and higher-quality pretraining data, KnoBo could potentially close the performance gap with black-box baselines.
We use classes selected by Imagenette: http://gtthuh.oom//sssss//mgggnt https:/ppyoorch..oo////ssssssba html
21

---

Confounded (ID)
90
Confounded (OOD)
Confounded (Avg)
Unconfounded (Test)
80
80
Accuracy (%)
Accuracy (%)
KnoBo (ours)
Accuraey (%)
L Linear Probe
80
75 1 Coooanan
85
KnoBo (ours)
75
KnoBo (ours)
A
70
KnoBo (ours)
Linear Probe
Linear Probe
Linear Probe
80
45 75 10S 135
60
45
75
105
135
45 75 105 135 # of Concepts/eeature
15
65
15
45
75 105 135
Concepts/Features
of Concepts/Foeeeeeccccc eatures
of Concepts/Feature
Figure 6: Ablation of bottleneck sizes Skin Lesion datasets. The x-axis is the number of randomly selected concepts (KnoBo) or visual features (Linear Probe). We report the ID, OOD, and domain- average performance on confounded datasets and test the accuracy of the unconfounded datasets.
Chest X-ray Standard Datasets
Method
Pneumonia
COVID-QU
NTH-CXR
Open-i
VinDr-CXR
Average
VIT-L/14
84.6
96.5
67.0
37.1
70.2
DenseNet
85.1
92.4
56.9
63.5
32,0
66.0
Linear Probe
88.6
94.9
68.4
73.3
44.0
73.8
LSL
87.0
86.9
58.4
64.8
37.7
67.0
PCBM-h
87.7
94.9
68.6
73.2
49.1
74.7
LaBo
88.3
91.8
66.8
68.9
44.6
72.1
KnoBo (ours)
90.1
88.0
66.5
73.5
47.4
73.1
Skin Lesion Standard Datasets
Method
HAM10000
BCN20000
PAD-UFS-20
Melanoma
UWtterloo
Average
VIT-L/14
87.1
76.6
94.1
75.0
84.3
DenseNet
79.0
69.6
69.5
91.9
45.0
71.0
Linear Probe
82.9
71.0
86.5
93.5
80.0
82.8
LSL
81.5
67.5
84.5
92.5
60.0
77.2
PCBM-h
82.9
70.9
86.0
93.6
75.0
81.7
LaBo
80.6
68.5
82.5
93.6
75.0
80.0
KnoBo (ours)
78.2
65.6
80.0
91.5
75.0
78.1
Table 12: Test accuracy on 10 unconfounded datasets of two modalities.
C.3
Ablate Bottleneck Size for Skin Lesion
Figure 6 compares the performance of KnoBo and Linear Probe across different feature sizes skin lesion datasets. Mirroring the trends observed in X-rays shown in Figure 4 our interpretable bottleneck representations consistently outperform black-box visual features.
C.4 Data Efficiency for Concept Grounding
The concept grounding module of KnoBo requires image-report pairs from multimodal medical dataset. By default, we select 2,000 pairs for train- ing the grounding function for each concept. How- ever, this number can be largely reduced. Table 13 shows KnoBo's performance doesn't 't decrease too much with fewer training examples. With only 100 reports, KnoBo can obtain 72.1 overall accu- racy, compared to 73.7, of which 2,000 reports are used. It highlights that KnoBo achieves much of its performance with smaller number of exam-
Table 13: Ablate the number of clinical reports for training each concept grounding function.
Reports Concept
Confd
Unconfd
Overall
100
72.5
71.7
72.1
250
73.7
72.1
72.9
73.1
73.0
73.1
1000
73.1
72.8
73.0
1500
73.5
73.2
73.4
2000
74.3
73.1
73.7
ples, suggesting that is not hata-hungry and has the potential to applied to rare modalities.
22

---

Query
CLIP
Ours (w/ concept grounding)
Are lung fields clear both sides?
Top-3
Is there a visible enlargement of the heart? Does the lesion have regularly shaped
Top-3
Top-3
border?
Does the lesion show dry scaly areas?
Top-3
Figure 7: Compare CLIP and our concept grounding function in retrieving images based on a text query. Images marked with green checkmarks are correct retrievals as assessed by medical students.
Edemaa
Top-3 Concepts
Cardiomegaly
Top-3 Concepts
NII-CNK CXR
Are abnormalities observed in subpicural region?
Tianne X
Is there enlarged cardiac silhouetie?
Can increased cardiothoracic ratios be identified?
Are there new alveolar or interstitial infiltrates?
Docs the radiograph: show any large
any hazy opacity present?
transverse dimension heart?
Melunocytic Nevi
Top-3 Concepts
Melanoma
Top-3 Concepts
Melunocytic Nevl
Does the lesson appeur un elevated nodule?
Does the lesion present flat pigmented arca?
Does the lesson have asymmetric disposition?
Does the lesion adhere to the ABCD demoscopy rule?
brownish-black verrucose plaques be body?
the show ulceration crust?
Figure 8: Top-3 concepts for class ranked by their weights in the linear layer of KnoBo.
D Qualitative Examples
the CLIP dot product and our concept grounding functions for retrieving images based on text queries. Our method demonstrates superior recall of correct examples compared to CLIP alignments. Figure showcases the top concepts selected based on the linear weights learned by KnoBo, which are most correlated with the corresponding disease class. These examples highlight that the top concepts utilized by KnoBo are essential features doctors use for diagnosing targeted diseases. Table 14 displays examples of failed concepts identified by medical students during human evaluation. Each concept can traced back to source document to verify its accuracy. Figure presents examples of GiPT-4V-annttated clinical reports for skin lesion images. Notably, the second example demonstrates GPT-4V's capability to assess lesion size using the scale provided in the image.
23

---

Modality
Irrelevant Concepts
Ungroundable Concepts
Repetitive Concepts
Are there any subcutaneous emphysema?
Are there changes in appear- ance that develop slowly?
Is there evidence of ground- glass opacity?
X-ray
Can you spot elevated hems in the diaphragm?
Can worsening consolida- tion be observed?
Is there ground glass opac- ity in both lung fields?
Is the lesion located on the surface of extremities?
Is the skin around the lesion rough without pain?
Does the lesion have regular margins?
Does it have scales near the lesion?
Is the lesion larger than 6mm in diameter?
Does the lesion possess an irregular shape?
Skin
Are skin lesions involving the face and scalp?
Is there any change the size of the lesion over time?
Does the lesion have irregu- lar borders?
Table 14: We let medical students annotate 300 concepts and categorize the failure cases into three categories: (1) Irrelevant: concepts are not essential to diagnosis; (2) Ungroundable: concepts cannot be grounded only one image and (3) Repetitive: the concepts are repeated in the bottleneck.
Location: Lower extremity
Lesion Description: Macroscopic Characteristios: The lesion presents as solitary macule with relatively
homogenceus area centrally and slightly ieregular periphery.
Color: Tan to light brown background with several shades of light to dark brown pigmentation centrally. portions the lesion, with network fading towards edges. Structures and Patiern:: Focal areas exhibit a more globular pattem, while the majority of the lesion lacks specific structural components.
Networks: faint reticular pattern
Symmetry: The lesion exhibits asymmetry in boch color and srneerure. Borders: Borders are ill-defined, the pigment diffusing into surrounding skin.
The lesion presents as well-cireumseribed, ovoid macule measuring approximatety 2.7mm in its largest diameter. located on the posterior torso. Dermosoopiaall, it is characterized by homogenous pattern with a central brown pigmentation that shows subtle variegation color intensity. ranging from light darker brown. The peripheral area of the lesion transitions to a lighter, more (an hue, with discrete fading into the surrounding nermal skin without evideace of sharp barder.
Vascular structures are not prominent within the lesion itself; howover, the background skin exhibits reticular patrem of superfieial blood vessels which consistent with anatomical site and passibly the patient's age. There is no evidence of atypical pigment networks, whitish veils, regression structures, or other sinister fearures typically associated with malignancy.
The lesion appears as well- delnede,, round to oval macule with notable centrll darker area lighter peripheral It measures approsimatelly cm in diameter.
General Appearance:
Color: The lesion exhibits a helcrogencous mix of dark brown and tan shades. The central area is densely pigmented, sumoundddd by intensely colored periphery.
Texture: The surface texture cannot be preciscly determined from the image alone, but there appears to be no significant elevation depression compared the surrounding skin.
Distribution and Symmetry: The lesion is solitary with symmetrical appearance.
Additional Observation: No visible seales, ulcers, signs bleeding are present. Hair follicles can be observed within and around the lasion, sugaesting not disrupting normal skin structures significanlly.
Figure 9: Examples of clinical reports on skin lesion images generated by GPT-4V 661.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are limited to our specific experiments and focus the behavior our models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in the corresponding section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
The paper contains no theoretical results or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Major details are supplied the main body, while more minor ones are the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: The code is provided in the submission as a zip file, and we will release the code and data Git Hub after review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.cc/public/giee guwde//ddee/ee/ Cot//ooodeeeiiseeeeeeee wwwpibmmsssioooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All details for experiments are supplied in the main body and appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Figure and 6 have error bars. Broadly, the paper covers many datasets and reports each individually. This provides enough evidence about the variance of the approach. We do not rerun methods on each dataset as this would be computationally wasteful.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
27

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is provided in the appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Having reviewed the policy, we follow all bullets across the three areas.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This information is provided the last section of the paper in the main body.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put place for the responsible release of data or models with a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will release the data and models with appropriate warning that these systems should not be used for any direct diagnosis without randomized trials supporting their efficacy.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We cite all dependencies of our work and verify that using their data, models, and methods in a way consistent with their terms.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper documents our models, and our GitHub release will contain usage instructions. Novel dataset components will be released that are consistent with license instructions from base datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: There is no crowdsourcing or human subject research in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no crowdsourcing or human subject research in this work.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
30

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "e5C0NKRPfd", "reviewer": "Reviewer_z571", "summary": "This paper proposed Knowledge-enhanced Bottlenecks (KnoBo) to leverage the concept bottleneck model (CBM) to improve model robustness to various domain shifts and confounders. KnoBo explores using retrieval augmented generation to generate concept space with a large number of concepts using a large language model (i.e., GPT-4). KnoBo brings significant overall improvement when large-scale image-text pairs are available for concept grounding. A comprehensive evaluation across various scenarios and various baselines confirms the effectiveness of KnoBo in reducing the gap between in-distribution and out-of-distribution performance.", "questions": "1. In Table 1, the best performance of ISIC-age should be ViT-L/14 and LSL, not the proposed KnoBo (which is bolded). \n2. The KnoBo with 150 concepts has an average OOD Acc of 58.8 across Chest X-ray Datasets, as stated in Tables 2 and 4. Why does Figure 4 indicate that KnoBo with 135 concepts has an accuracy below 50 for OOD X-ray datasets? As the performance of OOD datasets decreases with the increased number of concepts, these two results seem contradicted. \n3. Typo in the title of Algorithm 1 in Section 4.2. Should be 'retrieval'", "limitations": "As explained in the Weaknesses and Questions sections.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper explores exploiting LLM and retrieval augmentation generation, which enables building large concept space from the large-scale knowledge base. \n2. Overall, the presentation is clear. Technical details are clearly presented in the supplemental material. However, some results are a bit confusing and lack discussion of limitation and failure cases. \n3. Evaluation is comprehensive.", "weaknesses": "1. The concept grounding seems pretty data-hungry, requiring large-scale multi-modal datasets for pretraining (e.g., MIMIC-CXR with over 300k image-text pairs). For smaller datasets (ISIC with about 60k images with generated captions from LLM), the proposed KnoBo only brings marginal overall improvement. This requirement significantly limits the broad application of the proposed framework in the data-scarcity domain, such as cancer/tumor classification. \n2. Lack of a detailed discussion of limitations (one one-line limitation in Section 7) and presentation of failure cases. No failure cases (wrong classification / wrong concept) were included in either the main text or supplemental material. \n3. Some results are confusing to the reviewer (See No. 2 in the Questions section).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. Here we address your comments:\n\n**Q1. The concept grounding seems pretty data-hungry.**\n\nThis is a valid concern, but while we sample from a large dataset like MIMIC, in reality, we only use 22k total examples. Training the concept grounding component is just learning a linear classifier, so we sampled **a small subset of the data**.\n\nOn average, each X-ray grounding function uses 1,523 examples, and each skin lesion grounding function uses 1,342 examples. This can be reduced significantly with little loss in performance. \n\nThe table below shows the performance of KnoBo on X-ray datasets using varied sizes of samples for training each grounding function. It highlights that KnoBo achieves much of its performance with a small number of examples, suggesting that it is not data-hungry.\n\n| n. of examples | Confounded | Unconfounded | Overall |\n|-------------------------|------------|--------------|---------|\n| 250                                      \t| 73.7   \t| 72.1     \t| 72.9\t|\n| 500                                    \t| 73.1   \t| 73.0     \t| 73.1\t|\n| 1000                                   \t| 73.1   | 72.8     \t| 73.0\t|\n| 1500                                   \t| 73.5   | 73.2     \t| 73.4\t|\n| 2000                                   \t| 74.3   | 73.1     \t| 73.7\t|\n\n**Q2. For skin lesion datasets KnoBo only brings marginal overall improvement.**\n\nEmpirical challenges within the skin lesions datasets are not because of a lack of data but because the datasets have very strong non-generalizable cues (skin colors, hairs, etc). For this reason, it is easy to fine-tune models to high in-domain performance and learn non-transferable predictors.\n\nOur overall metric averages both in-domain performance and out-of-distribution performance. KnoBo is much better out of distribution (see OOD and Avg in Table 2, shows differences of **+22.9** and **+6.7**, respectively), but when averaging in domain performance on datasets with no explicitly identified confounds (unconfd), helps more moderately (+.3). We believe that OOD measures more accurately capture if models are able to learn the correct hypothesis and therefore the more important measure of if a model is trustworthy in real applications.\n\n**Q3. Confusing results in Tables 2 and 4.**\n\nIn Figure 4, we deactivated the prior loss for KnoBo, which resulted in lower OOD performance. In this ablation, we focus on comparing different inputs (black-box features vs. concept scores) and their sizes while removing prior loss forms a more comparable setting. We provide the **updated Figure 4 in the PDF of the global response**, which includes the curves of KnoBo with prior loss. We will clarify this experimental setup in the final draft of our paper.\n\n**Q4. Failure cases and limitation section.**\n\nFailure cases are difficult to examine and generally not very accessible to non-medical professionals (in the main text, we tried to avoid adding examples that might make some readers uncomfortable if they are squeamish about medical images). Sections C.4 (Table 11) and Section D (Figures 8 and 9) of the appendix contain some of this, where we recruited medical students to evaluate limitations.\n\nThe medical students annotated 300 concepts and often found the information in our bottlenecks highly relevant, but some were not groundable (**see Table A in the PDF for examples**). This did not vary much by information source, but skin lesion information was judged to be easier to ground than X-ray information. In a possible camera-ready version, given the extra page, we can try to integrate some more of this content in the main body of text, including example failures, and expand the limitations section to more explicitly discuss the data efficiency of our methods.\n\n**Q5.Typos.**\n\nWe have carefully proofread the manuscript again to correct the issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0KMgicxeJe", "reviewer": "Reviewer_T8n7", "summary": "The presented paper addresses the challenge of domain shifts in medical image classification, where conventional neural networks often lack effective priors for medical datasets. The authors introduce KnoBo, a novel class of concept bottleneck networks that integrate medical knowledge priors to enhance neural network performance in medical image classification tasks. KnoBo has three primary components: 1) Structure Prior:  leverages medical documents to construct a knowledge bottleneck; 2) Bottleneck Predictor:  maps images onto concepts; 3) Parameter Prior: imposes constraints on learning the linear layer's parameters, ensuring that the network's predictions remain consistent with the established medical knowledge. The proposed method of KnoBo is evaluated using two medical datasets, ISIC and X-ray, under both confounded and unconfounded settings.", "questions": "- Whats is the main reason to define $ \\Delta = | ID - OOD |$ as robustness measure (row 257)? In my opinion, this metric is more related to some \"fair predictions\" concept, since it measures how ID and OOD performances differ.  \n- Do the authors know why the diversity values for skin lesion datasets (Table 3, last column) are much lower than those for X-rays?", "limitations": "Not a limitation, but I recommend incorporating a filtering scheme into the concept generation process to enhance the paper. While the authors have conducted human evaluations, relying solely on such assessments may not be feasible due to the limited availability of human resources. Refining the concept generation could be achieved by improving the prompts or employing a medical foundation model to assess the quality of the generated concepts about the target image.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-organized and easy to follow. The authors propose a novel approach to incorporating medical knowledge priors into existing neural networks by optimizing three distinct terms. The theory behind each term is solid and clearly explained, making the methodology easy to understand. One notable aspect is the parameter prior (Sec. 4.4), which adds interpretability by aligning the estimations with medical concepts. They also use human evaluation on the learned concepts. \n\nThe authors conduct a comprehensive set of experiments to evaluate their proposed method, covering several scenarios, including confounded and unconfounded data. This broad range of experiments supports the validity of their findings.\n\nThe ablation studies are thorough, encompassing five knowledge sources, appropriate baselines, and various bottleneck sizes. These ablations provide valuable insights into the proposed method's performance and sensitivity.", "weaknesses": "The paper's primary weakness is its reliance on content generated by LLMs to create new medical concepts. Although GPT-4 is one of the most advanced models for generating concepts, it can still produce hallucinations, especially when the generated concepts are loosely aligned with common terms such as hair, skin color/tone, gel bubble, and rules (in the context of skin lesions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work! Here is our reply to your questions:\n\n**Q1. Hallucinations of GPT-4**\n\nThis is a valid concern, but the way we use GPT-4 highly encourages it to directly copy information from documents instead of inventing it. Our concept generation is conditioned on medical documents, which differs from previous work [1] that solely relies on LLM. We employ LLM as a text extraction tool, which basically copy-paste or paraphrases. In addition, each concept is attributed to a document that medical professionals can fact-check. Table 12 of the Appendix and supplementary materials (HTML files in the `example_concepts/` folder) show example concepts with corresponding reference documents.\n\nWe examined concepts from the X-ray bottleneck. 80% of the concepts were word-for-word copies of a feature in a sentence that the authors of a document said they examined. 14% included some paraphrases but were still grounded in the document and never completely invented (sometimes a combination of multiple factors in different places in the document). 6% were of confusing origin and potentially hallucinated but still mentioned relevant factors. This aligns with the analysis of the bottlenecks we did with medical students in section C.4 of the appendix.\n\n**Q2. Robustness measure ($\\Delta$)**\n\nOur analysis is focused on multiple metrics at once. The delta measure $\\Delta$ needs to be combined with the other metrics (ID, OOD) when measuring models’ robustness (it is trivially minimized by creating classifiers with chance performance). Our perspective is that a robust model should have high performance across ID and OOD while maintaining a small drop to demonstrate its robustness to domain shifts.\n\n**Q3. Diversity values for skin lesion datasets are much lower than those for X-rays**\n\nThis is because X-ray is a more information-dense modality as it shows multiple organs (e.g., lung, heart, stomach). Skin lesion diagnosis requires fewer medical factors in practice. Moreover, X-ray is the most accessible medical imaging modality, resulting in more available medical documents.\n\n**Q4. Filtering scheme of concept generation**\n\nThank you for the suggestion! To clarify a bit, we don’t have a phase in our method that requires medical professionals, although we used them later to verify the validity of our approach. We designed three filtering criteria to ensure the concepts are diverse and visually identifiable. (Appendix B.3, Line 712-717) As you suggest, more filtering of the concepts using auxiliary signals would likely substantially improve the results and is something we are exploring.\n\n[1] Yang et al. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aBaTPTezkg", "reviewer": "Reviewer_f9Nf", "summary": "The authors noticed the domain-shift issue of current medical dataset, and after taking inspiration from medical training, they propose giving deep networks a prior grounded in explicit medical knowledge communicated in natural language. The proposed network can incorporate medical knowledge priors to help model make decisions. Basically, the authors assumed that “medical concepts” are robust enough to against the domain shift. Valid assumption, interesting work.", "questions": "1. “In 5 such constructed confounds per modality, covering scenarios of race, sex, age, scan position, and hospital, we find models unable to generalize well, dropping over 63% on average over an in-distribution (ID) evaluation.” What are the tasks? Please specify.\n\t2. “As a baseline, we extract a subset of d pixels directly from the image as the feature without any model-based priors, represented as xp ∈ Rd. We compare the classification performance using x versus xp to probe the efficacy of the vision backbone’s priors.” I didn’t get the point. Why did you want to compare the effectiveness of x and x_p? \n\t3. It is very hard to read the 4.2 section. I cannot fully understand the algorithm 1. Please raise examples for better reading experience.\n\t4. Table 1, why the KnoBo always has not the best, or sometimes the worst performance in in-domain dataset?\n\t5. Is the Structure Prior learnt? Or the questions like “Is there ground-glass opacity?” were pre-defined? How ist he connection between “Is there ground-glass opacity?” and the top-right red box in Figure 1?\n\t6. Is the training end-to-end? Can you please provide an example for 1) how to pretrain using medical books 2) and given an X-ray image, how the workflow is to get the last result? I hold positive thought towards the study, but it is really hard to understand the workflow.\n\t7. Is the parameter prior a part of the bottleneck predictor? In figure 1 it seems like parallel, but in the main text it seems like existing sequentially. Hard to understand.\n\t8. How to determine the best number of concepts?\n\t9. How did you determine the preferred correlation between the label y and concept c?", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors noticed the domain-shift issue of current medical dataset, and after taking inspiration from medical training, they propose giving deep networks a prior grounded in explicit medical knowledge communicated in natural language. The proposed network can incorporate medical knowledge priors to help model make decisions. Basically, the authors assumed that “medical concepts” are robust enough to against the domain shift. Valid assumption, interesting work.", "weaknesses": "It is very HARD to read and interpret. Please raise examples for better reading experience. Maybe I have some questions after the clearance. Other points could be seeen under the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback! We will try to include more examples in future versions. We hope the answers below clarify:\n\n**Q1. What are the tasks?**\n\nWe studied the medical image classification tasks in a confounded setting (the medical class names are found in Tables 5 and 6). As explained in the second paragraph of the introduction (Lines 30-34), the samples are confounded with different factors during training/validation (ID) and testing (OOD) time. For example, in an X-ray classification task to classify COVID and normal patients, if we confound the data with sex, the dataset is constructed as follows:\n\nTrain/Validation (ID): (COVID, male) (Normal, female)\n\nTest (OOD): (COVID, female) (Normal, male)\n\nA robust model should predict based on COVID-related features rather than spurious correlations (sex in this example).\n\n\n**Q2. Why compare the effectiveness of $x$ and $x_p$?**\n\nThis experiment aims to validate our motivation that deep neural networks lack priors for medical domains. $x$ is the feature extracted by a deep network at initialization (i.e., a ViT). $x_p$ is a feature vector constructed by extracting pixel values in the image. If a network has good priors for a domain, we expect the network output at initialization ($x$) to be a more useful representation of the image content than the pixels themselves ($x_p$). For natural images, this is known from prior work on Deep Image Priors, but, as we show in Figure 2, this is not true for medical images. Without good priors, models can easily adopt incorrect hypotheses and rely on spurious features. This leads to catastrophic failures when the spurious features are unavailable in out-of-domain tests and motivates the need to incorporate priors from elsewhere.\n\n**Q3. explain algorithm 1**\n\nAlgorithm 1 aims to collect a set of concepts from medical documents. Given the class name (e.g., COVID), we want to extract useful knowledge from medical documents as concepts (e.g., ground glass opacity, a highly indicative feature of COVID). The whole process is a retrieval-augmented-generation task. We first use class names as queries (e.g., COVID) to retrieve relevant documents from a corpus (e.g., PubMed). The retrieved document may contain useful information, e.g., in the upper left of Figure 3, the document says, “the most frequent radiological patterns found were ground-glass opacities.” We feed these documents as contexts for an LLM, which extracts the information as a concept (ground glass opacity). We repeat this process using generated concepts as queries until a predefined number of concepts is reached.\n\n**Q4. Why doesn’t KnoBo always have the best ID performance?**\n\nThis is because, in the confounded settings (explained in Q1), models can achieve high ID performance by taking shortcuts using spurious correlations, e.g., learning to classify males and females instead of focusing on diseases. Models that achieve high ID performance in this way will have catastrophic failures on the OOD test (see OOD numbers of ViT-L/14 and DenseNet in Table 1). In this case, a robust model must have a high average performance across ID and OOD. There is a natural tradeoff between ID and OOD, but our models strike the best balance.\n\n**Q5. Is the Structure Prior learnt? The connection between “Is there ground-glass opacity?” and the top-right red box.**\n\nAs explained in Q3, the structure prior is not learned from training instances but instead constructed from medical documents.\n\nThe top-right red box illustrates the concept grounding module (Sec 4.3, Lines 180-188). Given a concept (e.g., is there ground-glass opacity?) and an X-ray image, we aim to estimate the probability of this concept existing on this X-ray.\n \nTo achieve this goal, We train a binary classifier for each concept. We obtain the positive and negative examples for each concept by leveraging a pretraining dataset with (X-ray, clinical report) pairs. We can estimate the presence of a concept in each example by prompting an LLM to generate a response indicating whether the clinical report implies the concept. For example, if the clinical report mentioned related terms about “ground-glass opacity,” it is a positive example of this concept. Otherwise, it will be a negative example. With positive and negative examples for a concept, we can train a binary classifier to predict the concept given an X-ray image.\n\n**Q6. Is the training end-to-end?**\n\nNo. To be computationally feasible, the training has stages: (1) constructing the prior from documents (explained in Q3), (2) concept grounding (explained in Q5), and (3) linear layer for final label prediction. Given an X-ray image, the workflow is: (1) execute the concept grounding functions to get the probabilities on all concepts in the structure prior, (2) use the concept probabilities as the input to a linear layer to get the final label prediction, e.g., predict COVID or Normal. Please also refer to Lines 142-144 for Preliminary on Concept Bottleneck Model and Sec 4.3 for details about the workflow.\n\n**Q7. Is the parameter prior a part of the bottleneck predictor?**\n\nNo. As explained in Sec 4.4, the parameter prior is used to regularize the learning of the linear layer. We will compute the L1 distance between the linear layer and the parameter prior as the loss to align the learned parameters with priors.\n\n**Q8. How to determine the best number of concepts?**\n\nWe did not tune this hyperparameter much and used 150 concepts. In Figure 4, we examine the impact of the number of concepts on ID and OOD performance. More concepts lead to better ID but worse OOD so that an appropriate trade-off can be selected for a problem.\n\n**Q9. How did you determine the preferred correlation between the label y and concept c?**\n\nA language model (explained on Line 198) determines the preferred sign of the correlation between y and c.\n\nPlease do not hesitate to ask more questions, and we will be happy to answer them. We look forward to your response!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fo61tyJAB9", "reviewer": "Reviewer_tMbd", "summary": "This paper addresses the issue of deep networks' sensitivity to domain shifts in medical image analysis, particularly for chest X-rays and skin lesion images. The authors propose Knowledge-enhanced Bottlenecks (KnoBo), a concept bottleneck model that incorporates explicit medical knowledge from resources like textbooks and PubMed to improve generalization. KnoBo leverages retrieval-augmented language models to create a clinically relevant concept space and demonstrates a significant performance improvement—32.4% on average—over fine-tuned models across 20 datasets, highlighting PubMed as a particularly effective knowledge resource for mitigating domain shifts.", "questions": "Application to 3D Medical Imaging:\nQuestion: Have you considered extending the KnoBo framework to 3D medical imaging modalities such as CT scans or MRIs? If so, what challenges do you anticipate, and how might they be addressed?\nSuggestion: Expanding your research to include 3D medical images would significantly increase the applicability and impact of your work. Discussing potential adaptations and preliminary results for 3D imaging could provide valuable insights and demonstrate the versatility of KnoBo.\nComparative Analysis with State-of-the-Art Methods:\nQuestion: How does KnoBo compare with recent state-of-the-art domain adaptation and generalization methods, especially those tailored for 3D medical imaging?\nSuggestion: Including a detailed comparative analysis with advanced domain adaptation techniques, particularly those used in 3D imaging, would strengthen your paper. This would provide a clearer benchmark and highlight the unique advantages and limitations of KnoBo.", "limitations": "No", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Innovative Integration of Medical Knowledge:\nThe paper introduces Knowledge-enhanced Bottlenecks (KnoBo), a novel approach that integrates explicit medical knowledge into deep networks. By using retrieval-augmented language models to incorporate clinically relevant factors from medical textbooks and PubMed, the model gains a robust prior that significantly enhances its generalization capabilities.\nComprehensive Evaluation Across Diverse Domain Shifts:\nThe authors conduct a thorough evaluation of KnoBo across 20 datasets and two imaging modalities (chest X-rays and skin lesion images). This extensive testing demonstrates the model's ability to handle various domain shifts, such as differences in data from different hospitals and demographic confounders, showcasing its practical applicability and robustness.\nSignificant Performance Improvement:\nKnoBo shows substantial performance gains, outperforming fine-tuned models by an average of 32.4% on confounded datasets. This impressive improvement highlights the effectiveness of incorporating medical knowledge for reducing sensitivity to domain shifts, with evaluations indicating that PubMed is a particularly valuable resource for enhancing model performance and information diversity.", "weaknesses": "Lack of Application to 3D Medical Imaging:\nIssue: The paper primarily focuses on 2D medical imaging modalities, such as chest X-rays and skin lesion images, without addressing the broader and more widely used domain of 3D medical imaging.\nImpact: This limitation raises concerns about the generalizability and applicability of the proposed KnoBo model to 3D imaging scenarios, such as CT scans or MRI, which are critical in many clinical contexts. Including evaluations on 3D medical images would significantly enhance the relevance and impact of the research.\nInsufficient Comparative Analysis with 3D Domain Adaptation Methods:\nIssue: There is a lack of comparative analysis with existing domain adaptation and generalization methods specifically applied to 3D medical imaging.\nImpact: The absence of such comparisons makes it difficult to gauge the relative effectiveness of KnoBo in a comprehensive manner, particularly given the significant advancements in 3D domain adaptation techniques. Addressing this gap would provide a more complete evaluation of KnoBo's capabilities and limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work! Here, we address your concern about applications in 3D images.\n\nWe agree with you that 3D modalities are important. However, we scoped our study to 2D modalities as these are cheaper in practice. Part of the motivation is to transfer to less-resourced hospitals where X-rays are generally available but more expensive imaging modalities are not (i.e., in developing countries). That being said, we are very concerned with establishing the broad applicability of our work, and so we evaluated 20 datasets within the modalities we considered.  \n\nThe KnoBo framework is flexible enough to adapt to 3D modalities because the key contribution is the factorization. Many components of KnoBo can be readily reused, such as the concept generation from medical documents. We only need an appropriate 3D encoder (e.g., 3D CNN) and a pretraining dataset. Fortunately, large-scale public 3D medical datasets are available, e.g., DeepLesion [1] has 32,000 annotated CT scans, CT-RATE [2] has 50,188 CT volumes with associated clinical reports, and the recently released CLIP-CT [2] could serve as a backbone.  We agree that 3D would enrich the work, but we feel we have provided significant evidence already that KnoBo is effective and that there is a clear path on how one could apply it to 3D domains in the future.\n\n[1] Yan et al. DeepLesion: Automated Deep Mining, Categorization and Detection of Significant Radiology Image Findings using Large-Scale Clinical Lesion Annotations. 2017.\n\n[2] Hamamci et al. A foundation model utilizing chest CT volumes and radiology reports for supervised-level zero-shot detection of abnormalities. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We deeply appreciate the time and effort all reviewers have contributed. We feel very encouraged to see that all the reviewers have overall positive attitudes towards our work. Reviewers found our work interesting (f9Nf) and praised the **novelty of our method** (T8n7, tMbd) with **comprehensive evaluations** (z571, T8n7, tMbd) and **clear presentation** (z571, T8n7, tMbd).

We respond to each reviewer's comments individually and hope your concerns can be addressed. Again, we appreciate your time and expertise in the evaluation process.

Meta-review: All reviewers are in agreement and there is a strong consensus regarding the novelties presented in the paper, the thoroughness of the evaluations, and the clarity of the writing. The paper demonstrates significant contributions to the field and provides comprehensive experiments that validate the proposed approach, making it a well-received submission. However, some ethical concerns were raised during the review process, which the ethics reviewers believe are addressable by the authors. These concerns mainly revolve around specific aspects of data usage and consent, but with appropriate revisions and clarifications, the paper is expected to meet ethical standards. The authors are advised to address all raised issues in the final paper.
Decision: Accept (spotlight)