CheXT: Knowledge-Guided Cross-Attention Transformer for Abnormality Classification and Localization in Chest X-rays
Authors: Yan Han
Ying Ding
Ahmed Tewfik
Yifan Peng
Zhangyang Wang
Venue: ICLR 2022 Submitted (2022)

Abstract: Classical chest X-ray analysis has designed radiomic features to indicate the characteristics of abnormality of the chest X-rays. However, extracting reliable radiomic features heavily hinges on pathology localization, which is often absent in real-world image data. Although the past decade has witnessed the promising performance of convolutional neural networks (CNNs) in analyzing chest X-rays, most of them ignored domain knowledge such as radiomics. Recently, the surge of Transformers in computer vision has suggested a promising substitute for CNNs. It can encode highly expressive and generalizable representations and avoid costly manual annotations via a unique implementation of the self-attention mechanism. Moreover, Transformers naturally suit the feature extraction and fusion from different input modalities. Inspired by its recent success, this paper proposes \textbf{CheXT}, the first Transformer-based chest X-ray model. CheXT targets (semi-supervised) abnormality classification and localization from chest X-rays, enhanced by baked-in auxiliary knowledge guidance using radiomics. Specifically, CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers. During training, the image branch leverages its learned attention to estimate pathology localization, which is then utilized to extract radiomic features from images in the radiomics branch. Therefore, the two branches in CheXT are deeply fused and constitute an end-to-end optimization loop that can bootstrap accurate pathology localization from image data without any bounding box used for training. Extensive experiments on the NIH chest X-ray dataset demonstrate that CheXT significantly outperforms existing baselines in disease classification (by 1.1\% in average AUCs) and localization (by a \textbf{significant average margin of 3.6\%} over different IoU thresholds). Codes and models will be publicly released.

Under review as a conference paper at ICLR 2022
CHEXT: KNOWLEDGE-GUIDED CROSS-ATTENTION TRANSFORMER FOR ABNORMALITY CLASSIFICATION AND LOCALIZATION IN CHEST X-RAYS
Anonymous authors Paper under double-blind review
ABSTRACT
Classical chest X-ray analysis has designed radiomic features indicate the characteristics of abnormality of the chest X-rays. However, extracting reliable radiomic features heavily hinges on pathology localization, which is often absent in real-world image data. Although the past decade has witnessed the promising performance of convolutional neural networks (CNNs) in analyzing chest X-rays, most of them ignored domain knowledge such as radiomies. Recently, the surge of Transformers in computer vision has suggested a promising substitute for CNNs. It can encode highly expressive and generalizable representations and avoid costly manual annotations via unique implementation of the self-attention mechanism. Moreover, Transformers naturally suit the feature extraction and fusion from different input modalities. Inspired by its recent success, this paper proposes CheXT, the first Transformer-based chest X-ray model. CheXT targets (semi- supervised) abnormality classification and localization from chest X-rays, enhanced by baked-in auxiliary knowledge guidance using radiomics. Specifically, CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers. During training, the image branch leverages its learned attention to estimate pathology localization, which is then utilized to extract radiomic features from images in the radiomics branch. Therefore, the two branches CheXT are deeply fused and constitute end-to-end optimization loop that can bootstrap accurate pathology localization from image data without any bounding box used for training. Extensive experiments the NIH chest X-ray dataset demonstrate that CheXT significantly outperforms existing baselines in disease classification (by 1. % average AUCs) and localization (by significant average margin of 3.6% over different IoU thresholds). Codes and models will be publicly released.
INTRODUCTION
In medical study, handcrafted radiomics (wanenburgg et al. 2016) refers to the process of extracting several quantitative and semiquantitative features from medical images for improved decision support. It has the potential to uncover disease characteristics that are difficult to identify by viewing raw images alone. Given their advantages, researchers have explored the performance of radiomie features for chest X-ray analysis. For example, (Shi et al.) Saygih 2021) extracted set of radiomic features to diagnose different types of pncumonia. (Bai et al. 2020) proposed hybrid model to encode the combination of radiomic features and clinical information. (Ghosh et al. 2020) presented new handcrafted feature to distinguish between severe and nonsevere patients. However, all the above methods rely accurate pathology localization annotations to extract radiomic features from the correct "region of interest" (aka bounding boxes) but not other irrelevant parts(Van Van Griethuysen et al. 2017). Such bounding boxes usually expensive and time-consuming acquire by humans and, if inaccurate, will tremendously degrade the reliability of radiomic features. There is thus an unmet need automatically localize pathologies on chest X-rays facilitate radiomic features extraction.
Under the rapid development of deep learning, many researchers have made their efforts to utilize the Convolutional Neural Networks (CNNs) in building automated systems of chest X-ray abnormality classification and localization (Rajpurkar et al., 2017] Wang et al.) 2017] Li et al.) 2018 Liu et al.) 2019b; Rozenberg al. 2020 Wang al. 2021). However, CNN methods witness several limitations.

---

Under review as a conference paper at ICLR 2022
First, chest X-rays valuable domain knowl- edge and comain-specifii features, such as ra- diomic features. Thus, they could have blessed better recognition but are unfortunately over- looked by most CNNs.. Second, chest X-rays have more subtle discriminative features com- pared to natural images, making their recogni- tion more challenging. Finally, CNNs are often criticized for being non transparntt and their predictions not traceable by humans, hence hin- ders their acceptance and adoption by clinicians.
Cantiomegaly
Diacase distriiutionn
CheXT
Localization
Redtfoomic
BY OA
Why Transformers for Chest X-rays? The latest surge of transformers provides a promis- ing alternative to model chest X-rays. Trans- former was first prevailing to Natural Language Processing (Vaswani et al. 2017; Devlin al.) 2018 Brown et al. 2020). lollowed by its recent success in computer vision (Dosovitskiy al.,
Figure 1: General overview of CheXT frame- work for Cardiopulmonary Disease Classification and Localization from chest X-rays. CheXT takes the chest X-ray image as the input and outputs heatmap for pathology localization, based on which the bounding box could be obtained. Ra- diomic features are further extracted from the bounded region and are fed to predict the disease.
2020 Carion al.) 2020; Zhu al. (2020) and multi-modal learning (Ying et al. 2021). is an "universal modeling tool" that unify the fea- ture extraction and fusion from different input modalities within one model, without domain- specific model tweaks. For example, (Akbari al. 2021) demonstrated to learn powerful multi-moda. representations from unlabeled video, audio, and text data, using one Transformer architecture.
Bringing that into the context of chest X-rays, we see the tantalizing potential that a Transformer could organically and jointly learn from two views of chest X-rays: (i) raw X-ray images that contain the richest details, hence benefiting from the data-driven learning capacity; and (ii) radiomic features that encode critical domain prior knowledge, hence effectively guiding and regularizing the learning problem: as
process. The appeal of Transformers is, however, blocked by a thiicke--andeege aforementioned, the extraction of reliable radiomic features hinge on the pathology localization, but then the pathology localization is often absent in images and also needs to be learned first.
This paper presents holistic framework of Showledge-Guider Cross-Attention Transformer for Chest X-ray analysis, named CheXT (Figure D CheXT consists of two Transformer- based branches that learn from two data formats characterizing the same patient: the image and radiomics branches. Both are deeply fused and interacted by cross-attention layers (Chen et al.) 2021a).. Notably, the radiomic features need be extracted from the learned pathology localizations, which not readily available. The key enabling technology to resolve this hurdle, is to construct "feedback loop" during training: the image branch leverages its learned attention estimating pathology localization, which is then utilized to extract radiomic features from images in the radiomics branch. Training under unified contrastive loss, such an end-to-end optimization loop can bootstrap accurate pathology localization from image data, with no bounding boxes used for training. Our contributions are outlined as follows:
We leverage the radiomic feature as an "auxiliary input modality" correlated with the raw image modality and encoded with domain knowledge. We then propose a novel knowledge- guided cross-attention Transformer, CheXT, to jointly extract and fuse image and radiomic feature representations for chest X-ray analysis.
To resolve the key "chicken-and-egg" problem of extracting radiomic features without available pathology localization, we construct an innovative optimization loop with the image and radiomic branches deeply interacting via attention. Such end-to-end loop can bootstrap accurate pathology localization from images without using human- -nnotttdd bounding boxes.
Our approach achieve superior classification and localization results against several com- petitive baselines, the NIH chest X-ray benchmark. Notably, compared existing
2

---

Under review as a conference paper at ICLR 2022
approaches, CheXT generates more accurate disease localization for extracting radiomic features, by a significant average margin of 3.6% over different IoU thresholds.
RELATED WORK
Radiomics in Medical Diagnosis. The design of radiomics involves biological and medical data and prior knowledge. Thus, radiomics vastly enriches images and expands the horizons of the image toward in-vivo biologic information extraction (Gillies et al., 2016). In image-based biomarkers for cancer staging and prognostication, radiomics had shown promising power (Nasief et al. (211)). Radiomics extracts quantitative data from medical images to represent tumor phenotypes, such as spatial heterogeneity of tumor and spatial response variations. (Eilaghi et al. (017)) demonstrated that radiomic of CT texture features are associated with the overall survival rate of pancreatic cancer.
(Chen et al., (017)) revealed that the first-order radiomic features (e.g., mean, skewness, and kurtosis) are correlated with pathological responses to cancer treatment. (Huang et al. (0118) showed that radiomics could increase the positive predictive value and reduce the false-positive rate in lung
cancer screening for small nodules compared with radiologists. (Zhang et al.] (018) found that
multiparametric MRI-based radiomics nomograms provided improved prognostic ability advanced nasopharyngeal carcinoma.
In comparison, deep learning is often criticized for being "black box" and lacks interpretability despite high predictive accuracy. This limitation has motivated many interpretable learning techniques including activation maximization (Erhan et al., 2009), network inversion (Mahendran & Vedaldi 2015), GradCAM (Selvaraju et al. (2016), and network dissection (Bau et al. (017)). We believe that the joint utilization of radiomies and interpretable learning techniques our framework can further advance accurate yet interpretable learning in the medical image domain.
Transformers for Medical Images. Recently, Vision Transformer (ViT) (Dosovitskiy et al. 2020 achieved state-of-the-art classification on ImageNet by directly applying Transformers with global self-atteniion to full-sized images. Inspired by the promising performance
VIT, researchers have recently applied the idea to medical images. For example. (Oktay et al. [018 Wang al.) 2019 Chen cet al. (021b) used the attention mechanism to boost the performance of medical image segmentation. (Valanarasu et al. 2021) proposed gated axial-attention model to introduce additional control mechanism in the self-attention module. (Park et al.) 2021) utilized hybrid framework of CNN and Transformer for Covid- prediction. However, those methods did not consider any domain prior knowledge. (Han et al. 202T) applied pre-extracted radiomic features to guide pncumonia detection from chest X-ray images. However, they adopted convolutional backbone for image encoder, while using a another specifically crafted radiomic features encoder. Therefore, the method involves no joint interaction between image and radiomic features, and need to use accurate bounding boxes during training in order to extract radiomic features. Hence, their method dramatically limits usability clinical practice.
METHOD
An overview of CheXT is illustrated in Figure 2 In the following subsections, will first present Cross-attention Vision Transformer (CrossViT), recent two-branch ViT backbone on which CheXT built, and then describe our many unique improvements customized for Chest X-ray analysis.
3.1 PRELIMINARY: VIT AND ATTENTION
ViT first converts an image into a sequence of patch tokens by dividing the image with certain patch size and linearly projecting each patch into tokens. special token (CLS) is added in front of the sequence, in the original BERT (Devlin et al., 2018).. Then, all tokens are passed through stacked Transformer encoders. Finally, the hidden state corresponding to the CLS token is used as the aggregate sequence representation for image classification.
Transformer encoder is composed of a sequence of blocks where each block contains multiheaded self-attention with feed-ford forward network. Layer normalization and residual shorteuts are applied before and after every block, respectively. The granularity of the patch size affects the accuracy and complexity of VIT. Therefore, ViT can perform better with fine- grained patch size but with higher
3

---

Under review as a conference paper at ICLR 2022
Image Branch
Cane Block
Tranfomer Elceeer
Trandimer Enoader
Epezzh
fi
Bnsoder
late
feedlback
Cross-
Unsuperviael Contrajave
Supervised Shcal
loop
A trantion
Ar
Radiomic Brunch
Reis
BY OA
Baoocer
Tranforrer
Spacck
Figure 2: Overview of proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure attention of the CLS tokens. Finally, the output of two CLS tokens (Lela and Rats) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage contrastive learning strategy. Specifically, we generate the image view 2; = ge(Teis) ) by projection head 9t- Similarily, we generate the radiomic view 2, = 9. (Rtts) by another projection head 9,- We maximize the agreement between and via a contrastive loss (NT-Xent)). Of note, the contrastive loss only active during the training.
respectively. The output tokens then fused by efficient module via cross
FLOPS and memory consumption (Chen al., 202 To relieve this problem, Cross (Chen al. 202 proposed with dual-branch ViT where either branch operates different patch size, as its own "view" of the image. The pross-attention module is then used to fuse information between the branches to balance the patch sizes and complexity. Similar to VIT, the final hidden vector of CLS from two branches are used for image classification.
3.2 OUR PROPOSED CHEXT MODEL.
Cross VIT supplies a graceful framework to simultaneously tackle and fuse two different "views" from the same imput data, c.g., different-size image patches in (Chen et al.) 2021a). In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image another "view" (Figure 2 The two views are then jointly learned by interacting through cross-attention. Transformer serves as the modality-agnostic backbone for both.
Specifically, we introduce dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while second (complementary) branch handles radiomic features. To dilemma in extracting reliable radiomic features without bounding
resolve the rchicken-and-egee boxes, we have designed a novel Bootstrap Your Own Attention (BYOA) module, using feedbacks to learn region localization for extracting radiomic features. A simple yet effective module is also utilized to fuse information between the branches. In the subsequent sections, we will describe the two branches, the BOYA module, and the fusion module.
The primary image branch uses a Progress-Sampling ViT (PS-ViT) Yue et al. 2021 its backbone. Unlike the vanilla ViT that splits images into fixed-size tokens, PS- ViT utilizes iterative and progressive sampling strategy to locate discriminative regions and avoid over-partinn ver-partiiion object structures. We experimentally observed PS-ViT outperforms ViT and other variants in our framework because it generates higher-quality and more structure-aware attention maps, which are crucial for estimating the pathology localization during training.
Image Branch.
Radiomics Branch. The complementary radiomics branch is for processing radiomic features. Handerafted features usually cover a wide range of categories, such as first-order (basic intensity and shaped-based features), second-order (texture features extracted from various matrices), and more advanced features including those calculated from Fourier and wavelet transforms. Specifically, the radiomic features composed of the following categories:
First-Order statistics features to measure the distribution of voxel intensities within the bound- ing boxes. The features include energy (the measurement of the magnitude of voxel values),
4

---

Under review as a conference paper at ICLR 2022
Cross- Attention
Attention map generator
Rndiomics
Radiomic
Extractor
Features
Figure 3: Overview of our BYOA module. For the input chest X-rays, we look at the self-attention of the CLS token of the Image branch the heads of the final output of the cross attention module. Then we apply a threshold (0.1) (which means we only keep the top 10% active pixels in the whole attention map) to generate the bounding boxes. Then with the generated bounding boxes, the Pyradiomic tool as the radiomic extractor to extract the radiomic features.
entropy (the measurement of uncertainty in the image values), and max/mean//median gray level intensity within the region of interest.
Shape- based features, such as Mesh Surface, Pixel Surface, and Perimeter.
Gray-level features, such as Gray Level Co-occurrence Matrix (GLCM), Gray Level Size Zone (GLSZM), Gray Level Run Length Matrix (GLRLM), Neighboring Gray Tone Differ- ence Matrix (NGTDM), and Gray Level Dependence Matrix (GLDM) features.
In short, radiomic features set of quantitative features that describe the characteristics of medical images. our framework, we aim to make the hidden features of the CLS similar to the radiomic features learn the localization of pathologies in the chest X-rays. For this branch, use the vanilla Transformer (L.iu et al., 2017) as the radiomic features encoder. Please note that the only difference is that the positional encoding module discarded, since there does not exist any positional relationship between the radiomic features.
Bootstrap Your Own Attention (BYOA): A Feedback Loop Module. Our main roadblock is how to generate robust radiomic features without pathology localization. On one hand, radiomic features are dependent on and highly sensitive to the local image regions of interest, for which we have no bounding box annotation. On the other hand, image features would benefit from the guidance from radiomic features that encode important domain knowledge. The learning of image and radiomic features are fully entangled, forming a challenging "chicken-and-egg" loop.
To address this issue, we design BYOA to constitute an end-to-end feedback loop that can bootstrap accurate pathology localization from image data, without any bounding box used for training (Fig- ure B BOA contains two components: attention map generation and radiomic feature extraction.
Attention Map Generation. Similar to the approach in (Caron et al. 202Tp, we look the self-attention of the CLS token on the heads of the last layer. Here, we have two CLS tokens from two branches, but the attention maps only come from the Image branch. Then we apply threshold on the self-attention maps to generate bounding boxes for the extraction of radiomic features. The choice of the threshold will influence the quality of the radiomic features. Specifically, the threshold is designed how much active pixels to keep in the generated attention map. The more active pixels we keep, the larger the generated bounding box is. Please see 4.3.3 for more details.
Radiomic Features Extraction. Given the original images and generated bounding boxes, we used the Pyradiomic tool to extract radiomic features (Van Griethuysen et al. 2017).
Cross-Attention Fusion Module. This fusion involves the CLS token of one branch and patch tokens of the other branch. As the CLS token is the aggregate representation of the branch, this interaction helps include information at different scale. Please refer to (Chen et 2021a) for more details about the cross-atee attsstatt mechanism.
3.3 SEMI-SUPERVISED LOSS FUNCTION
As shown in Figure CheXT trained using the linear combination of the supervised focal classification and unsupervised contrastive losses. For the supervised classification, considering that the chest X-ray dataset is usually highly imbalanced, we adopt the focal loss (Pasupa et al. 2020). For unsupervised contrastive learning, use the cross-view contrastive loss (Chen et al. 2020).
5

---

Under review as a conference paper at ICLR 2022
Supervised Classification Focal Loss. We feed the output of the CLS tokens of two branches Iets and Rele to a simple linear classifier. The supervised classification focal loss Ly is defined as
f -a (1 y²)²² logy' y (1- any log (1-y)). y= 0
==
(1)
allows us give different importance positive and negative examples. y is used to distinguish easy and hard samples and force the model to learn more from difficult examples.
Unsupervised Cross-View Contrastive Loss. Our contrastive loss extends the normalized tempera- ture scaled cross-entropy loss (NT-Xent). The difference is that we maximize agreement between two feature views extracted from different input formats, one the image and the other radiomies.
Given an anchor chest X-ray in a minibatch, the positive sample will be its radiomic feature view, and the negative samples will be other chest X-rays (either image or radiomics). Since the CLS could be regarded as the representation of all other tokens, we only need to maximize the agreement between them. Suppose Jets,k and Rets,k are the k-th image features and radiomic features the minibatch, respectively, and sim() the cosine similarity, the loss function Let is defined as
Ld = log exp(sim(gii
9r (Ress,)))/))
(2)
Σ exp(sim(g, (Ias,K) 9r (Rei,x))/T))
where 7 is the temperature. The final contrastive loss is summed over all instances in the minibatch.
Overall, we treat CheXT training as a semi-supervised multi-task learning. Since for chest x-rays, there exists two labels, disease class labels and pathology bounding box annotations. In our case, we only have access to the disease labels, however, bounding box annotations are more important than the disease class labels for chest x-rays. Here, when say "semi-supervvsed" bounding box annotations, not disease class labels. For multi-task, one task is supervised disease classification. The other is unsupervised cross- contrastive learning. The total loss is:
refer to the
L = (1 (1- A) x La + A x ft
4 EXPERIMENTS
4.1 DATASET AND PROTOCOL SETTING
The NIH Chest X-ray dataset (Wang et al. 2017) consists of 112,120 chest X-rays collected from 30,805 patients, and each image labeled with cardiopulmonary disease labels (Atelectasis, Cardiomegaly, Effusion, Infiltration, Mass, Nodule, Pneumonia, and Pncumothorax). The labels are extracted from the associated radiology report using an automatic labeler (Peng et al. 2018) with reported accuracy of 90%. We use the extracted labels ground-truth for training CheXT. The NIH dataset also includes high-quality bounding box annotations for 880 images by radiologists. We separated these 880 images from entire dataset, and they used only to evaluate disease localization. A significant difference between our method and existing baseline methods (Liu et al.) 2019a Li et al.) 2018)) is that our method does not require any training data related the bounding box while others use some percentage of these images for training.
In our experiments. we followed the same protocol as in the studies of (Wagg et al. 2017 Li et al. 2018), to shuffle our dataset (excluding images with Bounding Boxes annotations) into three subsets: 70% for training, 10% for validation, and 20% for testing. In order to prevent data leakage across patients, we make sure that there is no patient overlap between our train, validation, and test set.
4.2 IMPLEMENTATION DETAILS
We build our image branch encoder based on PS-VIT Yuue et al., 2021) and apply their default hyperparameters for training. The only difference is that we use more shallow PS- ViT with 6 layers. For the radiomic branch encoder, since the radiomic features are already informative features, we use more shallow Transformer (I-layers) to encode the radiomic features. And we add one more cross-attention layer to fuse these two format features. We set the batch size as 128 and train the model for 50 epochs (5 warm-up epochs). Other setup includes a cosine linear-rate scheduler with linear warm-up. up. an initial learning rate of 0.004, and a weight decay of 0.05. During the evaluation, resize the image to 256x256 and take the center crop 224x224 as the input.

---

Under review as a conference paper at ICLR 2022
Method
Atelectasis
Cardiomegaly
Effusion
Infiltration
Mass
Nodule
Pneumonia
Pneumothorax
Mean
CNN
Wang et
201
0.81
0.78
0.61
0.71
0.67
0.63
0.81
0.718
Wang af al
2018
0.73
0..44
0.79
0.67
0.73
0.69
0.72
0.85
0.753
100 2017
0.77
0.90
0.86
0.70
0.79
0.84
0.786
Kappurkar
2017
0.82
0.91
0.88
0.72
0.86
0.78
0.76
0.89
0.828
Kumar et m. 2018
0.76
0.91
0.86
0.69
0.75
0.67
0.72
0.85
0.778
Lau
Seyyed et al. (020)
0.81
0.92
0.87
0.72
0.83
0.78
0.76
0.88
0.821
Han et 2020
0.83
0.92
0.87
0.76
0.85
0.76
0.77
0.86
0.828
Transformer
VIT
0.74
0.78
0.81
0.72
0.70
0.66
0.65
0.76
0.728
CrossViT
0.69
0.71
0.72
0.72
0.74
0.79
0.82
0.88
0.759
PS VIT
0.73
0.766
CheXT
0.80
0.92
0.78
0.86
0.88
0.88
0.79
0.81
0.839
(1:0.02)
(::0.00)
(4:0.01)
(4:0.01)
(2:0.0)
(::0.00)
(4:0.01)
(4:0.02)
Table I: Comparison with baseline models for AUC of each class average AUC. For each column, bold values denote the best results.
4.3 ABNORMALITY CLASSIFICATION
Abnormality classification is multi-label classification problem. It assigns one more labels among 8 cardiopulmonary diseases (Atelectasis, Cardiomegaly, Effusion, Infiltration, Mass, Nodule, Pneumonia, and Pneumothorax) to each input image at inference time. We conducted 3-fold cross- validation (Table D. We compared CheXT with reference models, which have published statc-of-the- art performance of disease classification the NIH Chest X-ray dataset.
4.3.1 EVALUATION METRIC
We used Area under the Receiver Operating Characteristics (AUC) to estimate the performance of our model (Fawcett Tom 2006). A higher AUC score implies a better classification model. We also provide mean AUC across all the classes to highlight the overall performance of our model.
4.3.2 COMPARISON WITH THE SOTA MODELS
AUC scores for each disease and mean AUC score across diseases are presented in Table I Not only we compared CheXT with previous CNN-based SOTA models, but also several Transformer- based models. From the table, diseases. We report the average AUC of runs to show the robustness of our model. Compared to all baselines, CheXT achieves
find that CheXT outperformed the baselines in the majority of
AUC score of 0.839 using a dual-branch shallow Transformer across the different diseases, which is 0.011 higher than the SOTA (uses DenseNer-121) (Rajpurkar al. 2017) on disease classification. Specifically, our results significantly outperformed the best baseline models by 0.13, 0.09, and 0.02 in AUC for detecting Infiltration, Nodule, and Mass higher respectively. Besides, compared to the Transformer-base the extracted radiomic features for disease prediction, which improves the accuracy and adds the model's s interpretability due to the utilization of self-designed handcrafted radiomic features.
models, the key difference is that we utilize
4.3.3 EFFECT OF DIFFERENT THRESHOLDS OF ATTENTION MAPS
We investigate the impact the thresholds (T') the process of attention map generation, on the performance of CheXT for disease classification. Figure 4A summarizes the AUC comparison of CheXT for different values of T. Higher values of T imply smaller bounding boxes to extract radiomic features. During our experiments, found that CheXT performs worse when very large bounding boxes are generated. This observation meets our expectations since radiomic features are very subtle bounding boxes. Specifically, the extracted radiomie features more robust if more focal and accurate bounding boxes are given.
4.3.4 EFFECT OF CONTRASTIVE LEARNING
We also investigated the impact of contrastive learning. Specifically, we check the performance of CheXT for disease classification by varying À of equation (3). Figure summarizes the AUC
7

---

Under review as a conference paper at ICLR 2022
(A)
0.86
(B) 0.86
0.84
0.84-
0.82
AUC
AUC
0.80
0.80
0.78
0.78-
0.76
0.76-
0.74
0.74
0.1
0.3
0.5
0.7
0.9
0.5
0.6
0.7
0.8
0.9
1.0
Figure 4: AUC comparison for varying T in (A) attention map generation and (B) in Equation 3.
comparison of CheXT for different values of X. Higher values of À implies lower weight to contrastive loss. During our experiments, found that CheXT performs worse when small weight (1%) is given to the contrastive loss. CheXT's performance improves when we increase contrastive loss weight, but after a certain point, it starts decreasing. This proves our hypothesis that both contrastive and focal losses are important. In a calculated ratio, they help CheXT to learn both disease-level and patient-level discriminative visual features.
4.4 ABNORMALITY LOCALIZATION
The NIH Chest X-ray dataset has 880 images labeled by radiologists with the bounding box informa- tion. We have used this dataset to evaluate the performance CheXT for abnormality localization. Many prior works (Li et al. 2018 Liu et al. 2019a) have used fraction of ground truth (GT) bound- ing boxes for
do training and evaluated their system on the remaining. To ensure a robust evaluation, any GT for training. Table 2 presents evaluation results all 880 images. We used (Wang et al. (017)) as our baseline to compare our localization results since it has the same experimental settings.
4.4.1 EVALUATION METRIC
For localization, evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU). Our localization results are only calculated for 880 images which have ground truth annotation for diseases. The localization is defined as correct only if loU > T(IUU). We evaluated CheXT for different thresholds ranging from 10.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7) as shown Table 2 A higher IoU threshold preferred for disease localization because clinical usage requires high accuracy.
4.4.2 COMPARISON WITH THE SOTA MODELS
We compared disease localization accuracy under varying IoU, with baselines having similar settings CheXT (Table 2p. Unlike other baselines(Li al. 2018 Liu al.) 2019a) that portion of 880 images for evaluation (because they need the remaining data for training), we used all 880 annotated images for evaluation. Therefore, no k-fold tross-validation for localization was performed. CheXT average performance across 8 diseases is significantly better than the baseline under all loU thresholds. When the thresholds of IoU set 0.1, CheXT outperforms the baseline (Wang et al.) 2017) in Cardiomegaly, Infiltration, Mass, and Pneumonia. Even with higher thresholds, our model performs superior to baseline. For example, when evaluated T(IUU) 0.5, our "Cardiomegaly" accuracy is still 32%, while the reference model achieves only 18%. Our "Pneumonia" accuracy 12%, while the reference model has only 3% accuracy. Note that some diseases can appear at multiple places, but ground truth might have mentioned only one location. This can significantly impact the accuracy at high thresholds. More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and B BYOA module can help the model learn more important regions, which also improves the model's interpretability. Few examples the localization results of ViT and CheXT shown in Figure
8

---

Under review as a conference paper at ICLR 2022
Atlectasis
Cardiomegaly
Nodule
Pneumonia
Pneumothorax
CheXT
Figure Examples of visualization localization on the test images. The attention maps are generated from the self-attention maps of the CLS token. The ground-truth bounding boxes shown in blue. The left image in each pair is the localization result of VIT (Dosovitskiy et al. 2020). The right one is our localization results. All examples positive for corresponding disease labels. Best viewed in color.
T(loU)
Model
Atelectasis
Cardiomegaly
Effusion
Infiltration
Nodule
Pneumonnia
Pneumothurax
Mean
0.1
Wang al.
2017
0.69
0.94
0.66
0.71
0.40
0.14
0.63
0.38
0.569
V7I
0.58
0.91
0.61
0.77
0.44
0.11
0.75
0.25
0.553
CheXT
0.61
0.95
0.65
0.82
0.50
0.13
0.79
0.28
0.591
0.2
(Wing al. 2017
0.47
0.68
0.45
0.48
0.26
0.05
0.35
0.23
0.371
0.01
CheXT
0.41
0.91
0.41
0.59
0.26
0.05
0.57
0.19
0,424
0.3
Wann al. 2017
0.24
0.46
0.30
0.28
0.15
0.04
0.17
0.13
0.221
VIT
0.20
0.45
0.32
0.00
0.21
0.02
0.181
CheXT
0.28
0.79
0,22
0.38
0.12
0.01
0.41
0.05
0.283
0.4
Wang et al. 2017
0.09
0.28
0.12
0.01
0.07
0.115
VII
0.10
0.21
0.03
0.05
0.02
0.00
0.04
0.00
0.056
CheXT
0.17
0.54
0.13
0.18
0.07
0.01
0.26
0.02
0.173
0.5
Wang
2017
0.01
0.061
NTT
0.05
0.15
0.01
0.04
0.02
0.00
0.03
0.00
0.034
0.00
0.01
0.6
Wang al.
2017
0.02
0.08
0.02
0.01
0.02
0.03
0.029
VTI
0.01
0.03
0.01
0.01
0.01
0.00
0.01
0.00
0.010
CheXT
0.02
0.15
0.03
0.04
0.00
0.06
0.00
0.041
0.7
(Wang al. 2017
0.01
0.03
0.02
0.00
0.00
0.00
0.01
0.02
0.011
VIT
0.01
0.00
CbeXT
0.01
0.04
0.01
0.02
0.01
0.00
0.03
0.00
0,015
Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn't 't use any ground truth bounding box information, to fairly evaluate performance of our model, we only consider the previous methods' results under the same setting,
Table 2:
CONCLUSION
In this paper, we propose an end-to-end knowledge-guided cross-attention Transformer, named CheXT that can jointly model abnormality classification and localization without the supervision of localization annotation of chest X-rays. Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features prior knowledge, and feedback loop for image and radiomic features to mutually interact with each other. Additionally, the project aims to mitigate current gaps in radiology by making prior knowledge more accessible to image data analytic and diagnostic assisting tools, with the hope that this will increase the model's interpretability. Experimental results demonstrate that our method outperforms the -ote-----hhhttt art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text, 2021.
Xiang Bai, Cong Fang, Yu Zhou, Song Bai, Zaiyi Liu, Liming Xia, Qianlan Chen, Yongchao Xu, Tian Xia, Shi Gong, et al. Predicting covid-19 19 malignant progression with ai techniques. 2020.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretabilityy of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541-6549, 2017.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-suue vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv: 103. 14899, 2021a.
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. 2021b.
arXiv preprint
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xiaojian Chen, Kiyoko Oshima, Diane Schott, Hui Wu, William Hall, Yingqiu Song, Yalan Tao, Dingjie Li, Cheng Zheng, Paul Knechtges, et al. Assessment of treatment response during chemoradiation therapy for pancreatic cancer based quantitative radiomic analysis of daily cts: An exploratory study. PLoS One, 12(6)=e017896,, 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint vrXiv:/810.04860 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929. 2020.
Armin Eilaghi, Sameer Baig, Yucheng Zhang, Junjie Zhang, Paul Karanicolas, Steven Gallinger, Farzad Khalvati, and Masoom A Haider. Ct texture features are associated with overall survival in pancreatic ductal adenocarcinoma-a quantitative analysis. BMC medical imaging, 17(1):1-7, 2017.
Dumitru Erhan. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
FawcettTom. An introduction to roc analysis. Pattern Recognition Letters, 2006.
Biswajoy Ghosh, Nikhil Kumar, Nitisha Singh, Anup K Sadhu, Nirmalya Ghosh, Pabitra Mitra, and Jyotirmoy Chatterjee. A quantitative lung computed tomography image feature for multi-center severity assessment of covid-19. medRxiv, 2020.
10

---

Under review as a conference paper at ICLR 2022
Robert J Gillies, Paul E Kinahan, and Hedvig Hricak. Radiomics: images more than pictures, they are data. Radiology, 278(2):563-577 2016.
Yan Han, Chongyan Chen, Liyan Tang, Mingquan Lin, Ajay Jaiswal, Song Wang, Ahmed Tewfik, George Shih, Ying Ding, and Yifan Peng. Using radiomics as prior knowledge for thorax disease classification and localization in chest x-rays. arXiv preprint arXiv:2011.12555 2020.
Yan Han, Chongyan Chen, Ahmed H Tewfik, Ying Ding, and Yifan Peng. Pneumonia detection on chest x-ray using radiomic features and contrastive learning. arXiv preprint arXiv:2101.00 2021.
Peng Huang, Seyoun Park, Rongkai Yan, Junghoon Lee, Linda C Chu, Cheng T Lin, Amira Hussien, Joshua Rathmell, Brett Thomas, Chen Chen, et al. Added value of computer-aided ct image features for early lung cancer diagnosis with small pulmonary nodules: a matched case-control study. Radiology, 286(1):286-295 295, 2018.
Pulkit Kumar, Monika Grewal, and Muktabh Mayank Srivastava. Boosted cascaded convnets for multilabel classification of thoracic diseases in chest radiographs. In International Conference Image Analysis and Recognition, pp. 546-552. Springer, 2018.
Zhe Li, Chong Wang, Mei Han, Yuan Xue, Wei Wei, Li-Jia Li, and Li Fei-Fei. Thoracic disease identification and localization with limited supervision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8290-8299, 2018.
Chenxi Liu, Junhua Mao, Fei Sha, and Alan Yuille. Attention correctness neural image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Jingyu Liu, Gangming Zhao, Yu Fei, Ming Zhang, Yizhou Wang, and Yizhou Yu. Align, attend and locate: Chest x-ray diagnosis via contrast induced attention network with limited supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019a.
Jingyu Liu, Gangming Zhao, Yu Fei, Ming Zhang, Yizhou Wang, and Yizhou Yu. Align, attend and locate: Chest x-ray diagnosis via contrast induced attention network with limited supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10632- 2019b.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188-5196. 2015.
Haidy Nasief, Cheng Zheng, Diane Schott, William Hall, Susan Tsai, Beth Brickson, and X Allen Li. A machine learning based delta-radiomies process for early prediction of treatment response of pancreatic cancer. NPJ precision oncology, 3(1):1-10, 2019.
Ozan Oktay, Jo Schlemper, Loie Le Folgoe, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint 1rXiv:1804.0332 2018
Sangjoon Park, Gwanghyun Kim, Yujin Oh, Joon Beom Seo, Sang Min Lee, Jin Hwan Kim, Sungjun Moon, Jae-Kwang Lim, and Jong Chul Ye. Vision transformer using low-level chest x-ray feature corpus for covid-19 diagnosis and severity quantification. arXiv preprint arXiv:2104. 07235, 2021.
Kitsuchart Pasupa, Supawit Vatathanavaro, and Suchat Tungjitnob. Convolutional neural networks based focal loss for class imbalance problem: A case study of canine red blood cells morphology classification. Journal of Ambient Intelligence and Humanized Computing, pp. 1-17, 2020.
Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, and Zhiyong Lu. NegBio: perforerformecc tool for negation and uncertainty detection in radiology reports. In AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science, volume 2017, pp. 188-196,77 2018
11

---

Under review as a conference paper at ICLR 2022
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017.
Eyal Rozenberg, Daniel Freedman, and Alex Bronstein. Localization with limited annotation for chest x-rays. Machine Learning for Health Workshop, pp. 52-65. PMLR, 2020.
Ahmet Saygih.. A new approach for computer-aided detection of coronavirus (covid-19) from ct and x-ray images using machine learning methods. Applied Soft Computing. 105:107323, 2021.
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint
2016.
Shi, L Xia, F Shan, et al. Large-scale screening of covid-19 from community acquired pneumonia using infection size-aware classification. arxiv e-prints [preprint] 2020.
Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihalitoglu, and Vishal M Patel. Medical transformer:
Gated axial-attention for medical image segmentation. arXiv preprint arXiv:2 106., 2021.
Joost JM Van Griethuysen, Andriy Fedorov, Chintan Parmar, Ahmed Hosny, Nicole Aucoin, Vivek Narayan, Regina GH Beets-Tan, ChanSChrittophe Fillion-Robin, Steve Pieper, and Hugo JWL. Aerts. Computational radiomics system to decode the radiographic phenotype. Cancer research, x7041)xe04-- 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Xiaosong Wang. Yifan Peng. Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scae scale chest x-ray database and benchmarks on reakly-supervised classi- fication and localization of common thorax diseases. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2097-2106, 2017.
Xiaosong Wang, Yifan Peng. Le Lu, Zhiyong Lu, and Ronald M Summers. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9049-9058, 2018.
Xudong Wang, Shizhong Han, Yunqiang Chen, Dashan Gao, and Nuno Vasconcelos. Volumetric attention for 3d medical image segmentation and detection. In International Conference on Medical Image Computing and Computer-Assise Intervention, pp. 175-184. Springer, 2019.
Yirui Wang, Kang Zheng, Chi-Tung Cheng, Xiao Xiao-Yun Zhou, Zhilin Zheng, Jing Xiao, Le Lu, Chien- Hung Liao, and Shun Miao. Knowledge distillation with adaptive asymmetric label sharpening for semi-supervised fracture detection in chest x-rays. In International Conference on Information Springer, 2021.
Processing in Medical Imaging, pp.
Li Yao, Eric Poblenz, Dmitry Dagunts, Ben Covington, Devon Bernard, and Kevin Lyman. Learning to diagnose from scratch by exploiting dependencies among labels. arXiv preprint ar arXiv:1710. 10501, 2017.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform bad for graph representation? arXiv preprint arXiv:2106.05255 2021.
Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Philip Torr, Wayne Zhang, and Dahua Lin. Vision transformer with progressive sampling. arXiv preprint arXiv:2108.01684 2021.
Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christopher D Manning, and Curtis Langlotz. Learning to summarize radiology findings. arXiv preprint 18X9:v1809.44000
2018.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
Alex Zwanenburg, Stefan Leger. Martin Vallières, and Steffen Löck. Image biomarker standardisation initiative. arXiv preprint arXiv: 16/2.07023 2016.
12


Reviews:
[{"review_id": "nPZmW-e7BxO", "reviewer": "Reviewer_LbVM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) Radiomics is a term that most people in the ICLR community are not familiar with. What is it exactily? Is it a set of radiological/image features or bio-markers in medical images? Or something else? The authors should consider explaining it in plain language or ML language.\n\n2) Prior multi-modal feature fusion work in the chest x-ray domain should be discussed, examples below.\n\n    - Moradi, Mehdi, Ali Madani, Yaniv Gur, Yufan Guo, and Tanveer Syeda-Mahmood. \"Bimodal network architectures for automatic generation of image annotation from text.\" arXiv preprint arXiv:1809.01610 (2018).\n\n    - Wang, Xiaosong, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. \"TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays.\" arXiv e-prints (2018): arXiv-1801.\n\n    - Xue, Yuan, and Xiaolei Huang. \"Improved disease classification in chest x-rays with transferred features from report generation.\" In International Conference on Information Processing in Medical Imaging, pp. 125-138. Springer, Cham, 2019.\n\n    - Liao, Ruizhi, Daniel Moyer, Miriam Cha, Keegan Quigley, Seth Berkowitz, Steven Horng, Polina Golland, and William M. Wells. \"Multimodal Representation Learning via Maximization of Local Mutual Information.\" arXiv preprint arXiv:2103.04537 (2021).\n\n3) I don't see \"prior knowledge\" and how it's utilized in this work. Isn't the so called prior knowledge in this paper essentially extracted from the data (chest x-ray images)?\n\n4) What are the patch sizes? How are they determined? \n\n5) There are 14 pathology labels in the NIH chest x-ray dataset. Why/how did the authors select the 8 classes out of the 14?\n\n6) In Table 1, make the best results bold instead of in red. \n\n7) Are the AUC values generated on the test set? If so, I'm concerned about overfitting by over tuning the hyper-parameters. ", "summary_of_the_paper": "This paper has proposed a machine learning model to utilize radiomics features in order to improve chest x-ray image classification performance.", "summary_of_the_review": "Overall this paper is well written besides several comments I made above. The classification results on AUC are not convincing to me that integrating radiomics features in this proposed way is better than the state of the art. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mVY6NuR1l7v", "reviewer": "Reviewer_euAn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The proposed approach and considering radiomic features along with learnt features is valuable and very relevant to the community. \n\nMain Concerns and comments to improve the paper:\n- The paper does not consider evaluation of the method on more publicly available datasets and the study is only limited to NIH-14 Chest X-ray data. \n- Baselines and state of the art results on the test data: One important baseline is to make the prediction using only radiomic features.  Also,  [Han’20]  has reported much higher performance on NIH-14 and for the same task which they have not been included in the paper.\nImprovement over the CNN baseline on classification: We see 0.011 improvement in mean AUC over Rajpurkar’17 as one of the baseline which is not significant. \n- Details of the comparison setups are missing. I am wondering if these numbers are collected from the reported values by authors or they ran every SOTA against their data division (less likely since there is no error bound reported for SOTAs). And if they are-implemented or re-evaluated each method what are the details and consideration for a fair comparison and reproducing the results.    \n- Main contribution of the paper is to activate usage or radiomic features within the blackbox  structure of deep networks, however, there is no discussion or insight around the meaning and importance of multiple radiomic features that have been used. There are also works that include prior knowledge for Chest X-ray classification including [Han’20]. These features either unlock a better performance (which seems not being really successful in) or demystify the learnt feature space in a way that is pleasant for human interpreters. The paper does not provide such results. \n\n\nSuggestion to improve the work:\n- It would be useful to add some more explanation around radiomics term, feature extraction procedure and meaning of it, since this is not a common term for the machine learning community.  \n- Consider boosting the results and observation using a few more test data such as MIMIC and CheXpert that have the similar label space as NIH-14 and publicly being used in most of the related papers. \n- The above mentioned datasets also can be used for evaluation of the out-of-domain generalization of the method which has not been discussed in the paper. \n- Experimental setup and evaluation toward the SOTA can improve significantly. For the comparison purpose I would consider producing a statistical error range with either k-fold cross validation or boot-strapping for both CheXT and SOTAs. I would personally see a proper statistical comparison between a single best SOTA with the proposed method more valuable than having multiple models without any statistical validation.\n- Adding interpretation around radiomic feature meaning, how the proposed method unlocked one or more of the 3 listed limitations of CNN methods that has been discussed in the Introduction.\n- It is worth also discussing failure cases and the reasoning behind lower performance for some of the conditions such as effusion and pneumonia. \n\n[Han'20] Han, Yan, Chongyan Chen, Liyan Tang, Mingquan Lin, Ajay Jaiswal, Song Wang, Ahmed Tewfik, George Shih, Ying Ding, and Yifan Peng. \"Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays.\" arXiv preprint arXiv:2011.12506 (2020).", "summary_of_the_paper": "The paper presents a transformer-based architecture for joint localization and classification (of 8 different conditions) of the Chest X-ray images in NIH dataset. The main contribution of this paper is utilization of prior knowledge and radiomic features that have conventionally been used for Chest X-ray condition detection including first order statistics, second-order statistics, shape-based features, and etc. ", "summary_of_the_review": "The main contribution of the paper is enabling the deployment of radiomic features alongside ViT architecture for prediction of Chest abnormality. The study is somehow limited to only one dataset, the results are not supporting the claims and limited. There is no insight provided about how this new feature helps with current shortcomings of CNN including explainability. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3CH0rzW6r7", "reviewer": "Reviewer_ZMa3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ The paper is fairly well written and presents a well-studied literature review.\n+ The paper proposes an end-to-end transformer-based disease classification and localization model incorporating radiomic feature knowledge with cross-attention. The idea of bounding box generation and radiomic feature extraction in a feedback loop module is new and interesting.\n+ Experimental setting including the ablation study for the effect of thresholds on attention maps and the effect of contrastive learning is good. \n\nWeaknesses\n- Exposition of results is not good. CheXT is compared against some weak baselines. A more appropriate CNN baseline should be something that includes similar radiomic feature extraction and bounding box generation [1]. Also, [1] is not cited in Related Work.\n- The paper claims significant performance improvement by the CheXT model although no evidence (statistical test) is provided to support the claim. \n-  Classification performance drops for some of the disease classes (effusion, pneumonia, pneumothorax) after incorporating the radiomics branch. It looks like radiomics branch is hurting the performance for these classes. Is there any impact of label distribution? \n- In Fig. 3, two images look different. Why so?\n- The training of CheXT is not necessarily semi-supervised as every image has at least disease class label. \n- It is not clear how to obtain f_i and f_r; y, y' are not defined Eqn. (1). \n- It would be more viable to report both classification and localization performances on the same 880 images. Is there any correlation between e.g., classification AUC and localization IoU? \n- It is not clear how the model could improve generalization. There is no generalization assessment of the model for out-of-distribution cases. For example, using other datasets such as, CheXpert and MIMIC-CXR.\n- In 4.1, the paper mentions the train/test/val splits, but Table 1 reports results after 3-fold cross-validation. They don't look consistent.\n- In 4.4.2: \"when the IoU is set to 0.1\" should be threshold of IoU. \"For Cardiomegaly, Infiltration, and Pneumonia.\" incomplete sentence.\n- It is not clear how higher values of \\lambda implies lower weight to contrastive loss in Eqn. (3). How does the model perform without the contrastive loss at all (\\lambda=0)?\n- There is no mention of the execution speed, computing resource, and failure cases (if any) for the proposed model.\n- There are certainly other Transformer-based chest X-ray models such as [2].\n\n1. https://arxiv.org/abs/2011.12506\n2. https://arxiv.org/abs/2104.07235", "summary_of_the_paper": "This paper presents a Transformer-based model CheXT for abnormality classification and localization from chest X-rays with auxiliary modality of radiomic features via a feedback loop module. The output tokens from image and radiomics branches are fused by a cross-attention module for the localization and the disease classification is performed from the output of two CLS tokens. The model is trained by jointly optimizing supervised classification focal loss and unsupervised cross-view contrastive loss. The evaluation on the NIH chest X-ray dataset for 8 cardiopulmonary diseases shows improved performance for disease classification and localization.    ", "summary_of_the_review": "This paper presents a Transformer-based model for joint classification and localization of diseases from chest X-ray images, leveraging an image branch and a radiomics branch via a feedback loop module of bounding box generation. While the proposed method has novel and interesting components, the experimentation and evaluation in the paper need to be improved. The proposed CheXT should be compared against the more recent CNN baselines such as ChexRadiNet [1]. There are also some concerns with the experimental details and analyses that need to be addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is on the right track to be accepted after a revision but it is not ready yet. The reviewers were mostly puzzled by different details in the evaluation process, however, as far I can see, most of them should be possible to address. My impression is also that the authors might be more used to a slightly different style of paper writing, more similar to what is typically accepted at MICCAI, ML4H, MIDL etc. I think that a lot can be improved in this respect just by carefully analyzing the reviews.
Decision: Reject