Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a transformer-based architecture for joint localization and classification (of 8 different conditions) of the Chest X-ray images in NIH dataset. The main contribution of this paper is utilization of prior knowledge and radiomic features that have conventionally been used for Chest X-ray condition detection including first order statistics, second-order statistics, shape-based features, and etc. 
main_review: Strength:
- The proposed approach and considering radiomic features along with learnt features is valuable and very relevant to the community. 

Main Concerns and comments to improve the paper:
- The paper does not consider evaluation of the method on more publicly available datasets and the study is only limited to NIH-14 Chest X-ray data. 
- Baselines and state of the art results on the test data: One important baseline is to make the prediction using only radiomic features.  Also,  [Han’20]  has reported much higher performance on NIH-14 and for the same task which they have not been included in the paper.
Improvement over the CNN baseline on classification: We see 0.011 improvement in mean AUC over Rajpurkar’17 as one of the baseline which is not significant. 
- Details of the comparison setups are missing. I am wondering if these numbers are collected from the reported values by authors or they ran every SOTA against their data division (less likely since there is no error bound reported for SOTAs). And if they are-implemented or re-evaluated each method what are the details and consideration for a fair comparison and reproducing the results.    
- Main contribution of the paper is to activate usage or radiomic features within the blackbox  structure of deep networks, however, there is no discussion or insight around the meaning and importance of multiple radiomic features that have been used. There are also works that include prior knowledge for Chest X-ray classification including [Han’20]. These features either unlock a better performance (which seems not being really successful in) or demystify the learnt feature space in a way that is pleasant for human interpreters. The paper does not provide such results. 


Suggestion to improve the work:
- It would be useful to add some more explanation around radiomics term, feature extraction procedure and meaning of it, since this is not a common term for the machine learning community.  
- Consider boosting the results and observation using a few more test data such as MIMIC and CheXpert that have the similar label space as NIH-14 and publicly being used in most of the related papers. 
- The above mentioned datasets also can be used for evaluation of the out-of-domain generalization of the method which has not been discussed in the paper. 
- Experimental setup and evaluation toward the SOTA can improve significantly. For the comparison purpose I would consider producing a statistical error range with either k-fold cross validation or boot-strapping for both CheXT and SOTAs. I would personally see a proper statistical comparison between a single best SOTA with the proposed method more valuable than having multiple models without any statistical validation.
- Adding interpretation around radiomic feature meaning, how the proposed method unlocked one or more of the 3 listed limitations of CNN methods that has been discussed in the Introduction.
- It is worth also discussing failure cases and the reasoning behind lower performance for some of the conditions such as effusion and pneumonia. 

[Han'20] Han, Yan, Chongyan Chen, Liyan Tang, Mingquan Lin, Ajay Jaiswal, Song Wang, Ahmed Tewfik, George Shih, Ying Ding, and Yifan Peng. "Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays." arXiv preprint arXiv:2011.12506 (2020).
summary_of_the_review: The main contribution of the paper is enabling the deployment of radiomic features alongside ViT architecture for prediction of Chest abnormality. The study is somehow limited to only one dataset, the results are not supporting the claims and limited. There is no insight provided about how this new feature helps with current shortcomings of CNN including explainability.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper has proposed a machine learning model to utilize radiomics features in order to improve chest x-ray image classification performance.
main_review: 1) Radiomics is a term that most people in the ICLR community are not familiar with. What is it exactily? Is it a set of radiological/image features or bio-markers in medical images? Or something else? The authors should consider explaining it in plain language or ML language.

2) Prior multi-modal feature fusion work in the chest x-ray domain should be discussed, examples below.

    - Moradi, Mehdi, Ali Madani, Yaniv Gur, Yufan Guo, and Tanveer Syeda-Mahmood. "Bimodal network architectures for automatic generation of image annotation from text." arXiv preprint arXiv:1809.01610 (2018).

    - Wang, Xiaosong, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. "TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays." arXiv e-prints (2018): arXiv-1801.

    - Xue, Yuan, and Xiaolei Huang. "Improved disease classification in chest x-rays with transferred features from report generation." In International Conference on Information Processing in Medical Imaging, pp. 125-138. Springer, Cham, 2019.

    - Liao, Ruizhi, Daniel Moyer, Miriam Cha, Keegan Quigley, Seth Berkowitz, Steven Horng, Polina Golland, and William M. Wells. "Multimodal Representation Learning via Maximization of Local Mutual Information." arXiv preprint arXiv:2103.04537 (2021).

3) I don't see "prior knowledge" and how it's utilized in this work. Isn't the so called prior knowledge in this paper essentially extracted from the data (chest x-ray images)?

4) What are the patch sizes? How are they determined? 

5) There are 14 pathology labels in the NIH chest x-ray dataset. Why/how did the authors select the 8 classes out of the 14?

6) In Table 1, make the best results bold instead of in red. 

7) Are the AUC values generated on the test set? If so, I'm concerned about overfitting by over tuning the hyper-parameters. 
summary_of_the_review: Overall this paper is well written besides several comments I made above. The classification results on AUC are not convincing to me that integrating radiomics features in this proposed way is better than the state of the art.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a Transformer-based model CheXT for abnormality classification and localization from chest X-rays with auxiliary modality of radiomic features via a feedback loop module. The output tokens from image and radiomics branches are fused by a cross-attention module for the localization and the disease classification is performed from the output of two CLS tokens. The model is trained by jointly optimizing supervised classification focal loss and unsupervised cross-view contrastive loss. The evaluation on the NIH chest X-ray dataset for 8 cardiopulmonary diseases shows improved performance for disease classification and localization.    
main_review: Strengths
+ The paper is fairly well written and presents a well-studied literature review.
+ The paper proposes an end-to-end transformer-based disease classification and localization model incorporating radiomic feature knowledge with cross-attention. The idea of bounding box generation and radiomic feature extraction in a feedback loop module is new and interesting.
+ Experimental setting including the ablation study for the effect of thresholds on attention maps and the effect of contrastive learning is good. 

Weaknesses
- Exposition of results is not good. CheXT is compared against some weak baselines. A more appropriate CNN baseline should be something that includes similar radiomic feature extraction and bounding box generation [1]. Also, [1] is not cited in Related Work.
- The paper claims significant performance improvement by the CheXT model although no evidence (statistical test) is provided to support the claim. 
-  Classification performance drops for some of the disease classes (effusion, pneumonia, pneumothorax) after incorporating the radiomics branch. It looks like radiomics branch is hurting the performance for these classes. Is there any impact of label distribution? 
- In Fig. 3, two images look different. Why so?
- The training of CheXT is not necessarily semi-supervised as every image has at least disease class label. 
- It is not clear how to obtain f_i and f_r; y, y' are not defined Eqn. (1). 
- It would be more viable to report both classification and localization performances on the same 880 images. Is there any correlation between e.g., classification AUC and localization IoU? 
- It is not clear how the model could improve generalization. There is no generalization assessment of the model for out-of-distribution cases. For example, using other datasets such as, CheXpert and MIMIC-CXR.
- In 4.1, the paper mentions the train/test/val splits, but Table 1 reports results after 3-fold cross-validation. They don't look consistent.
- In 4.4.2: "when the IoU is set to 0.1" should be threshold of IoU. "For Cardiomegaly, Infiltration, and Pneumonia." incomplete sentence.
- It is not clear how higher values of \lambda implies lower weight to contrastive loss in Eqn. (3). How does the model perform without the contrastive loss at all (\lambda=0)?
- There is no mention of the execution speed, computing resource, and failure cases (if any) for the proposed model.
- There are certainly other Transformer-based chest X-ray models such as [2].

1. https://arxiv.org/abs/2011.12506
2. https://arxiv.org/abs/2104.07235
summary_of_the_review: This paper presents a Transformer-based model for joint classification and localization of diseases from chest X-ray images, leveraging an image branch and a radiomics branch via a feedback loop module of bounding box generation. While the proposed method has novel and interesting components, the experimentation and evaluation in the paper need to be improved. The proposed CheXT should be compared against the more recent CNN baselines such as ChexRadiNet [1]. There are also some concerns with the experimental details and analyses that need to be addressed.
review_title: null

Meta-review: This paper is on the right track to be accepted after a revision but it is not ready yet. The reviewers were mostly puzzled by different details in the evaluation process, however, as far I can see, most of them should be possible to address. My impression is also that the authors might be more used to a slightly different style of paper writing, more similar to what is typically accepted at MICCAI, ML4H, MIDL etc. I think that a lot can be improved in this respect just by carefully analyzing the reviews.
Decision: Reject