Automated ranking of chest x-ray radiological finding severity in a binary label setting

Authors: matthew  macpherson
keerthini  muthuswamy
ashik  amlani
vicky  goh
giovanni  montana
k  macpherson
a  muthuswamy
v  amlani
g  goh
  montana
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 20:23:50

Abstract: Machine learning has demonstrated the ability to match or exceed human performance in detecting a range of abnormalities in chest x-rays. However, current models largely operate within a binary classification paradigm with findings either present or absent using fixed decision thresholds, whereas many clinical findings can be more usefully described on a scale of severity which a skilled radiologist will incorporate into a more nuanced report. This limitation is due, in part, to the difficulty and expense of manually annotating fine-grained labels for training and test images versus the relative ease of automatically extracting binary labels from the associated free text reports using NLP algorithms. In this paper we examine the ability of models trained with only binary training data to give useful abnormality severity information from their raw outputs. We assess performance on a ranking task using manually ranked test sets for each of five findings: cardiomegaly, consolidation, paratracheal hilar changes, pleural effusion and subcutaneous emphysema. We find the raw model output agrees with human-assessed severity ranking with Spearman's rank coefficients between 0.563 - 0.848. Using patient age as an additional radiological finding with full ground truth ranking available, we go on to compare a binary classifier output against a fully supervised RankNet model, quantifying the reduction in training data required in the fully supervised setting for equivalent performance. We show that model performance is improved using a semi-supervised approach supplementing a smaller set of fully supervised images with a larger set of binary labelled images.

Main paper: 
 introduction: 
 Machine learning offers impressive capabilities in medical image classification, and has recently been shown to match or exceed human performance across a wide range of clinical findings in comprehensive multi-label settings (Wu et al., 2020; Seah et al., 2021; Dicente Cid et al., 2023) . However, machine learning approaches to abnormality classification generally operate within a binary classification paradigm, with pre-determined output thresholds for the presence or absence of an abnormality. This omits valuable clinical information since many abnormalities are usefully described on a spectrum of severity. A continuous measure of finding severity would be beneficial to both triage and disease progression monitoring and more closely replicate the skills of an experienced radiologist.This limitation is largely due to the difficulty of obtaining sufficient quantities of granular training labels. Whereas binary labels can be obtained reliably from the original plain text clinical reports using natural language processing (NLP) algorithms (Cornegruta et al., 2016; Irvin et al., 2019; Dicente Cid et al., 2023) , granular severity labels generally require labour intensive hand labelling by radiologists. Estimation of quantitative parameters using small sets of manually annotated training data has been explored for e.g. cardiothoracic ratio (CTR) as a measure of cardiomegaly (Li et al., 2019; Que et al., 2018; Chamveha et al., 2020) and Cobb angle (Chen et al., 2019; Tu et al., 2019; Kim et al., 2020) as a measure of scoliosis severity. Other work considers severity assessment using training data subjectively assigned to severity buckets by expert annotators, including for COVID19 (Zhu et al., 2020; Signoroni et al., 2021) , tumour malignancy level (Mukherjee et al., 2020) , edema progression (Horng et al., 2020) , or general infection severity (Chandra et al., 2022) . In the binary label setting, (Cohen et al., 2020) investigates predictive models of COVID19 severity based on fitting linear regression models to the outputs of a classifier trained on non-COVID19 datasets, demonstrating that the pre-trained outputs for lung opacity and related findings are predictive of COVID19 geographic extent and mortality.The focus of this study is the possibility of estimating the severity of radiological findings without direct, granular ground truth training data. Instead, we have access only to binary labels indicating the presence or absence of findings, regardless of severity. Our objective is to explore whether models trained solely on these binary outcomes can implicitly capture information related to the severity of findings when they are present. To facilitate this, we engaged radiologists to manually evaluate the severity of historical chest x-rays through a method based on pairwise comparisons. This approach requires radiologists to compare two images at a time, simplifying the process of establishing a comprehensive ranking of severity across a dataset. Crucially, these rankings were utilized solely for the purpose of evaluating our hypothesis and were not incorporated into the training data. This study design allows us to empirically investigate the capability of binary-trained predictive models to infer nuances of severity, offering insights into their potential utility beyond simple binary classification tasks.Our main contributions are:• We provide a novel investigation of the continuous logits outputs of a pre-trained chest x-ray multi-label classifier, trained only with NLP-generated binary labels, as a predictor of severity ranking across five radiological findings. We evaluate performance using test sets ranked in order of presentation severity by two radiologists.• We compare this approach with a fully supervised pairwise ranking model, RankNet (Burges et al., 2006) , using patient age as a test variable with complete ground truth ranking available, to assess the relative efficacy of binary labelled versus fully supervised training data.• We show that adding a small fraction of annotated data to a large volume of binary training data leads to stronger severity prediction performance in a combined model. Full details of the dataset acquisition, preprocessing and underlying patient populations can be found in (Dicente Cid et al., 2023) . 
ranked test set development: 
 Test sets of chest x-rays for each of four abnormalities were constructed, fully ranked in order from least to most severe presentation of the named abnormality. In each case, 50 images were sampled randomly from all images in the ground truth image set labelled positively with that finding (i.e. also potentially containing co-morbidities). Each set was then ranked in order of severity of the given finding independently by two radiologists. To facilitate the ranking process we used a pairwise approach using the merge sort algorithm to dynamically select pairs for comparison; such an approach offers a theoretical complexity of O(n log n), minimising the number of individual comparisons required to rank each image set to approximately 200 pairwise comparisons. A web interface (see appendix Figure 4 ) was developed to present the image pairs to the radiologists at 1024x1024 resolution, who then selected the image with the higher perceived degree of severity. Following completion of the ranking algorithm, the radiologist was given the opportunity to review the ordered images to correct ranking errors. Ranking and reviewing each set of 50 images took approximately 1 hour for each radiologist. In cases where there was a difference in rankings of 10 or more between operators, that image was left out of the final test set. After eliminations, we retained 41 ranked images each for pleural effusion and consolidation, 50 for subcutaneous emphysema and 32 for paratracheal hilar changes. The retained images were scored with the mean rank position of the two radiologists to reach a final ranking measure. Two test sets were also constructed with alternative ranking approaches. Firstly, cardiomegaly severity was assessed directly as the measured cardiothoracic ratio (CTR), the measured ratio of heart diameter to maximum thoracic width, giving a quantitative ranking for each image, yielding 173 ranked images. Secondly, patient age converted to a binary 'young/old' label through a boundary at 55 years was used as analogous to a radiological finding with full ground truth ranking available, allowing the entire dataset to be used as training/test data in either binary or ranked modes.We first considered the logit output from a binary classifier as a predictor for the severity of the finding in question, on the assumption that higher probabilities of a finding being present might correspond to more obvious presentation/more severe symptoms. For the radiological findings we used an open source pre-trained classifier, X-Raydar (Dicente Cid et al., 2023) , to generate the logit outputs. For patient age, we custom trained a new binary classification model with our training data using binary cross-entropy (BCE) loss. For this model we used an EfficientNet-B3 CNN backbone (Tan and Le, 2019) , which has reported strong performance in the literature with a relatively light computational overhead.We aimed to investigate if adding varying amounts of fully ranked training data to the full binary labelled training set could improve severity prediction performance, by combining the binary classification task with a fully supervised 'fine-tuning' ranking task. For this task we used patient age as a test variable; a binary label for patient age > 55 years was available for all training images, alongside randomly sampled fully age-ranked training sets of 1,000 to 100,000 images. This was intended to be analogous to binary labels for a radiological finding alongside a limited number of images hand-ranked by clinical experts.In order to incorporate this ranked data into the training process, a pair-wise ranking loss L RankNet was added to the BCE classification loss L BCE . We implemented RankNet (Burges et al., 2005) as the pairwise ranking algorithm. In this pairwise ranking approach, for each pair the difference between the image embeddings is passed through a sigmoid activation to produce a probability of correct ordering; this is compared with the true ordering via a BCE loss summation:EQUATIONwhere:p(y ij ) = 1 1 + exp(M(x i ) -M(x j )) (2)where x i and x j are the two images and M the ranking model. The total semi-supervised loss is the sum of the binary classifier loss with this ranking loss:EQUATION)where λ is a hyperparameter to balance the loss components. The minibatches for the binary and ranking loss components were drawn independently from the binary labelled and fully labelled subsets respectively. Training parameters are detailed in appendix B.  
consistency of radiologist rankings: 
 The reliability of the ranked test sets was assessed by comparing the independent rankings produced by the two radiologists. The relative rankings for each set are shown in Fig. 1 .There was a reasonable level of consistency between the operators, with the strongest relationship in subcutaneous emphysema (Spearman coefficient 0.967, Kendall Tau 0.864) and the weakest for hilar enlargement (0.693, 0.531), with obvious outliers which may result from disagreements early in the merge sort algorithm process leading to significantly different final orderings. In the appendix Fig. 5 we present example images of increasing assessed severity for each of the findings.In Fig. 2 we show plots of the named X-Raydar logits output against the mean ranking of that finding in the relevant test set. Positive Spearman's rank coefficients from 0.563 (pleural effusion) to 0.848 (cardiothoracic ratio) are seen for the findings indicating that the classifier output has some predictive power for finding severity, albeit with a noisy relationship to the assessed rankings.In Fig. 3 we compare age ranking performance for binary classifier models with RankNet models trained on equal sized training sets of varying sizes. Performance on all metrics is stronger (p < 0.001) using RankNet as compared with the binary model, reaching ME of 7.126 years for the binary classifier versus 5.653 years for RankNet with 1, 000, 000 training images. This is unsurprising given the much higher level of supervision being provided with the ranked training data. More interestingly we can approximately quantify the reduction Two main implications for severity ranking can be inferred from this experiment. Firstly, even with ranked data a relatively large amount of images are required to give a robust severity measure; in the case of patient age, approximately 10,000 ground truth ranked images are required to materially improve age prediction above the naive estimation MAE of 8.70 years where the mean patient age is estimated for all images in each category. Fully hand ranking this quantity of images is impractical. Secondly, a large quantity of binary labelled data can go some distance to emulating a smaller ground truth ranked dataset; we estimate that the full binary labelled training set gives a similar outcome to 20,000 fully ranked images which supports the idea that binary labels can yield a useful ranking measure. Finally, we investigate whether ranking performance can be improved in a hybrid semisupervised approach, leveraging the full training set for binary classification along with varying numbers of ranked images in the RankNet training subset. We compare the age ranking performance of (a) a binary classifier trained with the full binary labelled training set, (b) RankNet using only ranked training sets of 1,000, 10,000 and 100,000 images, and (c) a combined loss model using both the full binary labelled training set and the ranked subset. The results of this experiment are summarised in Table 1 (with scatter plots shown in the appendix Fig. 7 ). We observe that the combined loss model outperforms RankNet for all sizes of ranked training subsets, and outperforms the baseline classifier (MAE 7.126 years) with 10,000 (MAE 6.608 years) and 100,000 (MAE 6.145 years) ranked images, although the baseline classifier slightly outperforms the combined model with only 1,000 ranked images added (MAE 7.257 years) .From this experiment we conclude that combining ranked and binary data can lead to improved model performance in excess of either approach in isolation. There also appears to be a threshold amount of ranked data required to boost performance; it is likely that when only a small amount of ranked data is used for the ranking loss, our approach overfits to this subset and degrades overall performance. It is possible that this could be addressed with changes to the training process to reduce the number of batches being fed to the RankNet loss per epoch, as in the current implementation the ranked data is repeatedly cycled over during each epoch of binary classifier training. In any case, in the current set-up we can estimate the number of ranked training images required to give improved performance over the classifier baseline by interpolating between the the experiments performed. This yields an estimate of 1,800 ranked images above which there is improvement to the overall model performance. While this is a substantial amount of ranked data, it is well within the range of a practical number of images for human experts to annotate for real clinical findings, and may be possible to apply to real clinical findings in future. 
conclusion: 
 In this paper we have explored abnormality severity ranking in chest x-rays in the binary label and semi-supervised contexts. We first considered the degree to which nuanced intrafinding severity information can be extracted from a classifier model pre-trained on data with only binary abnormality labelling. The pre-sigmoid raw output from an open-source chest xray classifier, X-Raydar, was found to be predictive of the severity rankings produced by two radiologists on test sets of images for four radiological findings, with Spearman's coefficients from 0.563 (pleural effusion, N=41) to 0.730 (consolidation, N=41), and the cardiomegaly output was predictive of measured cardiothoracic ratio ranking (0.848, N=173).We then examined the relative value of binary labelled versus fully ranked training data in approaching the severity ranking problem, using patient age as a proxy for a radiological finding for which a full ground truth ranking is available. Comparing ranking models trained with binary versus fully ranked training data and corresponding objective functions, we were able to estimate the amount of binary auto-labelled training data which gave similar ranking performance to a fully supervised ranking model. We found approximately a 20 times increase in training data was required to give similar ranking performance using a binary classifier model. This illustrates the benefit of fully annotated data in the training process, but also demonstrates that a sufficiently large quantity of binary labels can potentially give similar benefit to a relatively large amount of resource-intensive annotated data.Finally we explored combining binary and fully ranked training data to to train a single age prediction model with a hybrid objective function. Adding a moderate amount of ranked data to the much larger set of binary labelled data was found to boost regression and ranking performance above the baseline binary classifier level. This indicates that a semi-supervised approach supplementing hand annotated labels with larger volumes of auto-labelled images could offer improvements in model performance versus either approach in isolation, which could be beneficially in optimising clinician time required for manual annotations in future machine learning studies.Approaching severity estimation in a full ranking setting gave valuable insights into both model performance and inter-operator ranking consistency. However, the additional time required to produce ranked test sets as opposed to using a graded severity bucket approach limited both the test set sizes and the number of findings we were able to evaluate with the available resources. To build on this work, incorporating additional radiologists' rankings would allow a more detailed analysis of inter-operator consistency and allow for more robust consensus rankings to be obtained to strengthen our conclusions.  
 
 Appendix:  


Decision: The paper proposes to train a classifier using image-level binary labels of chest pathology, and use the output logits as a measure of severity of that pathology. This is validated on a prioprietary set with radiologist-assessed severity, and a public dataset but with age labels. 

In the initial reviews, the reviewers agree on the relevance of the idea, but as the main issues point out lack of novelty or methodological grounding, the link between the severity and age experiments, and not using public data. After revisions and discussion, two out of three reviewers are not fully satisfied with the responses, and none of the reviewers update their score. On a positive note reviewer one thinks the paper should be presented, and one reviewer comments that the content is not sufficient for a full paper (but in my interpretation, potentially interesting for the conference). 

In addition to the discussion, I would like to add some points that might be relevant to consider for the authors:

•	“binary labels can be obtained reliably from the original plain text clinical reports “ – the reliability of this at least in public datasets has been questioned, see for example the work of Lauren Oakden-Rayner and colleagues. In this sense the age experiment is interesting to add since there is less ambiguity in the labels.

•	Training with binary labels but inferring severity from the classifier, with application in chest imaging, has been explored somewhat in papers on multiple instance learning, for example “Detecting emphysema with multiple instance learning” by Ørting et al. There are multiple differences from the current work so I this is not to comment on the novelty, but there could be some connections there worth discussing. 

 Although the overall score is below average and there are some shortcomings, writing this meta-review convinces me that this is a good paper to discuss at the conference, as one reviewer also points out. 

With this in mind I would recommend for the paper to be accepted if there is enough spots at the conference (this is up to the program chairs), and otherwise I would strongly recommend the authors to submit a short paper to still join the discussion.

Reviews: review: paper_summary: paper_summary: this is an interesting paper with the aim of examining the ability of models trained with only binary training data to give useful abnormality severity information from their raw outputs. Cardiomegaly, consolidation, paratracheal hilar changes, pleural effusion and subcutaneous emphysema are the ones pathologies in the x-ray, and authors tried to rank them with their binary models. Authors correlated the findings with automated vs human ranking and obtained from 0.5 to 0.8 Pearson correlation. Later there is a comparison with RankNet with semi-supervised setting, and improved performance is obtained compared to initially defined model.
main_review: main_review: I think this is a fair conference paper with some innovations, but not so strong in novelties.
The response were satisfactory to some extent,  
due to lack of novelty and coherence problems in the paper, I keep my score at weak accept for a fair conference acceptance, as this paper is in between strong accept and weak accept, to my opinion.
strength_weakness: strength_weakness: - the idea is incrementally novel, but there is technically almost no innovation. 
-lamda_rankinvg is. not available in the equations, I guess there is a typo. it should be LambdaRankNet, I assume.
-pleural effusion is not a superb difficult pathology to rank, any idea why the results are poorer?
questions: questions: 1) can authors have a subsection and clearly mention what is new in this paper? the idea is yes, but not technical.
2) there are some formulations wrong, I assume, not clear if lamda_ranking is ever defined.
3) why some pathologies are having higher uncertainties while their visibility is easier (pleural effusions).
4) are there any other uncertainty algorithm that can be applied to reduce the gap and increase the severity estimation?
paper_summary: paper_summary: This work examines the ability of models trained with only binary training data to give abnormality severity information from their raw outputs. They assess performance on a ranking task using manually ranked test sets for each of the five findings. The pre-sigmoid raw output from an open-source chest x-ray classifier predicted the severity rankings produced by two radiologists on test sets of images for four radiological findings. Then they examined the relative value of binary labeled versus fully ranked training data in approaching the severity ranking problem, using patient age as a proxy for a radiological finding for which a full ground truth ranking is available.
main_review: main_review: After the response given by the author, I think the contribution of this paper is the use of binary logits to assess disease severity. In this idea, the author uses loss_RankNet as the loss function of paired images to evaluate the difference in severity. This idea seems reasonable and there are many experiments. However, the paper lacks both a quantification of severity and an explanation of the phenomena. Based on these observations I give the opinion of weak rejection.
strength_weakness: strength_weakness: 1. The design of disease severity could be more clear, and more details of the relationship between disease severity and label could be provided.
2. The purpose of using age as a feature is not clear.
3. Lack of comparison on public datasets.
questions: questions: 1. Detailed introduction to the settings of disease severity, the relationship between severity and label value, and the relationship between severity and predict logits.
2. More details about the $L_{Ranking}$.
3. Experiments on public datasets would be welcome.
paper_summary: paper_summary: The paper proposes to study the correlation of network outputs, trained for multi-label disease detection with binary outputs in chest X-ray images, to disease severity. The authors show a positive correlation. In addition, they include explicit ranking annotations. and propose using RankLoss to embed this information during training, which shows with correlation with patient aging. Finally, they explore semi-supervised strategies to diminish the amount of necessary ranked labels to calibrate the network.
main_review: main_review: After the authors' rebuttal, I based my evaluation on their main contributions, that: "focuses on utilizing the logit outputs from models trained exclusively on binary labels to infer a nuanced spectrum of severity for radiological findings", and that "The rank loss discussion in our context is hypothetical and aimed at future research directions rather than a description of our current methodology". Then, the main contribution of this work is evaluating the previous authors' model, X-Raydar, trained on binary labels, to severity ranking (Figure 2). Further experiments using RankLoss and Classifier to evaluate age ranking do not provide a direct evaluation of the correlation to severity, and authors claim it to be a hypothetical, future research direction, to predict disease severity. Based on such observation, in my opinion, the paper does not provide enough content for its acceptance as a full conference paper. I justify this claim by the absence of methodological grounding or technical novelty, and the limited experiments on the target setting: disease severity ranking.
strength_weakness: strength_weakness: - The objective of the work, i.e. exploring the capability of a trained network to grade severity in findings/patients, is highly related to the problem of network calibration [a]. Nevertheless, the authors explore this issue through an experimental scope. A more methodological perspective, based on optimization basis, would be expected.
- Indeed, the rank loss explicitly calibrates the outputs, based on the ordering via annotated confidence of severity,  which then provides better metrics in the evaluated tasks, but requires an annotation process.
- Following the previous comment, several baselines on network calibration could be incorporated into the work to complement the Classifier baseline: label smoothing [b], focal loss [c], or logit penalties [d], for example.
- For instance, from Figure 3, comparing RankNet and Classifier methods, RankNet usually obtains an overall smaller logit absolute value scale (I would want to see concrete values for this, but it seems the case for most of the cases), which is known to be correlated with good calibration [d]. This observation is aligned with the better correlation of such logits to severity (in the form of patient age in this case).
- There are some experimental decisions that I do not fully understand: why the correlation to severity predictions (Section 3.2.) is only evaluated using X-Raydar, and not the same methods used for age ranking (e.g. so-called classifier and RankNet)?
- Finally, some experimental details are missing: train/val/test partition, or how the lambda hyperparameter on Eq. 3 is fixed.

[a] On the calibration of moder neural networks, ICML, 2017.

[b] Rethinking the inception architecture for computer vision, CVPR, 2016.

[c] Focal loss for dense object detection, CVPR, 2017

[d] The devil is in the margin: Margin-based label smoothing for network calibration, CVPR. 2022.
questions: questions: Please, see Weaknesses.

Comments: title: Invitation to reply to authors
Responses to reviewer 9jNV (cont 2)
Responses to reviewer 9jNV (cont)
Responses to reviewer 9jNV
Responses to reviewer NYud (cont).
Responses to reviewer NYud
Responses to reviewer tMQD
Response to reviewers
comment: Thanks for your reply and clarification. I think the feedback clarifies my doubts, I have no further questions.
I would want to thank the authors for their detailed clarifications and comments.
thank you for the comments.
Dear reviewers,

The authors have prepared responses to your comments, which you should now be able to see in OpenReview. We encourage you to reply to their comments, and where necessary, adjust your rating. Please do so before the 27th of March.
5/ Finally, some experimental details are missing: train/val/test partition, or how the lambda hyperparameter on Eq. 3 is fixed.

- These omissions have been corrected in the methods and appendix of the revised manuscript. Details of the training splits are provided in our earlier work (Cid et al. 2024, Development and validation of open-source deep neural networks for comprehensive chest x-ray reading: a retrospective, multicentre study. The Lancet Digital Health) as referenced on page 3 but could be detailed in an appendix section if that is insufficient.
3/ Following the previous comment, several baselines on network calibration could be incorporated into the work to complement the Classifier baseline: label smoothing [b], focal loss [c], or logit penalties [d], for example.

- For this study, we decided not to use calibration methods on purpose. Our main goal was to see if models that are only trained to say if an abnormality is present or not can still guess how severe an abnormality is. We're really focusing on what these models can do on their own, without changing them or using special techniques to adjust their outputs. Bringing in calibration techniques now would change the direction of this initial work. It might make it harder to see the main point we're trying to make: that these simple models might already be able to give us useful information about severity, just as they are. We think this is an important idea to explore all by itself, without making things more complicated at first.
That said, we know calibration methods are useful and might help make severity predictions even better in the future. We plan to look into these techniques in our next steps, building on what we've learned from this initial exploration.
For instance, from Figure 3, comparing RankNet and Classifier methods, RankNet usually obtains an overall smaller logit absolute value scale (I would want to see concrete values for this, but it seems the case for most of the cases), which is known to be correlated with good calibration [d]. This observation is aligned with the better correlation of such logits to severity (in the form of patient age in this case).
Your observation regarding the logit scale differences between RankNet and the classifier method offers a valuable perspective on model calibration, highlighting RankNet's smaller logit scale that might suggest better calibration, potentially contributing to its effectiveness in severity ranking as shown in Figure 3. However, it's crucial to reemphasize the core intention of our investigation within this broader discussion. Our study's primary aim is to explore the capability of binary-trained models to infer severity rankings directly from their output logits, without leveraging severity annotations for training. This approach is deliberately chosen to evaluate the inherent potential of these models to extend beyond their original binary classification purpose, venturing into the realm of severity estimation in a clinical context. 
Given this unique premise, the observed differences in logit scale and the subsequent implications for model calibration are intriguing but secondary to our main objective. While traditionally, smaller logit scales are associated with better-calibrated models, our focus remains squarely on demonstrating how unaltered outputs from binary-trained models can provide valuable insights into severity rankings. This exploration seeks to challenge and expand the conventional applications of machine learning models in medical imaging, without initially delving into optimization or calibration based on severity scales Thus, while we acknowledge the validity and potential importance of your points for a more traditional calibration study, they fall outside the direct scope of this initial experimental study.

4/ There are some experimental decisions that I do not fully understand: why the correlation to severity predictions (Section 3.2.) is only evaluated using X-Raydar, and not the same methods used for age ranking (e.g. so-called classifier and RankNet)?

- To clarify this point, in Section 3.2, we used only the pre-trained X-Raydar model for severity prediction on radiologist-ranked findings, while in Section 3.3, we compared our custom classifier and RankNet models on the age ranking task. The primary reason for this difference was the nature of the ground truth data available in each case. For the radiological findings, we had subjective rankings from radiologists, which were more suited for evaluation with the pre-trained X-Raydar model that was specifically designed for these findings. In contrast, for age prediction, we had exact ages available, allowing us to train and compare the performance of our custom classifier and RankNet models. We agree that evaluating the classifier and RankNet models on the radiologist-ranked findings alongside X-Raydar would provide a more direct and consistent comparison across the different severity prediction tasks. While we focused on X-Raydar for the radiological findings in this study due to its strong performance on this specific task in our previous work, we believe that extending the evaluation to include the classifier and RankNet models is a valuable direction for future research. This would help to provide a more comprehensive picture of the relative strengths and weaknesses of different model architectures for severity prediction across a range of tasks and ground truth label types.
Reviewer 9jNV 
Strengths:
We are glad that you found our observation of using output logits to evaluate severity in chest X-rays to be original. This was a key motivation for our study and thank you for noting the positive correlation between disease severity and age in most cases. 

1/ The objective of the work, i.e. exploring the capability of a trained network to grade severity in findings/patients, is highly related to the problem of network calibration [a]. Nevertheless, the authors explore this issue through an experimental scope. A more methodological perspective, based on optimization basis, would be expected.

- Thank you for your feedback and for highlighting the connection with network calibration. However, it's important to clarify the unique context and objectives of our study, which may have contributed to a misunderstanding regarding the applicability of traditional calibration methods here. 
Our research focuses on utilizing the logit outputs from models trained exclusively on binary labels to infer a nuanced spectrum of severity for radiological findings. This approach diverges from conventional scenarios where calibration is typically applied to adjust the confidence levels of binary or multi-class classification outputs to match observed frequencies.
The novel aspect of our work lies in the exploration of whether these binary-trained models, without modification or the direct application of calibration techniques, can inherently provide insights into severity gradations. This investigation is fundamentally experimental and aims to unearth new applications and interpretations of model outputs beyond their initial binary classification purpose.
We believe that the demonstration of a correlation between model logits and expert-derived severity rankings—achieved without the explicit introduction of calibration techniques—serves as a foundational step towards understanding the potential of existing models to offer clinical value in severity estimation tasks. 
We want to make clear that our training used only binary labels (just indicating if an abnormality was present or not) without any information about how severe the abnormality was, as this is not routinely assessed by the radiologists. But for testing, we didn't use any binary labels. Instead, we had radiologists rank the images based on how severe the condition looked, using a method where they compared many pairs of images. Our main goal was to see if the outputs from our model, which was only trained on binary labels, could still guess the right order of severity in these ranked images. We've made some changes to the manuscript to better explain this motivation and unique setup, hoping to clear up any confusion about how we used calibration methods in our study. This should help show how our approach is different and why traditional calibration methods might not fit exactly with what we were trying to do.

2/ Indeed, the rank loss explicitly calibrates the outputs, based on the ordering via annotated confidence of severity, which then provides better metrics in the evaluated tasks, but requires an annotation process.

- Thank you for highlighting the role of rank loss in calibration. Again here, it's important to clarify that our study does not use severity annotations for training the models. Instead, we explore the potential of binary-trained models to infer severity rankings based on their output logits. The rank loss discussion in our context is hypothetical and aimed at future research directions rather than a description of our current methodology. Our approach leverages binary labels for training, and the inference about severity is drawn post hoc from model outputs, without the necessity for additional severity-specific annotations.
4. Eq. 3 says the total loss is the sum of BCE loss and rank loss, and Eq. 1 represents the BCE loss, please describe rank loss in detail. How to set the balance hyperparameter lambda? 

- As per the response to reviewer tMQD (point 2), the notation here was inconsistent and has been corrected. The RankNet loss should have been referenced in Equation 3, and is fully described in Equations 1 and 2. The balance parameter lambda was set by a manual parameter sweep to approximately equalize the magnitude of the two loss components; this has been added to Appendix B. 

5. How to process the corresponding labels for diseases of different severities? Are they all 0-1 labels? Given two CXR images, one is with the most severe effusion, the other is with moderate severity, What are the label values of effusion for the two images? Are the prediction logits output by the model related to the predicted disease severity? Please provide more details. 

- The question regarding labels for different disease severities appears to stem from the misunderstanding discussed in point 2. To clarify, the classifier models in our study were trained using only binary labels (0 for absence, 1 for presence) derived from clinical reports using natural language processing (NLP). No severity information was used during the training process.
Severity rankings were used solely to construct the test sets for evaluating the models' ability to predict relative severity. In these test sets, the images were fully ranked from least to most severe using the pairwise comparison approach described in the response to point 2. There were no label values assigned to the images in the test sets in the sense of 0-1 or severity buckets.
Given two CXR images in a test set, one with the most severe effusion and the other with moderate severity, both images would have a binary label of 1 for the presence of effusion. However, they would have different ranks in the severity ordering produced by the radiologists. The goal of our study was to assess how well the prediction logits of the models, trained using only binary labels, could predict this severity ranking.
So in summary:
-	Training labels: Binary 0-1 only, no severity information
-	Test set labels: No 0-1 labels, images were manually ranked for severity by radiologists using pairwise comparisons 
-	Model predictions: Logit outputs, which we aimed to show were predictive of severity rankings despite the model being trained with only binary labels
Our paper does not claim to predict absolute severity levels, but rather relative severity rankings. The key finding is that the prediction logits of models trained with only binary labels can nonetheless be predictive of these severity rankings, suggesting that some implicit severity information is captured in the model's confidence scores.
We hope this clarifies the labeling and evaluation process used in our study. We have improved the explanation in the paper.

6. How does the model perform on publicly available data sets? For example MIMIC CXR dataset? 

- Unfortunately, we currently lack funding for additional clinician time to produce ordered test sets on MIMIC-CXR, but producing a public benchmark test set for ranking algorithms would be a valuable contribution. We are speaking with our data providers about making available the test images used in a public repo, which would require an exemption under our existing data sharing agreement. Our prior work on e.g. the X-Raydar model indicates that our data generalises well to other DICOM-available datasets, and reproducing the results there will be a priority when resources allow.
1. How accurate is CTR alone in diagnosing cardiomegaly?...

- Thanks for raising this point. The PA/AP effect on the measured vs ‘true’ CTR was not considered for this test set, which uses a mix of positionings. It is likely that making a consistent correction for AP images would somewhat improve the quality of the ranking performance, although at the same time it isn’t obvious if the pre-trained classifier has learnt an implicit adjustment in its output for the two cases, given that the model was pre-trained on comingled AP/PA films. 

2. Page 3 says: "After eliminations, we retained 173 ranked images for cardiomegaly... " How many severity levels are there for each disease? 

3. Page 3 says: "Patient age was used as a ’toy’ proxy for a radiological finding with ground truth ranking available via the chronological patient age.". What does this mean?... 

- We apologize for the confusion in our description of using patient age as a proxy for a radiological finding. In this section, we explored the relationship between a model's predicted age and the ground truth patient age, completely separate from any disease presence or severity. The purpose was to use age as a toy example of a variable with a known ground truth ranking, to allow us to compare the performance of different modeling approaches (binary classifier vs. RankNet) in a setting where we have a definitive ground truth ordering.
Specifically, we trained a binary classifier to predict whether a patient is 'young' or 'old' based on a threshold of 55 years. We then evaluated how well the model's raw output (logit) for the 'old' class correlated with the patient's actual age. This is analogous to training a binary classifier to predict the presence or absence of a radiological finding, and then evaluating how well its raw output correlates with the finding's underlying severity.
Figure 3 visualizes this relationship by plotting the model's output against the true patient age. It does not depict any relationship between age and disease severity. The intent was to use age as a stand-in for disease severity, since age provides a clear, indisputable ranking (older patients are always "more severe" than younger patients in this toy example).
We agree that in a real clinical setting, the severity of a disease may vary greatly over time and is not necessarily correlated with age. The practical value of this experiment was not to suggest that age is a useful proxy for disease severity, but rather to provide a controlled setting in which to compare the ability of different modeling approaches to capture an underlying severity ranking, using a variable (age) for which we have a perfect ground truth ordering. The insights gained from this experiment can inform the design of models for predicting the severity of actual radiological findings, even though age itself would not be a useful predictor in that context.
We have revised the description of this experiment in the paper to clarify its purpose and avoid any confusion about the relationship between age and disease severity.
Reviewer tMQD 
1. Can authors have a subsection and clearly mention what is new in this paper? the idea is yes, but not technical. 

-The final paragraph of the introduction has been redrafted into a bulleted contributions list to make clearer the intended paper contributions. 

2. There are some formulations wrong, I assume, not clear if lamda_ranking is ever defined. 

- The formulations in this methods section have been corrected to be consistent between the text and equations, and equation 3 amended slightly to show lambda as a separate hyperparameter preceding the ranking loss. 

3. Why some pathologies are having higher uncertainties while their visibility is easier (pleural effusions). 

-This is an interesting point that we didn't have space to fully discuss in the paper. Inter-operator ranking agreement was actually weakest for pleural effusion among the findings considered, indicating that objectively ranking this finding may have been more challenging or controversial for the clinicians. Recruiting additional radiologists to repeat the ranking could help shed light on this. Additionally, if the model is generally confident in its pleural effusion predictions, the severity range over which uncertainty is informative may be smaller. Investigating these factors would be interesting future work to better understand the performance variation across findings.

4. Are there any other uncertainty algorithm that can be applied to reduce the gap and increase the severity estimation? 

- Exploring alternative uncertainty quantification approaches is a great suggestion for potentially improving severity estimation. Techniques like model ensembling, test-time augmentation, or Bayesian approximation could be interesting to investigate. However, a thorough comparison of uncertainty methods was beyond the scope of this initial proof-of-concept study. We see this as a valuable direction for future work building upon our findings.
We thank the three reviewers for their detailed comments, in response to which we have drafted a revised manuscript. Below, we first provide a general response to the reviewers, followed by point-by-point responses to each reviewer referencing the relevant changes made where relevant. 

Overall, the reviewers recognise that the subject matter (i.e. assessing finding severity from weakly supervised data) has novelty and is of clinical interest but highlight the lack of technical innovation in the work. This is entirely fair, as the work is intentionally an experimental investigation of whether additional useful information can be derived from models pre-trained with binary auto-labelled data, rather than proposing a new methodology per se. 

We would argue that an experimental demonstration of a potential broader use case for the wide range of existing pre-trained models is of high interest to the community and outweighs the noted lack of technical innovation. As noted by reviewer tMQD, we feel that the experimental contribution is of sufficient interest for acceptance as a conference article, and developments on the technical and modelling side would form the basis of further work for conference or journal submission in future. 

Point by point responses are attached as comments to each review.