Kati8/ICLR_2023_2025 huggingface dataset:
authors: DIVYA MANOHARLAL BHATIA
authors: Aparna Kanakatte Gurumurthy
authors: Murali Poduval
authors: RUPSHA MUKHERJEE
authors: Avik Ghose
reviews: 
Comments: This paper addresses analysis of radiographs of hip protheses. Specifically, the authors propose a system based on deep learning to perform three analysis tasks automatically: (i) assessment of image suitability, (ii) segmentation of ten zones that are used in clinical practice, and (iii) detection of loosening. Evaluation is reported using a dataset of 206 images to which new annotations were added by a radiologist.
Strengths: The zone segmentation and loosening classification results seem to be good (using standard methods on a relatively small dataset).
Weaknesses: The dataset has only 19 “not fit” examples which is too few to train and test the stage 1 classifier. The accuracy reported for this binary classification task is 94% which is not significantly different from chance level which is 91%.

The dataset was annotated by one orthopaedic surgeon and no indication of annotation reliability was provided. It would be preferable to have multiple annotations of at least a subset to give some idea of inter/intra annotator variability. I imagine there is non-negligible variability in the boundaries between adjacent zones, for example. 

The methods used for image segmentation and classification are fairly standard. It would be good to compare with some pre-trained off-the-shelf architectures. The stage 2 network sounds like it is a U-Net but it is not called that in the paper nor is any prior work on U-Nets cited. The loss employed is also quite widely used but again prior work is not cited to make this clear.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper proposes a 3-stage deep learning algorithm for detection of implant loosening after hip replacement surgery. The method involves fitness check of images, region segmentation and loosening detection. Also, Gruen and Charnley zones annotation and zone-wise loosening information are built on a open-source dataset. The experimental results demonstrate high accuracy of the proposed method.
Strengths: 1. The 3-stage algorithm provide a thorough assessment of implants for the Charnley and Gruen zones, comparing with the common classification of other papers.
2. The experimental results provide evidence of the superiority of this method over other reported methods , which supports its effectiveness.
3. The overall writing is clear and well-structured.
Weaknesses: 1. The originality of the paper, which proposes a method to segment regions and classify loosening, may not be seen as highly innovative. The techniques of image segmentation and classification have been well-explored in deep learning research.
2. The limited availability of data for training and testing the algorithm may limit its effectiveness in real-world scenarios. The blind testing contains only 38 images, which is not enough to prove its robustness.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The author present a mult-step segmentation approach for a medical image segmentation task. They contribute to the anntoation of a public dataset and report favorable performances in their application
Strengths: * The authors adddress an interesting application
* Reported performances are good
* The contribution to a public resource / dataset is appreciated
Weaknesses: * The study focuses on a very narrow application and the solution proposed is highly specialized on this application
* It is not quite clear what type of segmentation algorithm is used, important technical details are missing (also elsewhere)
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The authors have proposed proposed a 3-stage algorithm that detects the sanity of the image for diagnosis, segments into the zones, and then identifies radiolucency within the zones.
Strengths: 1. It is a novel application of existing deep learning techniques to the field of orthopaedics. 
2. Overall empirical scores seem to be quite high, which indicates a good potential of clinical translation.
3. GradCAM maps (issue discussed below) seem to provide good qualitative confidence in terms of importance regions developed by the network.
Weaknesses: 1. The primary results stem from high quality annotations rather than theoretical/architectural advancements. Maybe more suited to a different conference. What could be interesting is a multi-task model as the it can be seen a common backbone can be used from all and the there is a definite inter task benefit which could be presented as a custom loss (for a baseline comparison at the least).

2. The overall outline is not very clear to me in terms of the actual task at the final stage. In Stage 3, to my understanding it is classification, but the section discusses of a segmentation loss? A general figure covering the entire pipeline is highly suggested, which combines Figure 4 and 5 along with an additional figure for stage 2.
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 1