# Can AI Automatically Assess Scan Quality of Hip Ultrasound?
## 3 Results  
As seen in Table 2, within DS1 (2187 3 DUS images from 508 infants) the AI technique correctly predicted 596/619 images as inadequate quality and 1501/1568 as adequate quality. In DS2 (107 3 DUS images), the AI technique gave the highest accuracy, sensitivity, specificity, NPV and PPV when compared to three non-expert readers. We also tried to address the data imbalance (651 high-quality images vs. 432 low-quality images) by assigning a higher weight to the underrepresented class (i.e., low-quality images). However, this gave a lower AUC of 0.89. In order to check for any systematic error in predictions, we divided the images into subgroups based on age and sex and evaluated all parameters within the subgroups as summarized in Table 3. Accuracies were similar in all subgroups, indicating no systematic error in prediction. Agreement of AI-based predictions with manual scoring was quantified using ICC and [PERSON]'s kappa as shown in Table 4. We used kappa to measure the inter-observer agreement between AI and the expert reader in Dataset (DS) 1. The DS was evaluated by three non-expert readers and one expert (JJ). Compared to the non-expert readers, AI showed higher agreement (0.77 vs. 0.72) with the expert. The difference in kappa scores \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **\# Images, (\#+, \#-)** & **\# Readers** & **Method** & **Scan Adequacy** \\  & & & & **Criteria** \\ \hline \hline \multirow{4}{*}{Dataset 1} & \multirow{4}{*}{2187 (1568, 619)} & \multirow{4}{*}{1} & Holistic score: & \multirow{4}{*}{} \\  & & & Low quality: 1 & \\  & & & High quality: 5 & \\ \hline \hline \multirow{7}{*}{Dataset 2} & \multirow{4}{*}{107 (75, 32)} & \multirow{4}{*}{4} & 10-point scoring 1  & \\  & & & High quality: 10 & \\ \cline{1-1}  & & & Straight and horizontal ilium: 2 & \\ \cline{1-1}  & & & Clearly visible & \\ \cline{1-1}  & & & Os ischium: 2 & \textgreater{}6 \\ \cline{1-1}  & & & Visible labrum: 1 & \\ \cline{1-1}  & & & Round femoral head: 1 & \\ \cline{1-1}  & & & No motion artifacts: 2 & \\ \cline{1-1}  & & & No other image artifacts: 2 & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Method of scoring used to evaluate Datasets 1 and Dataset 2. #+, #- indicate the number of adequate-quality and inadequate-quality images in each dataset. between AI and non-expert human readers was statistically significant (\(p<0.05\)) for readers 2 and 3. Overall agreement of AI with manual reading was quantified using ICC considering AI-based prediction as the 2 nd reader in DS1 and as the 5 th reader in DS2. In both cases, agreement among readers was high (DS1 \(-\)0.95; DS2 \(-\)0.88). Some examples of images of varying quality that were evaluated by AI are shown in Figure 3. All images in row 1 are high-quality images with all necessary landmarks that the CNN correctly identified as adequate. Similarly, images in row 2 are images that are correctly categorized as inadequate by AI. Images in row 3 are moderate-quality images in which there exists variability in manual assessment. The expert radiologist scored two of these images (A, B) as adequate (with score 7) and one image (C) as inadequate (with score 6). AI classified images A and B as inadequate and C as adequate. This row of images highlights the inevitable subjectivity of assessment of intermediate-quality scans. \begin{table}
\begin{tabular}{c c c c c} \hline
**Dataset** & **Readers** & **Ground Truth** & **Kappa** & **ICC** \\ \hline DS1 & AI & Expert & 0.90 [0.97, 0.82] & 0.95 [0.94, 0.95] \\ DS2 & AI & Expert & 0.77 [0.85, 0.68] & 0.88 [0.84, 0.91] \\  & Reader 1 & Expert & 0.72 [0.79, 0.65] & \\  & Reader 2 & Expert & 0.43 [0.50, 0.35] & \\  & Reader 3 & Expert & 0.44 [0.50, 0.38] & \\ \hline \end{tabular}
\end{table} 
Table 4: Agreement of AI predictions with scores provided by expert and non-expert readers. The 95% Confidence Intervals (CIs) of each value is provided in square brackets.