# Can AI Automatically Assess Scan Quality of Hip Ultrasound?
## 4 Discussion  
We developed a 3D CNN for automatic ultrasound scan quality assessment and validated it on two datasets, one large (>2000 scans) and one as part of a multi-reader exercise (four human readers, 107 scans). This is the first study of automatic ultrasound scan quality assessment on a large dataset with wide representation from infants of different age groups and sex. On the large data set, our CNN was 96% accurate (with 100% sensitivity and 87% specificity) when compared to expert manual assessment of quality. In the multi-reader study, the AI technique actually performed slightly better than each of the three non-expert human readers in agreeing with the 4 th expert human reader on diagnostic quality as rated in a 10-point scoring system [17]. Based on the commonly used interpretation of ICC (where values less than 0.5 indicate poor, 0.5-0.75 moderate, 0.75-0.9 good and greater than 0.90 excellent [32]), the CNN showed excellent reliability. The 3D CNN approach flags low-quality images containing imaging artifacts that are commonly seen in 3 DUS (as well as 2D sweeps). These artifacts usually occur due to patient movement, hand movement and ultrasound shadowing. Although we do not specifically segment these artifacts, since our technique considers temporal and spatial information around each pixel, images with high occurrence of image artifacts are categorized as inadequate. Since 3 DUS scans consist of a large number (-250) of slices, manual assessment of scan quality is tedious. As a result, anatomical landmarks crucial to DDH assessment  
Figure 3: Examples of images analyzed by AI. Row 1: Images of high quality that were correctly identified AI. Row 2: Low-quality images correctly categorized as inadequate by AI. Row 3: Cases where AI failed are indicated with a red box. Images (**A**,**B**) were categorized as inadequate by AI and two human readers but were categorized as adequate by the expert reader (these images represent false positives in classification). Similarly, image (**C**) was classified as adequate by AI and 2 human readers, but inadequate by the expert reader (false negative). could be missed in 3 DUS. Our automatic assessment technique is fast (average execution time \(\sim\)2 s/image on the Compute Canada Cedar cluster with an NVIDIA V100 GPU) and accurate and it can be used to provide feedback on scan quality to the sonographer in real time during hip examination. This can also be used as a preprocessing step for automatic interpretation of hip ultrasound. Automatic interpretation usually involves calculation of alpha angle and/or predicting the probability of DDH from ultrasound images. Since these techniques [33; 34; 35] generally rely on imaging landmarks such as ilium, acetabulum, os ischium and femoral head, their accuracy depends to a large extent on adequate visualization of these features. Our technique could flag inadequate images upfront and thereby improve the accuracy of these automatic hip interpretation systems. Since the focus of this paper is to apply a CNN model on the new use case of hip image quality assessment, we have not compared it to other 3D networks such as C3D, I3D or 3D U-Net. One limitation of our study is the variability in ground truth as it is based on manual assessment. Images in Dataset 1 were holistically scored from 1-5 by a manual reader. There could be variability in these scores if the same images were to be scored by a second reader. We address this issue to some extent in Dataset 2 which was semi-quantitatively assessed using the scoring system described in [17]. Using the scoring system, the human reader makes a series of (mostly binary) decisions based on landmarks found in the image. Although this reduces the variability, there are inevitably some variations in manual assessment of these individual features, most relevant in intermediate-quality images. For example, as shown Figure 2 row 3, two images were assessed as adequate quality by two human readers but with a low score of 7 points.