Delving deep into fetal ultrasound video analysis
Keywords: fetal ultrasound
computer vision
medical image analysis
medical computer vision
fetal echocardiography
public health
deep learning
ultrasound image analysis
multimodal learning
continual learning

Over the last five decades, ultrasound imaging has revolutionized the quality of prenatal healthcare through the course of pregnancies. There have been significant achievements towards improving the design, reliability and versatility of ultrasound hardware. A key aspect that has been a bottleneck in the widespread adoption of ultrasound imaging is the requirement of a high degree of expertise for both performing an ultrasound scan and inferring meaningful diagnostic information from the same. With recent innovations in medical image analysis, it is high time that we explored the possibilities of automating the diagnostic understanding of fetal ultrasound videos for clinically useful diagnostics. Specifically, ultrasound image and video analysis can potentially benefit from the improvements in pattern recognition and image understanding brought upon by the recent proliferation of machine learning algorithms. This thesis explores facets of automated analysis of fetal ultrasound videos and frames for a breadth of clinical applications including view classification, anatomical structure detection, continually learning algorithms over variable data distributions and utilisation of human expert knowledge into automated ultrasound image analysis tasks. Concretely, the contributions of this thesis are:
1. Temporal encoding algorithms for spatiotemporal aggregation of fetal ultrasound video sequences towards visibility assessments, standard fetal cardiac viewing plane identification and cardiac region localization: We showed that aggregating spatial features over temporal windows in a hierarchical fashion allowed the identification of fetal standard cardiac viewing planes with an overall accuracy of 0.8358 along with localization of the fetal cardiac region. Localization of anatomical structures in the Four Chamber standard view of the fetal heart was also obtained over available video sequences.
2. Ultrasound video interpretation algorithms that enable novel classes or instances to be added over time without significantly compromising prediction abilities on prior learnt representations: We showed that using the proposed incremental learning pipelines controls forgetting in deep neural networks for ultrasound video analysis as compared to standard transfer learning methods, achieving average drop in accuracies of 0.011 between additions of three novel classes over subsequent learning sessions. This is in addition to generalization to the new classes added with average accuracy on new classes of 0.6420 for the proposed pipeline compared to 0.4508 by naive transfer learning methods. Further, explainable continual learning pipelines for ultrasound image analysis were developed by extending concepts of activation maps to the class incremental learning problem so as to quantify metrics for assessment of multi-class incremental learning in terms of activation map quality.
3. Improving the efficiency of ultrasound image analysis models with sonographer gaze assisted distillation: We combine deep learning based methods with the expertise of sonographers by utilizing information obtained from eye-tracking data as a prior encoding domain expertise. We showed that it is possible to improve performance of reduced or memory-efficient models using sonographer gaze data with the best performing memory-efficient models attaining performance within 5% of conventional models 1000× larger in size.
The ideas explored in the thesis are aimed at contributing to the knowledge of how to design efficient ultrasound video analysis with the reliability of high-fidelity knowledge retention across learning sessions, and to explore the feasibility of improving model performance through the inclusion of human expert knowledge and experience.