Improving medical image segmentation by designing around clinical context
Keywords: Biomedical Informatics.

The rise of deep learning (DL) has created many novel algorithms for segmentation, which has in turn revolutionized the field of medical image segmentation. However, several distinctions between the field of natural and medical computer vision necessitates specialized algorithms to optimize performance, including the multi-modality of medical data, the differences in imaging protocols between centers, and the limited amount of annotated data. These differences lead to limitations when applying current state of the art computer vision methods on medical imaging. For segmentation, the major gaps our algorithms must bridge to become clinically useful are: (1) generalize to different imaging protocols, (2) become robust to training on noisy labels, and (3) generally improve segmentation performance. The current rigorous deep learning architectures are not robust to having missing input modalities after training a network, which makes our networks unable to run inference on new data taken with a different imaging protocol. By training our algorithms without taking into account the mutability of imaging protocols, we heavily limit the deployability of our algorithms. Our current training paradigm also needs pristine segmentation labels, which necessitates a large time investment by expert annotators. By training our algorithms with an underlying assumption that there is no noise in our labels with harsh loss functions like cross entropy, we create a need for clean labels. This limits our datasets from being fully largely scalable to the same size as natural computer vision datasets, as disease segmentations on medical images require more time and effort to annotate than natural images with semantic classes. Finally, current state of the art performance on difficult segmentation tasks like brain metastases is just not enough to be clinically useful. We will need to explore new ways of designing and ensembling networks to increase segmentation performance should we aim to deploy these algorithms in any clinically relevant environment. We hypothesize that by changing neural network architectures and loss functions to account for noisy data rather than assuming consistent imaging protocols and pristine labels, we can encode more robustness into our trained networks and improve segmentation performance on medical imaging tasks. In our experiments, we will test several different networks whose architecture and loss functions have been motivated by realistic and clinically relevant situations. For these experiments, we chose the model system of brain metastases lesion detection and segmentation, a difficult problem due to the high count and small size of the lesions. It is also an important problem due to the need to assess the effects of treatment by tracking changes in tumor burden. In this dissertation, we present the following specific aims: (1) optimizing deep learning performance on brain metastases segmentation, (2) training networks to be robust to coarse annotations and missing data, and (3) validating our methodology on three different secondary tasks. Our trained baseline performance (state of the art) performs brain metastases segmentation modestly, giving us mAP values of