Fellow in a Box: Combining AI and Domain Knowledge with Bayesian Networks for Differential Diagnosis in Neuroimaging.

I 2012, a method based on deep convolutional neural networks won the Stanford ImageNet Challenge for computer vision object identification by a wide margin, almost halving the error rate from the previous year (1). Since then, excitement about the power of deep networks has pervaded all research subjects of artificial intelligence (AI), including radiology. But relying on these methods to identify pathologic features on radiologic images has proved harder than many expected. One reason is the small size of most medical imaging data sets. Also, unlike many computer vision tasks, identifying pathologic features on medical images often requires examination of small and inconspicuous portions of the image. This is why we train radiologists through a prolonged residency and fellowship curricula with careful guidance and attention from senior physicians. In this issue of Radiology, Rauschecker et al (2) launch a combined assault on the task of automated disease classification by using the dual forces of deep learning and Bayesian inference, mimicking the way radiologists become trained. They show good performance on a small but diverse data set of 19 common and uncommon brain diseases, with 91% accuracy of finding the correct entity among the top three most probable conditions, a similar accuracy to that of academic neuroradiologists (86%). How does automated disease classification work? The first step is to allow deep learning to do something it is pretty good at—segmenting lesions on images and classifying them into human-interpretable concepts such as location, signal intensity, or enhancement (3). Given the progress with segmentation challenges such as the Brain Tumor Segmentation, or BraTS (4,5), common architectures and tricks have been established using tomographic data sets on hundreds of cases. In this implementation, lesions were first identified if they had a high signal at T2weighted fluid-attenuated inversion recovery (FLAIR), a sensitive MRI pulse sequence to detect pathologic features but which by itself is nonspecific. Then, the authors use a variety of non-AI tools, most involving thresholding and size criteria, to extract individual “imaging features” about these lesions. Of the 18 features they included, five were related to the signal intensity on other imaging sequences, six were related to volumetric features such as lesion size and mass effect, and seven were spatial features related to location. Most of the “states” of these features were simple, such as a signal on gradient-recalled-echo (low, high, absent) or enhancement (present or absent). While the selection of features was somewhat arbitrary and required much trial and error, the choices made by the authors cover typical items that radiologists look for in the reading room and include in reports as an “explanation” for favoring certain pathologies over others. Besides imaging features, the authors included clinical features, including age, sex, and duration of symptoms (acute or chronic). The 20% absolute drop in AI performance when excluding these features shows their importance. This highlights that a radiologist’s job involves not only looking at images (6), but also integrating imaging and clinical features, an inconvenient truth sometimes forgotten by our computer vision colleagues. With this list of features in hand, the authors applied the “magic” of Bayesian networks (7,8). In brief, a Bayesian network allows you to assign a probability to an event (for example, finding a certain feature, such as high signal on diffusion, given the presence of a specific disease, such as acute stroke). This table of probabilities and diseases is large (19 diseases 3 18 features, each with two or three states) and requires considerable domain knowledge. These probabilities were based on literature if it existed, filled in by the best guesses of neuroradiology experts if not, and then further tweaked on the images in the training set. Applied to radiology studies with success in many settings, this approach often requires features to be hand-fed to the systems rather than arising from an algorithm (8–10). The novelty is the automatic generation of features, although which features to include must still be determined by the investigators. In theory, a full “end-to-end” deep learning system should be able to learn the relevant features as well, but for reasons mentioned earlier, the number of training cases would likely exceed any currently existing database and would be at the expense of interpretability. Fellow in a Box: Combining AI and Domain Knowledge with Bayesian Networks for Differential Diagnosis in Neuroimaging