Comparison of two machine learning classification models for automated histomorphometry

Purpose: Histopathology is an essential component of orthopedic research. Standard methods of pathological quantification rely on panels of expert pathologists that score and grade histological images but these methods are time consuming and often introduce inter- and intra- rater variability. Whole slide image (WSI) analysis and histomorphometry have allowed researchers to evaluate pathologic features with increased reliability and efficacy. However, these histomorphometric approaches still rely on human expertise and manual input which can be laborious. Automated histomorphometry may accelerate this process while improving sensitivity, specificity, and reliability. Here, we evaluate the performance of a machine learning (ML) pipeline trained to discern knee joint tissues and automatically segment whole slides. Methods: Thirty-one Hematoxylin and Eosin (H+E) stained WSI from a murine model of inflammatory arthritis, with pathology ranging from no or mild pathology to severe synovitis, bone erosions and invasive pannus, were used for training and testing. The ML pipeline is described in Figure 1. Two classification approaches were compared: Model A, which was trained to detect 7 tissues, vs. Model B, which was trained to detect 10 tissues. Model A included the white background (other), fat and diffuse tissue, muscle and tendon, synovium, bone (includes bone, meniscus, and cartilage), bone marrow, and growth plate. Model B separates “bone” into cartilage, bone, and meniscus, and additionally classifies slide artifacts such as folded cartilage, blade marks, and larger areas out of focus. Annotation, tiling, feature extraction, and the random forest (RF) model were implemented in QuPath. We utilized a 2/3rd vs 1/3rd, training vs testing split on WSIs to build each model (A: 10 slides for training and 5 slides for testing; B: 22 slides for training and 9 slides for testing). Intensity, smoothing, and shape features were calculated. Model performance metrics (sensitivity, specificity, accuracy, and precision) were then calculated for each class on the testing set, and the weighted F1 (the geometric mean of precision and sensitivity weighted by class) was calculated for each model. Performance metrics were qualitatively compared between Model A and Model B on the test set. Results: We built an RF model with 510 color, shape, and smoothed features (features averaging the features of neighboring tiles) per tile. For Model A this represents ∼70k tiles for training and ∼30k tiles for testing; for Model B this represents ∼140k tiles for training and ∼60k tiles for testing (Figure 2A-D). We achieved a weighted F1 of 0.91 for Model A and 0.83 for Model B (Table 1 A-B). For Model A, the specificity was >0.96 for all classes, the sensitivity ranged from 0.81-0.97, and the F1 score ranged from 0.80-0.98 (Table 1A). For Model B, the specificity was on average >0.97 with a range of 0.75-0.99, the sensitivity ranged from 0.24-0.99, and the F1 score ranged from 0.38-0.99 (Table 1B). Conclusions: We developed a ML pipeline to segment joint tissues and tested how the additional separation of histologically similar tissues affected the performance of our model. Increasing classes in our RF model led to a decrease in weighted F1 score (despite having a larger training sample size). When separated, individual classifications of meniscus, bone, and cartilage in Model B were worse than when one combined class was used, potentially because of the similarities in cell types, morphology, and staining patterns between these tissues. Separately classifying the menisci, bone, and cartilage is important to identify changes in tissues commonly affected in prevalent orthopedic diseases, including osteoarthritis or bone fracture healing. Limitations of this work are (1) the relatively small training dataset and (2) that we choose to not account for staining batch effects, which are common in histopathology. We hope to implement a larger training set (∼500 WSIs) and a deep learning pipeline that utilizes convolutional neural network (CNN) architecture while applying transfer learning techniques from other large image databases which may improve the performance of individual segmentation while controlling for staining batch effect and histological artifact.View Large Image Figure ViewerDownload Hi-res image Download (PPT)View Large Image Figure ViewerDownload Hi-res image Download (PPT)