# Potential of Deep Learning Methods for Deep Level Particle Characterization in Crystallization
## 3 Material and Methods
### Image Processing
It is to be noted that the two-class Mask-RCNN M2_1 is also used separately, as detailed below, and thus explicitly labeled, while the other one is not. These two Mask-RCNN are then applied consecutively and the individual results are combined to form a solution to the three-class task. Thirdly, this approach is further complemented by a post-processing step in order to incorporate prior knowledge into the detection process. As pc\({}_{\text{ag}}\) are by definition always part of ag, this information is used to improve the detection results further. This method is referred to as M3_2 PP. For the two-class task, which mainly serves as a point of direct comparison to the approach of [PERSON] et al. [15], a Mask-RCNN is directly trained on this task, without any modification. This variant will be referred to as M2_1. In addition to the Mask-RCNN approaches, the method of [PERSON] et al. that is only applicable to the two-class task is represented in Figure 2 by the box labeled Feature Extraction+ANN. The method of [PERSON] et al., not being based on deep learning techniques, conceptually differs from the Mask-RCNN approaches. In a first step, crystals are detected on binary images and image descriptors (features) are calculated, which serve as input for an ANN for classification in a second step. To obtain binary images of the recorded grayscale images, a binarization is performed [23]. Reinforced by image enhancement and dynamic background subtraction, differences in contrasts between crystals and background are used by global thresholding (Otsu method [54]). The detected crystals are further processed by morphological closing and region filling, before they are passed to the classification. For classifying single crystals and agglomerates in total 19 different image descriptors, e.g., circularity, convexity and solidity, are calculated and ranked based on proportional similarity identifying the most striking ones to distinguish between single crystals and agglomerates. These image descriptors are the input of a shallow artificial neural net (ANN) capable of classifying the crystals. The detailed inputs and the architecture of the ANN are given in Table 1.  
The classical approach is directly compared to the two-class task implementation of the CNN differentiating between single crystals and agglomerates. As a basis of comparison and further evaluation of the developed deep learning approaches, all results are evaluated with respect to the COCO-metrics [53]. The principal measure is the mean average precision, \(mAP\) or just \(AP\), for short. This measure is an extension of the precision/recall-scheme of binary classifiers for the task of object detection and semantic instance segmentation [55]. Along with the main \(AP\) measure, the \(AP_{50}\) variant is considered here, as well. In the \(AP_{50}\) measure, a 50% overlap of detection and ground-truth is sufficient for positive evaluation, meaning that it is usually more tolerant with respect to inaccurate location. Furthermore, COCO-metrics can be evaluated grouped by object size for small, medium-sized and large objects, this leads to \(AP_{s}\), \(AP_{m}\), and \(AP_{l}\) for our application. Table 1 gives a more detailed overview of the ranges and associated sizes. Finally, the resulting AP are also given for every class (sc, ag, pc\({}_{\text{ag}}\)). All metrics are derived for the pixel-wise segmentation masks (\"segm\") of the detected objects. Besides the COCO-metrics also the target parameters agglomeration degree \(Ag\) and the average number of primary crystals in agglomerates \(\overline{pc/ag}\) are evaluated. \(Ag\) is the ratio of the number of agglomerates to the total number of crystals including agglomerates and single crystals and \(\overline{pc/ag}\) is the number of primary crystals in agglomerates divided by the number of detected agglomerates. Thus, \(Ag\) and \(\overline{pc/ag}\) characterize the material systems in terms of agglomeration.