# Potential of Deep Learning Methods for Deep Level Particle Characterization in Crystallization
## 3 Material and Methods
### Image Processing  
Image processing is an essential part of characterizing the recorded crystals. In this work, the crystal's size and class, i.e., whether the crystal is a single crystal (sc), agglomerate (ag) or a primary crystal, which is part of an agglomerate (pc\({}_{\text{ag}}\)), is determined. Within this work, this will be referred to as the three-class task. For comparison to previous methods, a restricted variant of this task, differentiating only between single crystals and  
Figure 1: Exemplary image taken with the QICPIC equipped with LIXELL wet dispersing system of the model system L-alanine/water. agglomerates, is considered as well. Consequently, this will be referred to as the two-class task. In the approach presented, this task is accomplished based on the Mask-RCNN method [14] performing object detection and semantic instance segmentation on the images captured. Object detection denotes the task of locating object instances within a given image. This routinely also includes classification, i.e., assigning a class from a given set of classes to each detected object. Semantic instance segmentation additionally performs pixel-wise classification, aiming to provide pixel-accurate masks of objects. In detail, a Mask-RCNN [14] with ResNet-50 Backbone [50] and Feature Pyramid Network (FPN) [51] was used. The implementation deployed here was based on Detectron2 [52]. Training was performed with a manually annotated image data set consisting of 80 images with 5034 annotated instances in total (2362 single crystals, 660 agglomerates and 2012 primary crystals in agglomerates). The images are from previous experiments performed according to Section 3.1. The sizes of the crystals in the data set varied in the range of \(d_{eq}=7.6\) um to \(d_{eq}=1441.7\) um where \(d_{eq}\) is the equivalent sphere diameter. This broad range of expected crystal sizes led to a diverse data set (Table 1), which is important because it has already been shown that scale variation can lead to a loss of accuracy [47]. The data set was split in a ratio of 0.8/0.1/0.1 for training, validation and testing, respectively. For reduced effort in computation and training time, while simultaneously increasing detection accuracy, transfer learning was used in the training process. With transfer learning, the network at hand is pre-trained on (usually more general) data from another domain, before it is finally trained on the domain-specific data. Here, a network pre-trained on the COCO-dataset [53] as provided with the Detectron2 implementation is used as a basis. The actual training was then performed for 30,000 iterations, with a learning rate of \(\eta=4\cdot 10^{-4}\). To increase the variance of the underlying data set, several augmentations were applied in the training process in a randomized fashion. These augmentations comprise transformations (mirroring), as well as filtering operations, including a change of brightness and contrast. To simulate possible noise in the image acquisition process, Gaussian noise of varying intensity was added to the images. As the particles are not always equally within camera focus, also random blurring and sharpening were applied as part of the augmentation. Further details are given in Table 1.  
Based on this method, several Mask-RCNN-based approaches, which are more adapted and refined to the detection task at hand and its application in the process of crystallization, are developed and tested in this study (see Figure 2). For the three-class task, this comprises three different methods. Firstly, a plain Mask-RCNN is trained on the three-class task. This will be referred to as M3_1 in the following. The second variant (M3_2) consists of two separate Mask-RCNNs, the first of them (M2_1) being trained on the two-class task, detecting and differentiating only between single crystals and agglomerates, while the second one being trained for only detecting primary crystals in agglomerates.