# Potential of Deep Learning Methods for Deep Level Particle Characterization in Crystallization
## 2 State of the Art  
Image analysis consists of two major coupled steps, the image acquisition as well as the image processing to determine the desired particle attributes. Images can be recorded in various ways. On the one hand, offline measurements are conducted, which are characterized by time- and labor-intensive sample preparation and potential change in particle properties, e.g., size and agglomeration degree due to downstream processes [16; 17]. Instead, automated measurements are aimed for fast dynamic image acquisition, which can be implemented in the crystallization process later on. A detailed overview about applications and publications of image acquisition set ups during crystallization is provided by [PERSON] et al. [18] and is out of the scope for this work in such a detailed manner. In-line set ups record images with a probe directly inside the crystallizer, such as the PVM from Mettler Toledo or from outside the crystallizer, e.g., [PERSON] et al. [19] and [PERSON] et al. [20]. In-line acquisition is characterized by a very low up to no impact on the crystallization, but often by a reduced image quality as well. High solid concentration, which inevitably occurs in the course of crystallization, out of focus particles as well as inconsistent blurry and noisy background have to be tackled in the image processing, making it challenging [21]. Better-quality images are recorded with flow cell microscopy. Suspension is pumped directly from the crystallizer or storage tank through the flow cell where images are taken. By adjusting the flow cell, e.g., cell depth, and the solid content by dilution, images are acquired with uniform background, good contrast and adapted focus suitable for the investigated crystals. In this way, images can be taken after the process or online during crystallization. Nevertheless, it must be considered that flow cell microscopes coming along with the expense of a periphery that can potentially disturb the process [12]. For cooling crystallization, the periphery can only consist of a tempered sampling loop [7; 22; 23; 24; 25; 26] or to adjust particle concentration by an additional dilution system [12; 13; 27]. Image processing is the next inevitable step to obtain the desired quality attributes of the product. It is by far not trivial and still many challenges have to be overcome, especially image processing is limited for high solid concentrations and in the level of detail regarding overlapping and agglomerated particles [28]. By identification of primary crystals in agglomerates instance segmentation is necessary including the steps detection of crystals and subsequent classification of the detected crystals. To distinguish between objects and background, there are different segmentation approaches, e.g., the simplest methods, thresholding and edge detection [29]. For a more detailed characterization in terms of classification or identification of the crystal habit, machine learning methods including feature extraction, encoding and classification can be implemented. Based on extracted image features and the calculation of image descriptors such as roundness, elongation and many more, discriminant function analysis (DFA) [7; 30], principal component analysis (PCA) [4; 31; 32] shallow neural nets [15; 33; 34] or support vector machine (SVM) [35; 32] can be applied to differentiate between single crystals and non-single crystals. A more in detailed classification to distinguish between agglomerates with different number of bounded primary crystals has been performed by [PERSON] and [PERSON] [4] and by [PERSON] et al. [7] by differentiating between medium and large agglomerates. [PERSON] and [PERSON] [34] apply neural nets to additionally estimate not only the number but also the size of primary particles in agglomerated nano-particles. Other methods focusing on segmentation to disassemble crystal complexes into primary crystals. Either they are based on geometrical considerations [20; 36; 37; 38; 39] or take into account gray values of the image [40].