<figure><img src="image_1.png" /></figure>

HAL
open science

# Application of Mask R-CNN for lab-based X-ray diffraction contrast tomography

H. Fang, E. Hovad, Y. Zhang, D. Juul Jensen

► To cite this version:

H. Fang, E. Hovad, Y. Zhang, D. Juul Jensen. Application of Mask R-CNN for lab-based X-ray diffraction contrast tomography. Materials Characterization, 2023, 201, pp.112983. 10.1016/j.matchar.2023.112983. hal-05080001

HAL Id: hal-05080001

https://cnrs.hal.science/hal-05080001v1

Submitted on 22 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

# Application of Mask R-CNN for lab-based X-ray diffraction contrast tomography

H. Fang<sup>a,b,c,*</sup>, E. Hovad<sup>d</sup>, Y. Zhang<sup>a</sup>, D. Juul Jensen<sup>a,*</sup>

## ARTICLE INFO

### Keywords:

Instance segmentation

Deep learning

X-ray diffraction

Grain mapping

Tomography

## ABSTRACT

Segmentation of spots in diffraction images is critical for accurate grain mapping in 3D. When the grain mapping is done by the recently established lab-based X-ray diffraction contrast tomography (LabDCT), diffraction spots suffering from low signal-to-noise ratios impose a severe challenge in precise identification of the spots using conventional image filters, thereby hindering the detection of small grains and limiting the spatial resolution. To overcome this challenge, we have applied an automatic instance segmentation deep learning network based on Mask R-CNN (two-stage region-based convolutional neural network) for finding spots in LabDCT images. The training data for the neural network was synthesized by combining virtual noise-free images (obtained from a forward simulation model) and noise-only images (obtained by filtering out diffraction spots in experimental images). Based on the diffraction spots deducted by the forward simulation model, data labelling and annotation was thus performed in an unsupervised manner without the need for tedious human labelling. By applying the network in a PyTorch framework called Detectron2, we show that the trained model performed significantly better than the conventional method in spot segmentation, resulting in a better grain reconstruction, subsequently. The work illustrates the potential of deep learning for improving LabDCT and other grain mapping techniques in a broader sense.

## 1. Introduction

Techniques for non-destructive grain mapping in 3D have been well established using high-flux X-rays from synchrotron facilities during the last two decades. These techniques, including three dimensional X-ray diffraction (3DXRD, [1,2,3]), diffraction contrast tomography (DCT, [4]), differential aperture X-ray microscopy (DAXM, [5]) and X-ray dark field microscopy [6], offer outstanding possibilities to characterize grain orientations, sizes and shapes on broad length and temporal scales [7]. To avoid constraints related to limited access to synchrotron sources, lab-based X-ray diffraction contrast tomography (LabDCT) has been developed as a 3D grain mapping technique that can be routinely used at home laboratories to characterize grain orientations and shapes [8,9,10]. Several works have used this novel technique to shed new lights on processing of e.g. powders and bulk alloys (e.g. [11,12,13]).

LabDCT operates in a transmission geometry, where a conical poly-chromatic beam travels through a sample. The directly transmitted

photons are blocked by a beam-stop placed in front of the center of a 2D detector, while the diffracted photons (that fulfill Bragg's law) are recorded by the 2D detector and form diffraction spots. Grain structures are reconstructed from these spots recorded at different angles of a full 360° sample rotation. For an accurate grain reconstruction, a key step is to obtain precise spot segmentation from the diffraction images. However, undesired background noise, caused by many factors such as inelastic scattering and fluorescence of the sample, inhomogeneity of the X-ray source, defects and point spreads of the detector, affects the precision of the spot segmentation. A resulting poor spot segmentation may lead to wrong indexing (including un-indexed and falsely indexed grain orientations) and unsatisfactory grain shape reconstruction. Conventionally, a rolling median background correction combined with various filtering processes, such as the Laplacian of Gaussian filter, are applied for spot segmentation. This method generally works well for spots with good signal-to-noise ratios, whereas it can lead to over or under segmentation of spots with low signal-to-noise ratios. The latter hinders

<footnote><sup>a</sup> Department of Civil and Mechanical Engineering, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark</footnote>

<footnote><sup>b</sup> Université Grenoble Alpes, Grenoble-INP, CNRS SIMaP, 38000 Grenoble, France</footnote>

<footnote><sup>c</sup> European Synchrotron Radiation Facility, 71 Avenue des Martyrs, 38000 Grenoble, France</footnote>

<footnote><sup>d</sup> Department of Applied Mathematics and Computer Science, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark</footnote>

<footnote>* Corresponding authors at: Department of Civil and Mechanical Engineering, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark. E-mail addresses: haixing.fang@grenoble-inp.fr, haixing.fang@esrf.fr (H. Fang), doje@dtu.dk (D. Juul Jensen).</footnote>
improvement in spatial resolution and sets a grain size detection limit (small grains cannot be resolved). An additional complication is that this conventional routine requires extensive trial and error testing to find an optimal set of processing parameters [14].

Recently, there has been significant progress in using deep learning methods for image analysis, driven by an increase in compute power and academic competitions using public dataset. Specifically, for image segmentation, two different types, namely semantic segmentation (partitioning different regions into different labeled classes) and instance segmentation (identifying each instance of a class in an image), are dealt with by deep learning methods. We previously established a deep learning model [15,16] to perform semantic segmentation of the diffraction spots in LabDCT images. This model allows efficient removal of background noise and a better segmentation of diffraction spots compared to the conventional routine, and hence an improved 3D grain mapping. However, due to the high demand of computing power for the model, the starting training images, as well as the real images, had to be down sampled from high to low resolutions. And even more importantly, with semantic segmentation overlapped spots cannot be separated as they are considered as one collected object class. The latter is not optimal for grain reconstruction [14], and together both issues restrict the performance of this model.

State-of-the-art instance segmentation models, which identify and separate individual instances from each other and do not compromise the image resolution for making predictions if the image content is not too large, can overcome both these issues. Today, Mask R-CNN, developed by Facebook AI Research in 2017, as an extension from previous models (Faster R-CNN, [17]; Fast R-CNN, [18]; R-CNN, [19]), is widely recognized as one of the most accurate models for instance segmentation, e.g. achieved top scores on the COCO instance segmentation challenge [20]. It has a flexible architecture and can be used for a wide range of applications. Mask R-CNN is thus an appealing tool to tackle the challenges of the spot segmentation.

With the aim to explore the potential of Mask R-CNN for i) improving the accuracy and ii) automating the process of segmenting diffraction spots, we established Mask R-CNN particularly for LabDCT, but also with a flexibility to be applicable for diffraction images acquired from similar techniques, such as synchrotron DCT. We first trained a Mask R-CNN model using synthetic image data containing realistic noise, and then applied it for real LabDCT images. Similar to our previously developed deep learning framework for removing background noise [15], the synthetic images were generated by a forward simulation model combined with a generic approach to extract a realistic experimental background noise. The diffraction spots of the synthesized training images were tuned to capture the common range of signal-to-noise ratios for experimental images. This approach avoids laborious manual data labelling. The segmentation results are compared to those from the conventional spot segmentation method. The resulting 3D grain reconstructions are further compared to results obtained by synchrotron X-ray diffraction (to be considered as the ground-truth) and an assessment is carried out to evaluate the performance of the novel Mask R-CNN method for improving the LabDCT technique.

## 2. Data and methods

### 2.1. Generation of synthetic data

Synthetic image data was created to train and test the deep learning Mask R-CNN model for segmentation of the diffraction spots in LabDCT images. We first used a forward simulation tool to generate noise-free LabDCT images ($I_{simu}$), where only diffraction spots are present, and they are labeled. Three virtual cylindrical Al samples, with random grain orientations and different grain size distributions, were synthesized as input for simulating LabDCT diffraction images at sample rotation angles from -180° to 180° with a step size of 2°, resulting in 181 projection images for each sample. Each projection image contains all the possible

diffraction spots from the first 4 {hkl} families of all grains in each sample, excluding the central region where a virtual beam-stop shield all the signals (including both the direct beam and any diffraction spots expected in this region). The simulated diffraction image has a size of 2032 × 2032 pixels with an effective pixel size of 3.36 µm in a 16-bit format assuming a dynamic range covering the pixel value from 0 to 65,535. The simulations were performed in a Laue focusing geometry, i. e. with the sample-to-source distance ($L_{ss}$) being the same as sample-to-detector distance ($L_{sd}$), each being 11 mm. An X-ray spectrum from a tungsten anode X-ray tube, operating at 140 kV, was used. The maximum pixel value of the simulated image did not saturate the synthetic detector, but some pixel values were larger than 50,000. More details about the forward simulation model can be found in [21] and the code is freely accessible (https://github.com/haixingfang/LabDCT-forward-simu-model). To maximize variations in the projections, concerning the number of spots, spot sizes and intensities, we carefully tuned the number of grains and the grain size distribution in each of the three virtual samples, with the intention of capturing features of small spots as precisely as possible. Table 1 reports the main characteristics of the virtual input samples, with the grain size given as the equivalent spherical grain diameter.

Next, the simulated noise-free images were combined with a normalized background image ($I_{BG}$) containing noise extracted from typical experimental images (representing the long-range noise) plus a Poisson-type noise, which mimics the short-range (high-frequency) noise. The process to generate the diffraction images including the spots and the noise ($x$) for each pixel can be expressed as:

$$x_0 = I_{simu} + R_{BG} * I_{BG}, \\ R_{BG} \in random([I_0 - \sqrt{I_0}, I_0 + \sqrt{I_0}]), \quad (1)$$

$$x = x_0 + \tau \text{Poisson}(x_0/2), \quad (2)$$

where $x_0$ is the image containing the spots and the long-range noise, $R_{BG}$ is a value randomly generated in the range $[I_0 - \sqrt{I_0}, I_0 + \sqrt{I_0}]$ determined by a constant $I_0$, Poisson($x_0/2$) represents the short-range noise following a Poisson distribution with a mean value of $x_0/2$ for each pixel, and $\tau$ is a factor used to tune the short-range noise level. Typically, we set $\tau = 1$ and $I_0 = 110$, which is a common background pixel value for experimental LabDCT diffraction images [22]. We tuned these two parameters to enhance the variations in the signal-to-noise ratios for the generated diffraction images. This method has shown to be able to capture well the features of the real experimental LabDCT diffraction images. An example of $I_{simu}$ and $x$ is shown Fig. 1a-b.

Labeled image data serves as the target for training the deep learning network. Accurately labeled spots is essential for obtaining successful training. Although there is no background noise affecting the spot segmentation for $I_{simu}$, spot overlap affects the accuracy of the spot labeling. As reported in our previous work [14], a better separated segmentation of the overlapped spots results in a better grain indexing, even though the grain reconstruction algorithm can tolerate a small degree of spot overlap. We have tried different ways of spot labeling. Direct binary thresholding segmentation labels the overlapped spots as one single spot (Fig. 1c), whereas a Laplacian of Gaussian (LoG) based segmentation can handle the 'gentle' spot overlap, but not severe overlap (Fig. 1d). Therefore, we proposed a method combining the LoG segmentation with a spot splitting algorithm. In the splitting algorithm, a gradient matrix (g) and a Hessian matrix (h) are calculated for each connected region in $I_{simu}$, then an operation of $(g' * h * g)/(g' * g)$ is performed (inspired by [23]). As shown in Fig. 1e, almost all the overlapped spots can, with this method, be fully split and just a few (with a number fraction <1%) are only partially split. It should be pointed out that we also tried a marker-controlled watershed [24] method, but the results are poorer than those obtained by our current method.

In total, 1086 input images $x$ were created by using $I_{simu}$ from the three virtual samples and a set of $I_{BG}$ from 3 different experimental
<caption>**Table 1**</caption>

Characteristics of three virtual input Al samples. Mean values and standard deviations are given for grain size (equivalent spherical diameter) and number of spots per projection. Note that the later corresponds to the total number of isolated spot regions, which is smaller than the total number of unique *hkl* reflections as spot overlap occurs in the three samples.

<table><thead><tr><th>Name</th><th>Cylinder dimension,<br>Diameter × height (µm²)</th><th>Number of grains</th><th>Grain size,<br>(µm)</th><th>Number of spots<br>per projection</th></tr></thead><tbody><tr><td>Al-fine</td><td>150 × 225</td><td>891</td><td>17.5 ± 7.2</td><td>1797 ± 38</td></tr><tr><td>Al-medium</td><td>200 × 300</td><td>667</td><td>27.0 ± 9.2</td><td>1411 ± 31</td></tr><tr><td>Al-large</td><td>400 × 600</td><td>485</td><td>63.6 ± 14.1</td><td>1028 ± 26</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>**Fig. 1.** Synthetic image data for training the Mask R-CNN model. (a) Simulated LabDCT image ($I_{simu}$) containing spots (pixel value >0) and zero-background (pixel value = 0) from the sample Al-large; (b) input image (x) containing both spots and noise. (c)-(e) labeled images after thresholding the spots in $I_{simu}$ using: (c) a binary thresholding; (d) a Laplacian of Gaussian (LoG) based method and (e) a combination of LoG and a splitting algorithm. Full-size images are shown in the top row and zoom-ins are shown in the bottom row. The labeled spots are shown in random colors in (c)-(e). The light blue and yellow boxes (both contain a 3-spot overlap) show that the overlapped spots are labeled as one single instance in (c) and (d), but split into 3 instances (i.e. are fully split) and 2 instances (i.e. are partially split), respectively, in (e). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

conditions, resulting in 1086 spot segmented and labeled images, $y_c$. These images were subsequently used for training a Mask R-CNN model.

## 2.2. Data labeling and augmentation

The labeled images were transformed into a COCO dataset format

[25] where each spot was labeled as one individual instance with a mask and a bounding box (see Fig. 2a). To increase the amount of training data, image augmentation, including random flipping, random brightness, random contrast and cropping, was applied. Fig. 2b shows an example of an augmented image. The settings of image augmentation are given in Table S1 in *Supplementary Materials*. To reduce the demand

<figure><img src="image_7.png" /><figcaption>**Fig. 2.** Example image illustrating the image labeling and augmentation. (a) labeled image where each spot is labeled as one individual instance in a COCO dataset format, image width × height = 2032 × 2032; (b) augmented image after resizing, cropping and flipping of the area in image (a) indicated by the red rectangle, image width × height = 348 × 163. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>
of training computation, we also reduced the widths and heights of the input images from 2032 × 2032 to 1200 × 1200 pixels by binning neighboring pixels using a nearest neighbor approach. Note that this resizing might reduce the resolution. However, the effect is expected to be limited to an error of about 1–2 pixels for the ultimate prediction output.

## 2.3. Algorithm of Mask R-CNN for spot segmentation

The applied Mask R-CNN is a two-stage algorithm with a backbone network, which extracts an image feature map from the input image at different scales, and a regional proposal network, which detects object regions (as an object or not an object) from the multi-scale features and generates corresponding cropped features using a ROI (region-of-interest) Head [20]. Then the cropped image features are re-used in the second stage to find the classes, the bounding boxes and the masks for instance segmentation. A feature pyramid network (FPN) is included to help improve the feature extraction.

In the second stage, a Box Head is used for refining the classes and bounding boxes, while another Mask Head creates the instance segmentation. In this work, only one class (namely “spot”) is used and for each spot the network as output gives a bounding box and a segmentation mask with a confidence score (see Fig. 3).

To train the Mask R-CNN model, the synthetic images for both input image $x$ and labeled content image $y_c$ (ground truth) were split into a training dataset (95% = 1031 images) and a validation dataset (5% = 55 images). Similar to [20], at each iterative loop of training, a total loss (loss_total), measuring the difference between an output image ($\hat{y}$) and a content image ($y_c$), is calculated for both the training and validation datasets. This total loss is a sum of five different losses, as expressed by:

$$ loss\_total = loss\_rpn\_cls + loss\_rpn\_loc + loss\_cls + loss\_box\_reg + loss\_mask \quad (3) $$

where the first two losses (loss_rpn_classes and loss_rpn_loc, corresponding to the classification loss and localization loss, respectively, in the regional proposal network) measure the performance of the first-stage region proposal network (RPN), loss_classes and loss_box_reg are losses in the ROI Head and loss_mask is the loss in the Mask Head (output masks from an input of detected bounding boxes). More specifically, loss_rpn_classes detects if there exists an object or not in the anchor box (that is a set of pre-defined bounding boxes); loss_rpn_loc measures the loss in localization of the predicted regions in the RPN; loss_classes measures how good the model is at labeling a predicted box with the correct class; loss_box_reg measures the difference between the predicted box and the true box; loss_mask measures the accuracy of the predicted binary masks. When both the training loss and the validation loss are stable (do not decrease further),

the model is considered as trained and ready for detecting spots in real experimental LabDCT images.

During the following detection of real diffraction spots from experimental projections, there are mainly two steps to control/filter the predicted boxes and masks: 1) filter out the bounding boxes with low confidence scores (as shown in Fig. 3, each spot has a confidence score); 2) apply non-maximum suppression (NMS) to remove overlapping bounding boxes. This NMS runs for each box from the highest to the lowest confidence scores, and removes those with a too large area-overlap (defined as intersection-of-union, IoU). Notably, the IoU can be calculated based on the overlap of either the bounding boxes (IoU_box) or the masks (IoU_mask); an illustrative sketch of this is presented in Fig. S1 in Supplementary Materials. Since the diffraction spots are usually crowded in the LabDCT images, the overlap of bounding boxes is more severe than the spot masks. In this work, we use both IoU_box and IoU_mask for the calculation to improve the accuracy of the prediction.

## 2.4. Implementation

The Mask R-CNN model was, in the present work, implemented based on a PyTorch framework with a Detectron2 model (https://github.com/facebookresearch/detectron2). The training was performed using the above mentioned 1086 synthetic images, where 95% (= 1031 images) were randomly grouped into the training dataset and 5% (= 55 images) into the validation dataset.

A standalone computer equipped with a 16-GB GPU (NVIDIA Quadro P5000) was used to train the model. Fig. 4a shows different contributions of the five different losses to the total training loss. It shows that the four losses, loss_rpn_loc, loss_rpn_classes, loss_box_reg and loss_classes, decay fast to be stable with relative small values, whilst loss_mask requires more iterations to become flat and this loss function plays a major role for the total loss. Notably, a spike of the total loss is observed in Fig. 4a at around the 26139th iteration and is mainly attributed to an increase of about 0.2 for the loss_rpn_classes, which may be caused by over-fitting but was quickly regularized back to normal. Fig. 4b shows that both the total training and validation loss curve saturates before 50,000 iterations, which took ~33 h. The model training was thus stopped there.

The main settings for the training and prediction are listed in Table S2 in Supplementary Materials. It was found that suitable criteria for generating the output are a confidence score ≥ 0.05 and IoU_box ≤0.5. Notably, the threshold for the confidence score was set at a relative low value with the intention not to miss any spot predictions, especially for the very weak spots. However, this can also lead to many redundant and even wrong predictions. To overcome this issue, we further filtered the predictions by checking whether the prediction has overpassed the trust confidence score or fulfils a combinatory threshold of confidence score and IoU_mask, as can be expressed by

<figure><img src="image_8.png" /><figcaption>Fig. 3. Mask R-CNN framework for spot segmentation from proposed regions of interest (an example shown by the blue bounding box in the left figure). A quantization-free layer (RoIAlign) was implemented to compensate for the misalignment between network inputs and outputs as Mask R-CNN is built on Faster R-CNN, which was not designed for pixel-to-pixel alignment. The figure is adapted from [20].</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>Fig. 4. (a) The total loss and the corresponding five loss contributions for the training dataset and (b) the total losses for the training and validation datasets as a function of iteration number.</figcaption></figure>

$$ \text{Accepted prediction :} \left\{ \begin{array}{ll} \text{confidence score} \geq 0.75 & \\ \text{or} & \\ 0.05 \leq \text{confidence score} < 0.75 \text{ and } \text{IoU\_mask} \leq 0.5 & (4) \end{array} \right. $$

For the experimental images that we focus on in this work, the number of spots is very high (> 1000). This is too many for direct application of the Mask R-CNN model, because too much computation on the inference would be required. This would lead to low prediction accuracy and failure in detecting many spots. To solve this issue, we split the original experimental image into four parts where model prediction was performed for each of them. Subsequently, all the four individual predictions were merged back into one and the prediction filtering (as expressed in Eq. 4) was applied to generate the final output image, which does not influence the result of the instance masks.

## 2.5. LabDCT measurements and grain reconstructions

Experimental LabDCT projection images were acquired for a recrystallized pure iron (99.99 wt% Fe) sample using a Zeiss Xradia 520 Versa X-ray microscope and applied to evaluate the Mask R-CNN model performance. A LabDCT scan in a Laue focusing geometry ($L_{ss} = L_{sd} = 13.0$ mm) was performed (181 projections with an exposure time of 500 s), followed by a conventional absorption tomographic scan (1601 projections with an exposure time of 1.5 s) using a 2032 × 2032 detector with an effective pixel size of 3.36 µm. In both scans, the sample rotation was 360°; an accelerating voltage of 160 kV and a power of 10 W were used. A tomographic volume was reconstructed from the absorption scan to provide a volume mask for subsequent grain reconstruction. Note that both the sample material and the Laue focusing distance are different from the training. This is to test the robustness of the trained model.

The trained Mask R-CNN model was applied to the LabDCT projection images for predicting instance masks, i.e. for spot segmentation. The obtained binary images were directly used as input for grain reconstruction with a voxel size of 2.5 µm applying the commercial software (GrainMapper3D version 3.1) developed by Xnovo Technology ApS. A standard LabDCT spot segmentation routine was also applied using a rolling median background noise correction and then a Laplacian of Gaussian (LoG) based filtering method to obtain binary images for grain reconstruction. Reconstruction parameters were set as standard (the same as in [14]) and held fixed for both reconstructions. Grains were identified from neighboring regions with a maximum misorientation of 0.5°. Thereby, effects of reconstruction parameters are excluded and the

reconstruction is only dependent on the segmentation method (Mask R-CNN or standard routine) for creating the binary images. More details of the standard routine and the GrainMapper3D can be found elsewhere [26,14,27].

## 2.6. Comparison of the grain maps

The two LabDCT reconstructed grain maps, based on the standard routine (referred to as Lab-Routine) and the Mask R-CNN method (referred to as Lab-Mask-RCNN), were compared with a grain map obtained by synchrotron DCT measurements of the same iron sample (referred to as SR-DCT) on the beamline ID11 at the European Synchrotron Radiation Facility (ESRF). Since synchrotron DCT has much better detection limit and spatial resolution than LabDCT [28,29,30,31], the grain map reconstructed from synchrotron DCT is considered as ground truth in the present work, although the indexing of small grains might not be fully correct even in the synchrotron case. More details of the synchrotron DCT measurement for this sample were reported in [32] and details on the synchrotron technique can be found in [28,4,33].

Two aspects of the LabDCT grain maps are compared to SR-DCT: how well the grain indexing is and how accurate the grain boundary position is for the commonly indexed grains. The former was done by pairing grains based on their orientations and positions between the two LabDCT grain maps and the SR-DCT. The latter was assessed by calculating the average grain boundary deviation ($\delta_{GB}$):

$$ \delta_{GB} = \frac{1}{N_{voxel,GB}} \sum_{\epsilon_{GB}} \epsilon_{GB}, \qquad (5) $$

where $\epsilon_{GB}$ is the Euclidean distance between a grain boundary voxel in the SR-DCT dataset and the nearest voxel on the boundary of the paired grain in the LabDCT dataset (either Lab-Routine or Lab-Mask-RCNN); $N_{voxel,GB}$ is the total number of grain boundary voxels in the SR-DCT dataset. More details on the method for the comparison can be found in [14].

# 3. Results

## 3.1. Prediction for real experimental images

Fig. 5 shows an example of the output images obtained by applying the trained Mask R-CNN model to the LabDCT images of the iron sample. Fig. 5a shows all the original spot predictions, which have confidence score $\geq 0.05$ and $IoU\_box \leq 0.5$ fulfilling the prediction settings (see
Table S2 in *Supplementary Materials*). However, it can be seen in Fig. 5a that some predicted masks with low confidence scores are superimposed onto others with higher confidence scores, thereby creating redundant and even wrong predictions. To suppress these ‘over’ predictions, we filtered them out by a combinatory threshold of confidence score and IoU_mask (as expressed in Eq. 4) to obtain a new output image containing qualified predictions (Fig. 5b). These un-qualified predictions in this case are shown in Fig. 5c.

As shown by an example highlighted by the red arrows, a prediction mask with a confidence score of 9% is largely overlapped with another one with a confidence score of 95% (Fig. 5a). Hence, the high-scored mask was considered as a qualified prediction (Fig. 5b) and the low-scored mask was regarded as a bad prediction and thus was discarded (Fig. 5c). This highlights the importance of using non-maximum suppression based on IoU_mask for improving the prediction output.

By applying this filtering to all original predictions from the Mask R-CNN, we show prediction outputs for a full experimental image in Fig. 6. Comparing the left figure to the right, nearly no spots are missing and the shape and size of the spots fit nicely with the real image. Zoom-ins on a corner of the images, where the background noise level is usually much higher than in other areas, further indicates that diffraction spots have been identified accurately in the prediction masks, even for the very weak ones that are very difficult to be recognized by human eyes.

## 3.2. Comparison of the Mask R-CNN spot segmentation with the standard routine

The performance of spot segmentation by the Mask R-CNN model is compared with the standard routine. Fig. 7 shows the comparison for a cropped region. It can be clearly seen from the figure that many more spots are identified by the deep learning method than by the standard routine. In general, the standard routine performs the segmentation well for the relatively bright spots, while for the weak ones either it can only segment a small fraction of them (under-segmentation) or it completely misses them. This implies that the resulting reconstructed grains may not be accurate enough (as the spot shapes are not retrieved accurately)

or may not be able to be indexed (as the spots are totally missed).

This is not the case for the Mask R-CNN. As highlighted by the yellow arrows, many weak and small spots are identified and nicely retrieved. Additionally, the plots of line profiles reveal that a better spot segmentation is obtained by Mask R-CNN than the standard routine, as marked by the red arrow for additional peaks.

## 3.3. Comparison of grain reconstructions

Fig. 8 shows grain maps reconstructed from the synchrotron measurement (SR-DCT) and those from the LabDCT images segmented by the two different methods. Although both Lab-Routine and Lab-Mask-RCNN are largely consistent with SR-DCT, significant differences in details of some grains can be observed by closer examinations on the 3D volumes and the 2D slices: Some grains are present in the slice of SR-DCT but are not in any of the two LabDCT slices (e.g. the ones indicated by the red arrows in Fig. 8a). Except for the small grains #673 and #693 at the sample surface, other missing grains can be found in other sections of both LabDCT datasets, indicating a spatial shift for reconstructions of these grains (note that this is caused by the imaging technique rather than the physical structural change of the sample between different measurements). This also indicates that grains at the sample surface are more likely to be missed in the LabDCT reconstructions, when they are small. Comparing the two LabDCT reconstructions with the SR-DCT, we found that more grains are present in the same slice of Lab-Mask-RCNN than Lab-Routine (see e.g. those marked by the blue arrows in Fig. 8c). Further examination indicates that except for grain #393, other grains highlighted by the blue arrows exist in other sections of Lab-Routine. Interesting to note is that there are more empty indexed space in the 2D slice of Lab-Mask-RCNN (shown in black) than of Lab-Routine. These empty space generally correspond to small grains neighboring with large grains in SR-DCT. The successful reconstruction of these small grains depends on the grain reconstruction algorithm and the segmented spots. The small grains can be “eaten” by large grains if the diffraction spots of the large grains are over-segmented, resulting in a higher completeness value for the reconstruction of large grains. Conversely, the small grains

<figure><img src="image_10.png" /><figcaption>**Fig. 5.** Output images for (a) all original spot predictions, (b) qualified predictions filtered by combinatory threshold of confidence score and IoU_mask using Eq. 4 and (c) un-qualified predictions that are discarded in the final output. White boxes indicate the zoom-in areas. Red arrows highlight the removal of one prediction by the filtering as expressed in Eq. 4. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>**Fig. 6.** Output for (a) a real experimental image by (b) making a prediction for each of the four split parts (top-left, top-right, bottom-left and bottom-right, respectively) before (c) finally merged to generate a full-size binary mask. Bottom-row shows the zoom-ins indicated by the red rectangles. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**Fig. 7.** Comparison of spot segmentation using the two different methods illustrated for a cropped region. (a) An experimental image and its outputs using (b) the standard routine based on a Laplacian of Gaussian filtering and (c) the Mask R-CNN model. For (b) and (c), binary images are shown on the top row and line profiles along the yellow diagonals are shown in the bottom row together with the experimental line profile. Arrows show the identified spots by the Mask R-CNN, which are badly segmented or totally missed by the standard routine. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>**Fig. 8.** 3D view (top line) and 2D slices (bottom line) of reconstructed grain structures. (a) synchrotron DCT and LabDCT reconstructions based on binary images obtained by different methods: (b) standard routine based on Laplacian of Gaussian filtering and (c) Mask R-CNN. Grains are colored by IPF-Z (Z direction in inverse pole figure). Grains of interest are marked by arrows in the slices (some shown with grain IDs).</figcaption></figure>

can remain un-reconstructed if their completeness values cannot surpass the minimum acceptable completeness defined by the reconstruction as well as the spot segmentation is rather accurate for the large grains (thus no chance to “eat” the empty space).

It is found that the majority of the empty space in the LabDCT-Mask-RCNN was generated due to a correct stop for the over-growth of large grains. This suggests that the region growth routine, implemented by the forward modelling based reconstruction algorithm, does not overgrow the reconstructed grains into space belonging to other grains (even they are not indexed), suggesting the model does not over-segment the spots.

Using the same method for comparing grain indexing as reported previously [15,14], all grains in the datasets can be classified into four categories: one-to-one indexed, one-to-multi indexed (one grain in SR-DCT is indexed by more than one grain in the LabDCT dataset), false-negatively indexed (a grain that is not indexed in LabDCT dataset but actually exists in SR-DCT) and false-positively indexed (a grain that is indexed in LabDCT dataset but actually does not exist in SR-DCT). The first two categories are true-positively indexed. The latter two categories can be further classified according to the locations of the grains: at sample surface or in sample interior. A summary is given in Table 2. It is found that 22 more grains are correctly indexed (mostly grain size <40 µm, including 1 one-to-multi indexed grain), resulting in 22 fewer false-negatively indexed grains (8 in the sample interior and 14 at the sample surface) in Lab-Mask-RCNN than Lab-Routine. Notably, 13 more false-positively indexed grains are found in Lab-Mask-RCNN. This might

because the mask RCNN model wrongly identified some noise as diffraction spots. However, we cannot rule out the possibility that these grains might truly exist and SR-DCT failed to index them, because they are reconstructed as very small (with average diameter of 9.7 ± 4.5 µm) whilst with considerably high completeness values (0.41 ± 0.04). The likelihood may be low though.

Nevertheless, to assess the performance of the grain indexing we consider that the false-positively indexed grains do not truly exist in the sample and the SR-DCT data is the ground truth. We calculated precision (P = TP / (TP + FP)) and sensitivity (S = TP / (TP + FN)), where TP is true-positives, FP is false-positives and FN is false-negatives. Hence, the F₁ score is calculated as F₁ = 2 * P * S/(P + S), a higher score thus corresponds to more accurate indexing. Based on the results in Table 2, we derived F₁ = 0.815 for the Lab-Routine and F₁ = 0.830 for the Lab-Mask-RCNN. It is noteworthy that the current Lab-Routine obtained by the reconstruction with a more recent version of GrainMapper3D software (version 3.1) has better performance than an older reconstruction with version 2.3 (F₁ = 0.802 as reported in [14] for the present iron sample), using the same reconstruction settings. A recent summary of the advances on GrainMapper3D software is reported in [34].

There are 397 commonly indexed grains between the two LabDCT datasets and SR-DCT. For these grains, we can assess the performance of grain shape reconstructions by comparing the deviation of each grain boundary voxel (ε<sub>GB</sub>) and average grain boundary deviation (δ<sub>GB</sub>, see Eq. 5) for the individual grains. Fig. 9a shows the distributions of ε<sub>GB</sub> for all

<caption>**Table 2**</caption>

Comparison of number of indexed grains (N) between the SR-DCT dataset and the two LabDCT datasets (Lab-Routine and Lab-Mask-RCNN). Grain size, <D>, is expressed as mean value and standard deviation. For one-to-multi indexed grains, the number of grains in SR-DCT is given with the paired number of grains in the LabDCT datasets given in brackets.

<table><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">&lt; D &gt;, µm</th><th colspan="3">N</th><th colspan="2">False-negatively indexed</th><th colspan="2">False-positively indexed</th></tr><tr><th>Total indexed</th><th>One-to-one indexed</th><th>One-to-multi indexed</th><th>At sample surface</th><th>In sample interior</th><th>At sample surface</th><th>In sample interior</th></tr></thead><tbody><tr><td>SR-DCT</td><td>39.6 ±<br>22.4</td><td>596</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Lab-routine</td><td>42.7 ±<br>26.9</td><td>423</td><td>410</td><td>4 (9)</td><td>148</td><td>34</td><td>4</td><td>0</td></tr><tr><td>Lab-mask-<br>RCNN</td><td>43.0 ±<br>25.2</td><td>458</td><td>431</td><td>5 (10)</td><td>134</td><td>26</td><td>14</td><td>3</td></tr></tbody></table>
the grain boundary voxels. It can be seen that more voxels have small $\epsilon_{GB}$ in Lab-Mask-RCNN than Lab-Routine: 86.7% of all the grain boundary voxels have spatial deviation no greater than 4 pixels in Lab-Mask-RCNN, whilst it is 80.2% in Lab-Routine. For each grain we plot the ratios of the grain boundary deviations ($\delta_{GB}$ (Lab-Routine) / $\delta_{GB}$ (Lab-Mask-RCNN)) in Fig. 9b to quantify how much shape reconstruction has been improved by the Mask R-CNN model. The figure shows that the ratio is >1 for 82% of the 397 grains, i.e. the spatial accuracy has been improved for most of the grains by Mask R-CNN. The average $\delta_{GB}$ ratio is 1.33 ± 0.48. The grain shape is thus improved by about 30% in Lab-Mask-RCNN, and this improvement is more pronounced for grains <50 µm.

An example of direct comparison is illustrated for grain #376 ($\delta_{GB}$ ratio = 3.38). This grain is located in sample interior and has a grain size in the range where the mask R-CNN model show significant improvement (see Fig. 9b). It can be clearly seen in Fig. 9c and e that the grain shape in Lab-Mask-RCNN (Fig. 9d) is much closer to that in SR-DCT (Fig. 9c) compared to its counterpart in Lab-Routine (Fig. 9e). Comparisons of the reconstructed grain sizes and $\delta_{GB}$ further confirm this observation.

# 4. Discussion

## 4.1. Advantages of the Mask R-CNN method

Mask R-CNN has pronounced advantages in finding more weak spots and segmenting individual spot shapes more accurately compared to the standard routine (see Figs. 6 and 7), and the resulting grain reconstruction has shown improvements, demonstrating the merit of using the model for improving LabDCT grain mapping. More grains were indexed because more (weak) spots were identified and grain shapes were better reconstructed because spot shapes were segmented better (see Table 2 and Fig. 9). This was true in particular for the smaller grains, suggesting that Mask R-CNN is a very promising approach to improve the detection limit and spatial resolution for the LabDCT technique in addition to other approaches with a better source, detector system or geometry.

Mask R-CNN is also advantageous to use for the spot segmentation compared to the standard routine. Once the model is trained, it can be directly applied to experimental images, even for somewhat different samples and experimental set-ups. For the standard routine, on the other hand, a set of parameters needs to be tuned for each LabDCT dataset and rely on experience in order to generate high quality spot segmentation.

Compared to our previously proposed deep learning method used for

<figure><img src="image_14.png" /><figcaption>**Fig. 9.** Comparison of reconstructed grain boundary accuracies. (a) Histogram of spatial deviations for all the boundary voxels (~9.76 × 10⁵) in the 397 commonly indexed grains between the two LabDCT datasets and SR-DCT. (b) Ratio of grain boundary deviation ($\delta_{GB}$ (Lab-Routine) / $\delta_{GB}$ (Lab-Mask-RCNN)) for individual grains as a function of grain size; a ratio larger than 1 indicates a better grain shape reconstruction in Lab-Mask-RCNN than Lab-Routine; otherwise, it is worse. (c-e) 3D voxelized visualizations of grain #376 (highlighted in (b)) in SR-DCT, Lab-Mask-RCNN and Lab-Routine, respectively. In (d-e) grain boundary deviation maps of $\epsilon_{GB}$ (see definition in Section 2.6) are shown in the bottom row together with the colour bar in unit of pixel (here 2.5 µm/pixel). This grain is determined to be 37.7 µm, 32.3 µm and 23.8 µm, respectively, in (c-e). The boundary deviation for this grain is 1.4 ± 1.1 pixels in Lab-Mask-RCNN and 4.8 ± 3.7 pixels in Lab-Routine.</figcaption></figure>
removing background noise (referred to as DL-BG, [15]), the mask-RCNN model has two advantages. First, it outputs instance masks directly and does not require a second step for binarization as the DL-BG does. Second, the Mask R-CNN model does not require a resizing of the input image for making prediction output, whereas DL-BG usually requires a down sampling of the input image for making predictions using the same computing resources. This suggests that the Mask R-CNN does not compromise the image resolution to make predictions, and it partly explains the observation that more improvement in the grain boundary accuracy is obtained in the present work using mask-RCNN (~30%) than our previous work using DL-BG (~17%).

It is worthwhile to point out that the performance of the standard
routine has also been improved significantly with the newest version of
GrainMapper3D (version number increased from 2.3 to 3.1). As a result,
we experienced that DL-BG does not have dramatic improvement in
grain reconstruction compared to the standard routine using the same
version 3.1. Nevertheless, the improvement is still significant, further
highlighting the benefits of Mask R-CNN.

## 4.2. Further improvement of Mask R-CNN for spot segmentation

The Mask R-CNN model requires a lot of computation for the con-
volutions, especially if the images have a high resolution, as the LabDCT
images presented here have (e.g. see Fig. 6). We have solved this issue by
splitting the images into four parts, which makes it possible to output a
full-size prediction image with a "standard GPU". Thereby, no pixels
were lost and the image resolution was not compromised. This splitting
treatment is, however, not optimal in terms of the computational speed
for inference. In the future, more computational power combined with a
more efficient neural network will solve this issue. Promising results are
starting to occur using new and simpler one-stage models for instance
segmentation (e.g. SPRNet by [35]; SG-Net by [36]). These models
typically add new branches to the output prediction masks from e.g.
single pixels in the convolution feature maps directly, or sub-regions of
each detected instance dynamically, showing better performance than
previous one-stage models (which were designed for performing fast
inference object detection), such as Single Shot Detection (SSD, [37]),
YOLO [38], YOLO9000 [39], YOLOv3 [40], RetinaNet [41] and Cen-
terNet [42]. The prediction accuracy of these new models has been
demonstrated to be comparable with Mask R-CNN. In addition, these
new models avoid the use of expensive regional proposal network for
RoI features, making them simpler to train (fewer parameters to set) and
having a higher inference speed.

Proper data labelling is the key to obtain a good model, which can
make accurate predictions. In the present work, we proposed a new
method to segment and split overlapping spots (see Fig. 1e). However,
still few (<1% of the overlapped spots) are only partially split. This
subtle imperfection of data labelling may falsely train the Mask R-CNN
and might lead to less accurate predictions for overlapped spots in the
real experimental images. Manual interference, splitting these very
challenging overlapped spots, may be a solution. However, in general, a
small fraction of imperfections in treating the spot overlap does not
cause big problems for the final grain mapping result, because the grain
reconstruction algorithm is based on forward simulation, being tolerable
to a certain degree of overlapped instance masks.

## 5. Conclusions

We have applied Mask R-CNN for instance segmentation of LabDCT
images with the state of the art library (open source software package
*Detectron2*) implemented in the python deep learning framework
PyTorch. We used synthetic data for training the model to perform
segmentation of diffraction spots from the LabDCT images. This is the
first time that deep learning instance segmentation is used for this image
type and the results are superior compared to the standard way of seg-
mentation as well as to our previous deep learning approach, especially

in terms of finding small spots and individually segmenting the spot
shapes accurately.

The Mask R-CNN model enables a better grain mapping compared to the standard routine. As demonstrated for a recrystallized iron sample with many spots in every LabDCT projection image, 22 additional grains were correctly indexed (out of a total of 458 grains) and the spatial resolution, characterized by the grain boundary deviation, was improved by 30% on average. Both these improvements are more pronounced for smaller grains (< 50 µm), making the application with Mask R-CNN a promising approach for further improving the detection limit and spatial resolution of the LabDCT technique.

## Funding information

This work is funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (M4D - grant agreement No. 788567).

## Declaration of Competing Interest

The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.

## Data availability

The raw data required to reproduce these findings are available upon
request.

## Acknowledgement

The authors thank Xnovo Technology for valuable discussions regarding the use of grain reconstruction software and preparation of the manuscript.

# Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.matchar.2023.112983.

## References

[1] Y. Hayashi, Y. Hirose, Y. Seno, J. Appl. Crystallogr. 48 (2015) 1094-1101.

[2] H.F. Poulsen, Three-Dimensional X-Ray Diffraction Microscopy: Mapping Polycrystals and their Dynamics, Springer, Berlin, 2004.

[3] R.M. Suter, D. Hennessy, C. Xiao, U. Lienert, Rev. Sci. Instrum. 77 (2006), 123905.

[4] W. Ludwig, S. Schmidt, E.M. Lauridsen, H.F. Poulsen, J. Appl. Crystallogr. 41 (2008) 302-309.

[5] B.C. Larson, W. Yang, G.E. Ice, J.D. Budai, J.Z. Tischler, Nature 415 (2002) 887-890.

[6] H. Simons, A. King, W. Ludwig, C. Detlefs, W. Pantleon, S. Schmidt, F. Stöhr, I. Snigireva, A. Snigirev, H.F. Poulsen, Nat. Comm. 6 (2015) 6098.

[7] H.F. Poulsen, Curr. Opin. Solid State Mater. Sci. 100820 (2020).

[8] A. King, P. Reischig, J. Adrien, W. Ludwig, J. Appl. Crystallogr. 46 (2013) 1734-1740.

[9] A. King, P. Reischig, J. Adrien, S. Peetermans, W. Ludwig, Mater. Charact. 97 (2014) 1-10.

[10] S.A. McDonald, P. Reischig, C. Holzner, E.M. Lauridsen, P.J. Withers, A.P. Merkle, M. Feser, Sci. Rep. 5 (2015) 14665.

[11] X.C. Lei, Y.B. Zhang, J. Sun, F. Bachmann, X.F. Yang, R.E. Sanders, D. Juul Jensen, Mater. Res. Lett. 9 (2021) 65-70.

[12] S.A. McDonald, C. Holzner, E.M. Lauridsen, P. Reischig, A. Merkle, P.J. Withers, Sci. Rep. 7 (2017) 5251.

[13] J. Sun, C. Holzner, H. Bale, M. Tomita, N. Gueninchault, F. Bachmann, E. M. Lauridsen, T. Inaguma, M. Kimura, ISIJ Int. 60 (2020) 528-533.

[14] H. Fang, E. Hovard, Y. Zhang, L.K.H. Clemmensen, B. Kjær Ersbøll, D. Juul Jensen, IUCrJ 8 (2021) 719-731.

[15] H. Fang, D. Juul Jensen, Y. Zhang, IUCrJ 8 (2021) 559-773.

[16] E. Hovad, H. Fang, Y. Zhang, L.K. Harder Clemmensen, B. Kjær Ersbøll, D. Juul Jensen, Integrat. Mater. Manuf. Innov. 9 (2020) 315-321.

[17] S. Ren, K. He, R. Girshick, J. Sun, Faster R-CNN: towards real-time object detection with region proposal networks, Adv. Neural Inf. Proces. Syst. 28 (2015).
[18] R. Girshick, Fast R-CNN, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1440–1448.

[19] R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580–587.

[20] K. He, G. Gkioxari, P. Dollar, R. Girshick, Mask R-CNN, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2961–2969.

[21] H. Fang, D. Juul Jensen, Y. Zhang, Acta Cryst. A 76 (2020) 652–663.

[22] A. Lindkvist, H. Fang, D. Juul Jensen, Y. Zhang, J. Appl. Crystallogr. 54 (2021) 99–110.

[23] M. Katzer, F. Kummert, G. Sagerer, IEEE Trans. Nanobiosci. 2 (2003) 202–214.

[24] D. Legland, I. Arganda-Carreras, P. Andrey, Bioinformatics 32 (2016) 3532–3534.

[25] T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, C. Lawrence Zitnick, Microsoft COCO: common objects in context, in: European Conference on Computer Vision, 2014, pp. 740–755.

[26] F. Bachmann, H. Bale, N. Gueninchault, C. Holzner, E.M. Lauridsen, J. Appl. Crystallogr. 52 (2019) 643–651.

[27] J. Oddershede, J. Sun, N. Gueninchault, F. Bachmann, H. Bale, C. Holzner, E. M. Lauridsen, Integ. Mater. Manu. Inno. 8 (2019) 217–225.

[28] G. Johnson, A. King, M.G. Honnicke, J. Marrow, W. Ludwig, J. Appl. Crystallogr. 41 (2008) 310–318.

[29] P. Reischig, A. King, L. Nervo, N. Viganó, Y. Guilhem, W.J. Palenstijn, K. J. Batenburg, M. Preuss, L. Ludwig, J. Appl. Crystallogr. 46 (2013) 297–311.

[30] L. Renversade, R. Quey, W. Ludwig, D. Menasche, S. Maddali, R.M. Suter, A. Borbely, IUCrJ 3 (2015) 32–42.

[31] M. Syha, A. Trenkle, B. Lödermann, A. Graff, W. Ludwig, D. Weygand, P. Gumbsch, J. Appl. Crystallogr. 46 (2013) 1145–1150.

[32] J. Zhang, Y. Zhang, W. Ludwig, D. Rowenhorst, P.W. Voorhees, H.P. Poulsen, Acta Mater. 156 (2018) 76–85.

[33] W. Ludwig, P. Reischig, A. King, M. Herbig, E.M. Lauridsen, G. Johnson, T. J. Marrow, J.Y. Buffire, Rev. Sci. Instrum. 80 (2009), 033905.

[34] J. Sun, F. Bachmann, J. Oddershede, E. Lauridsen, IOP Conf. Series: Mater. Sci. Eng. 1249 (2022), 012045.

[35] J. Yu, J. Yao, J. Zhang, Z. Yu, D. Tao, SPRNet: single-pixel reconstruction for one-stage instance segmentation, IEEE Trans. Cybernetics 51 (2020) 1731–1742.

[36] D. Liu, Y. Cui, W. Tan, Y. Chen, SG-Net: spatial granularity network for one-stage video instance segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9816–9825.

[37] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.Y. Fu, A.C. Berg, Ssd: Single shot multibox detector, in: European Conference on Computer Vision, 2016, pp. 21–37.

[38] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: unified, real-time object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 779–788.

[39] J. Redmon, A. Farhadi, YOLO9000: better, faster, stronger, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 7263–7271.

[40] J. Redmon, A. Farhadi, Yolov3: an incremental improvement. arXiv:1804.02767, 2018.

[41] T.Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollar, Focal loss for dense object detection, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2980–2988.

[42] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, Q. Tian, CenterNet: keypoint triplets for object detection, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6569–6578.