# High Performing Facial Skin Problem Diagnosis with Enhanced Mask R-CNN and Super Resolution GAN
## 4 Design of Tactics for Remedying the Technical Challenges
### Design of the Main Control Flow
The right-side of the figure shows the JSON representation of the annotations, which is required to train the model. By using our collection of 2225 face photos with annotations on facial skin problem instances, a total of 31 Mask R-CNN models were trained. The data collection is utilized as a training set, validation set, and test set, as shown in Table 4.

The training set consists of 1557 photos that are 70% of the data collection. The validation set consists of 223 photos that are 10% of the data collection. The test set consists of 445 photos that are 30% of the data collection. Hyperparameters for training models are set to 0.001 as the learning rate, 1000 as epochs, and 5 as batch size. The early stopping is applied to prohibit overfitting the model to the training set. ### Proof-of-Concept Implementation

A web-based system of facial skin diagnosis has been implemented in Python using the following libraries: TensorFlow for developing CNN models, NumPy for processing operations for mask data, OpenCV for processing facial photos, MySQL for managing database, and Django framework for building the web site. Figure 12 shows the web user interface of this system. The original image, masks generated around the facial skin problems, and overlay of the mask on the original image are shown on the left-side of the figure. The right-side of the figure shows the results of identifying facial skin problem instances. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Subset Type** & **Training Set** & **Validation Set** & **Test Set** & **Total** \\ \hline
**Ratio (\%)** & 69.98 & 10.02 & 20.0 & 100 \\
**\# of Photos** & 1557 & 223 & 445 & 2225 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Distribution of Data Collection. Figure 11: Face photo with annotations and its JSON representation. ### Performance Metric for Facial Skin Problem Diagnosis

Dice Similarity Coefficient (DSC) is an appropriate measure in evaluating the performance of segmenting the models. DSC is to measure the ratio of matching the segmented areas on the labeled areas on images. Its metric is given below. \[DSC=\frac{2*|LB\cap SEG|}{|LB|+|SEG|}\]

Let \(LB\) be a mask for the label data of a photo, and Let \(SEG\) be a mask of the segmentation results from the photo. Let \(|X|\) be the area of masked instances in an input mask \(X\). \(DSC\) is measured by the sum of masked areas for \(LB\) and \(SEG\) over the area containing the intersection area in \(LB\) and \(SEG\). The range of \(DSC\) is between 0 and 1. The more accurate the segmented results, a \(DSC\) value close to 1 is returned. ### Experiment Scenarios and Results

A set of experiments were conducted to evaluate the effectiveness of the 5 tactics, an experiment to evaluate the integration of all the tactics, and an experiment to compare our approach to other known approaches. #### 5.4.1 Experiment for Tactic #1: Refined Mask R-CNN Segmentation Models

This experiment is to evaluate the effectiveness of the Refined Mask R-CNN model. This is done by training two segmentation models: a model with Mask R-CNN structure and a model with the refined Mask R-CNN structure with fusion and deconvolution layers. The dataset of 410 face photos was used and each photo contains one or more facial skin problems of acne, age spots, and moles. These facial skin problem types are quite small in size, and the bounding box of the largest problem instance in the dataset is sized to (12 \(\times\) 12) pixels. The performances of the two models are compared in Figure 13.

Figure 12: User Interface of Facial Skin Problem Diagnosis System. In this experiment, a segmentation model of conventional Mask R-CNN was applied to detect facial skin problems on all the face photos, the performances were measured with DSC, and computed the average of all DSC measurements. Then, a segmentation model trained with the enhanced Mask R-CNN was trained and applied to measure its performance in DSC.