# Instance Segmentation Method of User Interface Component of Games
## 4 Experimental Results
The network seems to learn stably although the number of the UI components to be detected is larger than that of Kingdom Rush. Unlike Kingdom Rush, the UI components of Blade and Soul are quite small and have high image resolution. Therefore, if the corresponding UI component has a similar color and shape to the background image, it may be difficult for the network to detect it. However, our network demonstrates its capability to segment the UI components. Figure 11 shows the results of applying our system to another PC MMORPG game, Black Desert. Black Desert is a skill-based PC MMORPG game. As in Blade and Soul, quite complex types of information are displayed on the screen, and players interact by learning them. Unlike Blade and Soul, Black Desert is a skill-based game, thus a large number of skills are exposed on the UI. In this experiment, we attempted to confirm whether the system could automatically detect these skills. As a skill-based game, a number of skill shortcuts can be placed around the characters in the center, thus the UI positional characteristics are not apparent. The system automatically generated synthetic data segmented into 37 groups and trained the network through this. When applying the data trained in this way to the original Black Desert, we could accurately detect the corresponding UI area with a pixel accuracy of 84%. In this experiment, the number of the UI components to be detected was more than that of Blade and Soul, which led to smaller UI components. The network seems to learn stably, though. The experimental results show that the performance of our system decreases with an increase in the number of UIs for segmentation, and at the image level of 256 \(\times\) 256, the relatively stable UI segmentation is expected to be achieved up to approximately 40. Figure 10: Result of Blade and Soul UI segmentation. Table 2 shows the overall segmentation performance of our network. First, we reduced the size of UI screens of the three games to 256 \(\times\) 256, then checked the number of segmented UI components, the size of the most basic UI button, and the approximate spacing between each UI component and defined approximate complexity from this. We measured pixel accuracy, mean accuracy, mean IU, and frequency weighted IU values for Kingdom Rush, Blade and Soul, and Black Desert. Mean accuracy values for them were 0.820, 0.797, and 0.777, respectively. It is confirmed that, overall, pixel accuracy is reduced, being inversely proportional to the number of UI component classes we intend to classify. Since Kingdom Rush, a mobile game, is based on a touch-based UI, our system could automatically extract all UI components. However, as for Blade and Soul and Black Desert, MMORPG games, the number of the UI components detected by the system is approximately 50% below of the actual number of UI icons. This is because the UI components are subdivided into skill units. Therefore, our system shows that UI segmentation is more effectively achieved relatively in mobile games. Figure 12 shows the loss graph during training. Red indicates Kingdom Rush, pale green Blade and Soul (BnS), and dark green Black Desert. Only Black Desert is shown to have relative difficulty in learning. Overall, learning was stabilized at around 200 epochs. With an increase in the number of the UI classes to be classified, the loss of the generator drops. \begin{table}
\begin{tabular}{l l l l} \hline \hline  & **Kingdom Rush** & **Blade and Soul** & **Black Desert** \\ \hline Number of Segmented UI Components & 7 & 30 & 37 \\ Size of a Single Button (Approx.Pixel) & Large (25 \(\times\) 25) & Small (5 \(\times\) 5) & Small (5 \(\times\) 5) \\ Spacing between UI Components (Approx.Pixel) & Long (5 pixel) & Short (1–2 pixel) & Short (1–2 pixel) \\ Overall Complexity & Low & High & High \\ \hline Pixel Accuracy & 0.882 & 0.851 & 0.842 \\ Mean Accuracy & 0.820 & 0.797 & 0.777 \\ Mean IU & 0.707 & 0.665 & 0.671 \\ Frequency Weighted IU & 0.793 & 0.734 & 0.791 \\ \hline \hline \end{tabular}
\end{table}