# Instance Segmentation Method of User Interface Component of Games
## 3 Methods
### Network  
With the paired synthetic data generated above, when receiving the general game screen as an input value, we infer the UI area from it and segment this part. We used the Pix2 Pix framework that showed quality performance in paired data learning as the baseline network. In the network, when the general game screen is inputted as a network input value, the generator produces a segmented image of each UI area as an output value. In Pix2 Pix, the generator is a convolutional network with U-net architecture. Since Pix2 Pix has the same input and output resolution, and has the characteristic of maintaining the detail and shape to a degree, information loss occurs when the encoder-decoder structure is used. This can be reduced by using skip-connection. Here, the game screen, which is an input image, and the segmented UI image, are paired up with each other and exist in the form of a 512 \(\times\) 256 size image. After the generator, the synthetic image is concatenated with the target color image. Therefore, the number of color channels will be six (_height_\(\times\)_width_\(\times\)_6_). This is fed as input to the discriminator network. In Pix2 Pix, patchGAN is used as a discriminator. The patchGAN network takes the concatenated input images and produces an output that \(N\times N\).  
Figure 4: Automatically generated experimental data.  
Figure 5: Images of cases where auto labeling does not work well.  
The objective function of the Pix2 Pix GAN is as follows:  
\[V(G,D)=\underset{G}{min}\underset{D}{max}\mathbb{E}_{y}[log(D(y))]+\mathbb{E}_{x} [log(1-D(G(x))]+\mathbb{E}_{xy}[\parallel y-G(x)\parallel_{1}] \tag{2}\]  
Here, y refers to the real image of the target domain and x refers to the real image of the source domain. The first and second terms are called the adversarial loss, and it is characterized by using the loss employed the GAN.  
\[\mathbb{E}_{y}[log(D(y))]+\mathbb{E}_{x}[log(1-D(G(x))] \tag{3}\]  
In other words, that the real y is real, and G(x), the y mapped from x, is fake, is the Pix2 Pix discriminator loss. On the contrary, that G(x) is real is the generator loss. The last term is the reconstruction Loss, which is the loss employed in the conventional CNN-based learning method.  
\[\mathbb{E}_{xy}[\parallel y-G(x)\parallel_{1}] \tag{4}\]  
The difference between the real y and G(x) made from x is minimized. Figure 6 shows the Pix2 Pix GAN structure we used. Figure 7 shows the generator and discriminator network structure. Figure 8 shows the results of our network performance experiment. We experimentally have the best visual results when we removed the last encoder and first decoder of the original Pix2 Pix generator. The generator's optimization objective function includes an L1 loss, which has a direction to minimize the Euclidean distance between the original image and the generated image. Therefore, there is a tendency to focus on the average value of the image, that is, low frequency. Since the L1 loss is in charge of the low frequency region, the discriminator focuses on high frequency to determine whether it is real or fake. The discriminator's patch size should be determined in consideration of the overall image size and the relationship between pixels. In this experiment, when a size of 70 \(\times\) 70 was used, a clear image could be obtained.  
Figure 6: Proposed GAN architecture.