## 6 CNN-Based Synchronization and Color Pattern Classification
```
1:Input: Captured Video with color barcodes
2:\(\textit{Synchronization}\leftarrow\textit{False}\)
3:while Captured Video do
4:if Color barcode is detected then
5:\(\textit{Area}\leftarrow\text{Color barcode area detected by YOLO}\)
6:\(\textit{Mask}\leftarrow\text{Differential current Area}\) with previous Area
7:\(\textit{Mask}\leftarrow\text{Morphological operation for Mask}\)
8:\(\textit{Barcode}\leftarrow\text{Extract Barcode using Mask}\)
9:if Synchronization then
10:\(\textit{Data}\leftarrow\text{Barcode classification using CNN model}\)
11:\(\textit{Result}.\text{append}(\textit{Data})\)
12:else
13:if Pilot symbols R, G, B are detected consecutively then
14:\(\textit{Synchronization}\leftarrow\textit{True}\)
15:endif
16:endif
17:endwhile Return Result

Figure 8: Synthetic data using data augmentation techniques such as enlargement/reduction, rotation, position transformation, and vertical inversion. (**a**) Original data, (**b**–**d**) Synthetic data. Figure 9: The CNN model’s structure for packet synchronization and data decoding. ## 7 Experimental Results

To evaluate the performance of the DeepCCB-OCC system proposed in this study, communication experiments with a commercial display and a smartphone were performed in an indoor space. The experimental environment simulating an actual environment is shown in Figure 10. When capturing a display using a camera, the formation of the color barcode image is affected by various yaw angles of the camera. In fact, the shape of the collected color barcode is deformed under the influence of distance changes and rotation changes according to the position of the handheld camera and the user's position, in addition to the yaw angle between the display and the camera. In this study, the deformation of color barcode images caused by influences other than the yaw angle is solved by using a synthesized color barcode image generated using the data augmentation technique. The display device serving as the transmitter had a resolution of 1920 \(\times\) 1080 and a refresh rate of 60 Hz. Among the receiving devices for DeepCCB-OCC, the most commonly used is a smartphone, and in this experiment, a Samsung Galaxy S9 was used to record video at 30 fps. In the experiment, the same space and illumination were considered as in the experiment performed in the previous paper. In the previous paper, the video encoded with 3-bit data was transmitted through the display and captured by the Samsung Galaxy S9 camera, which is the same camera used in the experiment of this manuscript. In addition, the distance and angle range were set to be the same. To collect color barcode data based on the yaw angle during capture, the angle between the camera and the display remained steady for five minutes and changed between \(-\)20 and +20 degrees at intervals of 10 degrees. For data to train the YOLO-based color barcode extraction model, 600 frames were extracted from the images taken at each angle, and thus, a total of 3000 color barcodes were collected. Here, 2500 images were used for training, and 500 images were used for validation. Note that we have a total of 6000 images acquired for pilot symbols and 16,000 images for data symbols. To implement the CNN model for pilot symbol detection, 4800 images were used as training data, and 1200 images of data were used as test data. In the CNN model for decoding data, 12,800 images were used as training data and 3200 images were used as test data. The CNN model structure and the experimental setup are described in Tables 1 and 2. Figure 10: The experimental environment with a commercial display and a mobile phone. Learning curves of the CNN models for pilot and data symbol detection are shown in Figure 11. The purpose behind checking the learning curves is to find the final model using the weights from the epoch with the best loss score. The initial epoch value for training the two CNN models is set to 1000.