## 5 Annotation of Color Barcode Patterns  
In CCB-OCC, color barcodes appear in various patterns in the image received from the same transmitted signal due to synchronization mismatch and alignment problems between the transmitting and receiving devices. A previous study [20] used a histogram analysis approach for a limited number of patterns that could appear in the captured images to identify the colors in the barcodes. The RGB values of color barcodes could be extracted by finding the maximum and minimum values of the histogram for each channel, and this was used to determine pilot symbols and data symbols. This method has limitations in that it is difficult to accurately detect RGB values for various histogram patterns, and a heuristic approach was required to extract RGB values in the worst cases. In particular, the histogram distribution of the RGB color values constituting the color barcode change significantly when the capturing environment changes, resulting in inaccurate color value extraction. Unlike the CCB-OCC scheme relying on heuristic histogram analysis [20], we propose a method that can improve the reliability of symbol detection performance by using a CNN model (a well-known technique showing robust performance in object classification fields).  
To train the CNN model, annotation of color barcodes is required. The color barcode classes in the proposed DeepCCB-OCC consist of three pilot symbols (R, G, B) and eight data symbols (0-7) in the color constellation, a total of 11 classes. In the color barcode extracted at the receiving end, the primary and complementary colors encoded with the data do not appear exactly as the color barcode is halved, but appear in various shift patterns due to the asynchronous problem between the sending and receiving devices, so additional labeling is required. Labeling of the color barcode pattern for each symbol class should be performed before training the CNN model.  
In this paper, as shown in Figure 6, four shift patterns of received color barcodes are defined. The first pattern is an ideal type of color barcode in which the pattern of primary and complementary colors is halved. Here, the shift pattern of the received color barcode is demonstrated with pilot symbol G. The first pattern, which is an ideal shape, is represented by GG' in Figure 6a. In the second pattern shown in Figure 6b, the primary color is located in the center of the color barcode, while the complementary color of the previous color barcode is displayed on the left side, and the complementary color of the primary color is on the right side. As a result, it appears as R'GG'. In the third pattern, complementary colors of the previous color barcode are captured on the left, and the color barcode of the primary color is displayed on the right: R'G is shown in Figure 6c. The fourth pattern shows the color barcode in the form GG'B, as seen in Figure 6d, so the color barcode for a specific color signal has a total of four patterns. Each pattern consists of three pilot symbols and eight data symbols, so data can be received from a total of 44 classes.  
Figure 6: Four different patterns in color barcodes: (**a**) GG’, (**b**) R’GG’, (**c**) R’G, (**d**) GG’B.  
As mentioned above, continuous color patterns can be divided into four types, but the color barcode pattern may change depending on the synchronization timing between the transmitting end and the receiving end. In such a situation, the approach to classify 44 classes using a single deep learning model limits color pattern classification accuracy, and ultimately may adversely affect communication performance from the DeepCCB-OCC system. Therefore, to maximize classification performance while maintaining the generalization performance of the DeepCCB-OCC system for various color patterns, CNN models for pilot symbol classification and data symbol classification were independently developed.