# A Distributed Automatic Video Annotation Platform
## 4 Spatiotemporal Based Video Annotation
### Color Local Directional Ternary Pattern (CLDTP)  
LBP [40] captures texture features from the gray image; however, it ignores the color as well gradient information which may decrease the performance. According to [PERSON] [46] better results can be achieved if we can keep the three prominent features, i.e., color, shape, and texture information. Therefore, in [41], the authors extended the LDP [47] to LDTP (local directional ternary pattern), which captures the shape information by applying the gradient operation; however, they did not consider the color information. The author in [48] used color information with LBP; the LBP operation was performed on each color channel. Afterward, the obtained features from each channel were fused to create a single feature vector that provided better accuracy. Hence, in this work, we introduce the color local directional ternary pattern (CLDTP), which includes texture, shape, and color information. The block diagram of our proposed CLDTP is presented in Figure 3 where each frame is divided into five parts--center, bottom, upper, left, and right--to capture the local and global information from a frame. Afterward, we derive the color information from each frame by dividing the RGB image into three color channels--red, green, and blue. Then each channel is fed to the Equations (1) and (2) to calculate the local directional ternary pattern (LDTP), upper and lower respectively. We use Equation (3) to calculate the mean (\(\mu\)) to remove noise from given 3 \(\times\) 3 square patches. Then Equations (4) and (5) are used to calculate the gradient magnitude for the center pixel \(\alpha\) and neighboring pixel \(\beta\) respectively which contains the shape information of an image. Here, \(I_{C}\) and \(I_{P}\) are the center and neighbor pixels respectively.  
Finally, the ternary pattern calculation of LDTP upper and LDTP lower are performed using Equations (1) and (2) respectively. Here, \(ER_{C}\) represents the edge response for center pixel and \(ER_{P}\) presents the edge response for the \(p^{th}\) neighbor pixel. In this approach, we use a modified Frei-Chen mask to reduce the computation complexity illustrated in Figure 4. Since square root operation is costly, we remove the square root from the Frei-Chen mask, which leads to faster calculation without compromising the accuracy. The details of edge response (ER) computation are depicted in Figure 5. Then the resultant edge response (ER), including the gradient information, is used to calculate the ternary pattern using Equations (1) and (2). Finally, we concatenate the results of the three channels (red, green, and blue) to obtain the color local directional ternary pattern (CLDTP).  
\[CLDTPU_{P,R}=2^{8}.\delta(\alpha,ER_{C})+\sum_{p=0}^{7}2^{p}.\delta(\beta_{p}, ER_{p}) \tag{1}\]  
\[CLDTPL_{P,R}=2^{8}.\eta(\alpha,ER_{C})+\sum_{p=0}^{7}2^{p}.\eta(\beta_{p},ER_{p}) \tag{2}\]  
Figure 2: End-to-end flow diagram of the proposed spatiotemporal-based video annotation platform.  
where:  
\[\mu=\frac{1}{9}(I_{C}+\sum_{P=0}^{7}I_{P}) \tag{3}\] \[\alpha=\mu-I_{C} \tag{4}\]  
\[\beta_{p}=I_{p}-I_{c} \tag{5}\]  
and  
\[\delta(x,y)=\begin{cases}1&if\quad x\leq 0\quad\text{and}\quad y\leq 0\\ 0&else\end{cases} \tag{6}\]  
\[\eta(x,y)=\begin{cases}1&if\quad x\geq 0\quad\text{and}\quad y\geq 0\\ 0&else\end{cases} \tag{7}\]  
Figure 4: (**a**) Modified Frei[PERSON] masks in eight directions; **(b)** 2 nd derivative of the Gaussian filter.  
Figure 3: Block diagram of proposed color local directional ternary pattern (CLDTP).