# A Distributed Automatic Video Annotation Platform
## 1 Introduction  
With the rapid advances of the Internet and digital devices (e.g., mobiles and video cameras) millions of videos are being taken and shared on the Internet through social media. For instance, on YouTube, more than 250 h of videos are uploaded every minute, and over 25 million visitors see more than 4 billion videos per day. Due to the increase of video data on the Internet, automated technologies to retrieve the information from the videos are required. For example, textual information plays a significant role in content-based video retrieval [1; 2], searching, browsing [3], and semantic indexing. Therefore, researchers worked on automated video annotation which aims to assign tags that belong to the frame of a video. To provide video annotation, several works have been presented [4; 5; 6; 7; 8]. Video annotation reduces the semantic gap between low-level features and high-level semantics. In the literature, video annotation annotates context, which can be categorized into two categories: the visual context and the semantic context. The visual context in video annotation provides visual information that is relevant to the frame of a video. In [8; 9; 10] single or multiple tags/labels were annotated for a frameaccording to the relevant information. However, they did not consider semantic video information. In order to improve the video annotation techniques, [4; 5] provided semantic information with the visual context which led to higher performance in terms of video retrieval, searching, and browsing. However, to obtain the semantic context, post-processing is required, such as the re-ranking method or pseudo-relevance feedback. In [5], the author used the conditional random field to gain the semantic video context. However, they ignored the weak semantic parts of information which caused loose essential semantic information. Moreover, this post-processing is complex and hard to calculate. Furthermore, the video annotation process is time consuming, especially the feature extraction part. In order to reduce the processing time, several studies have been done [11; 12; 13; 14; 15] wherein the author performed large-scale video processing in a distributed manner. However, these works did not consider distributed video annotation. In order to overcome the aforementioned issues, we introduce and develop a spatiotemporal-based distributed video annotation platform. Our works provide both visual information (spatial information) and spatiotemporal (semantic) information. We divide our work into two parts: spatial annotation and spatiotemporal annotation. Each part further divided into three parts: preprocessing, feature extraction, and tag retrieval. For feature extraction, we introduced a spatiotemporal descriptor, namely, volume local directional ternary pattern-three orthogonal planes (VLDTP-TOP). Furthermore, we developed the distributed version of the existing state-of-the-art algorithms: LBP, LTP, and LDTP for the spatial feature extraction, and MBP, VLBP, and LBP-TOP for the temporal feature extraction. We employed open-source Apache Spark to distribute the feature extraction algorithms to make the platform scalable, faster, and fault-tolerant. In addition, we provide video annotation APIsfor the developer. Since in this work, we propose a spatiotemporal-based video annotation platform, we required a new video dataset that contained spatial and temporal ground truth tags. Therefore, we introduce a new video dataset, the SpatioTemporal video Annotation Dataset (STAD). Experimental analysis was conducted on STAD, which showed the excellence of our proposed approach. The main contributions of our works are:  
* We introduce a new distributed spatiotemporal-based video annotation platform that provides visual information and spatiotemporal information of a video. * Moreover, we propose a spatiotemporal feature descriptor named volume local directional ternary pattern-three orthogonal planes (VLDTP-TOP) on top of Spark.