# Natural Language Description of Videos for Smart Surveillance
## 1 Introduction  
There is an exponential increase in digital multimedia, resulting in the generation of enormous amounts of video data. This can be used to understand videos that have inspired development for a broad range of applications. The growing rate of multimedia content uploaded on the Internet involves automatic interpretation and description of the videos for the retrieval of important information. This can also be useful in surveillance, security, human-computer interaction, robotic intelligence and even helps visually impaired people. Among these applications, an automatic description of videos in a natural language is gaining interest, where we give a video to the deep learning framework that converts it into one or multiple sentences. The problem of automatically describing videos has been explored for several years. The first rule-based method [1] described human actions and activities with natural language in a restricted setting. The semantic features obtained from the video were linked with syntactic constituents, such as the subject, verb or objects, and then interpreted into natural language sentences. A sentence is formed by filling the predetermined templates with part-of-speech tags. The major problem faced at that time was bridging the semantic gap when converting videos into text. There are several papers [2; 3; 4] that applied relatively similar rule-based systems on different datasets that contained large instances of objects in diverse situations. Later, more complex rules were applied in [5] that contained a relatively large vocabulary to generate sentences. These approaches require monotonous work when the data is huge. To eliminate this problem, statistical models [6; 7] have been used, which can train even larger datasets that contain many lexical entries and many hours of videos. However, the results of these approaches were lacking with large datasets like Microsoft Common Objects in Context (MS COCO) [8]. Lately, a lot of research was carried out for applying deep learning methods to video captioning. Encouraging results have been attained by using a convolutional neural network (CNN) or a recurrent neural network (RNN) for labeling [9], image captioning [10] and video descriptions. These approaches have been applied to short video segments that contain action, anomaly detection or scene surroundings. For each high-level feature (HLF), the state-of-the-art methods used different methods of feature extraction, such as for human action recognition (star skeleton and a hidden Markov model), object detection (Haar features) and age (facial features). This might result in the feature extraction of narrow domains, rigid template-based sentences and missing information, thus making it difficult to produce correct results for large data. Our proposed deep learning model is a single carefully designed and trained network that can extract various HLFs simultaneously with language model incorporation. The model focuses on better visual information extraction from the frames, which is reflected in the results. We have captured the power of semantic feature extraction with a relatively simple deep learning architecture. This has been achieved by fine-tuning a multitask CNN, which can learn dense features for a scene, human, dress, object and action. In terms of computational complexity, the proposed model is quite inexpensive for the prediction of unseen data. This paper suggests a deep neural network with multitask feature learning to interpret videos into natural language descriptions.