# Natural Language Description of Videos for Smart Surveillance
## 2 Literature Survey  
The prior video description approaches were established on finding subject-verb-object (SVO) triplets from the video frames, and captions were produced through a language model which was built on predefined sentence templates. This requires training multiple classifiers that could identify anomalies, activities, human beings and their properties, such as emotion or gender, objects, actions and scene settings. These extracted features are combined with a language model to generate sentences from templates. In [14], the authors extracted SVO triplets and learned a semantic structure on subjects, objects and verbs by using a multichannel SVM (Support Vector Machine) to predict these values. Once the triplet values are found, a sentence is formed using templates. A factor graph model was proposed in [15] that integrated visual detection with language statistics to learn the subject-verb-object-place (SVOP) tuple for a video. The sentences are generated using a template-based approach. Although the problem in both cases is simplified, this still entails choosing suitable objects and actions. Additionally, the sentences generated from this approach are not flexible enough to generalize unseen, erroneous and misidentified data and cannot satisfy the richness of natural language. Based on the reasons above, research on image or video description generation has shifted to deep learning approaches. The simple plan for description generation is to use a CNN for encoding visual information and an RNN for decoding natural language sentences. CNNs are like a simple neural network, but the architecture makes a clear supposition that the inputs are images, which allows certain properties to be encoded. These networks are mostly used in image and video recognition systems. RNNs are networks that have looping capability and grant information to prevail. These networks are extremely popular for multiple problems such as speech recognition, modeling languages, image or video captioning, translation and description. The hierarchical recurrent neural network (hRNN) [16] is the most credited state-of-the-art deep learning technique for video captioning and description. This network can generate sentences (multiple sentences) or a paragraph for a lengthy video which contains multiple scenarios or scenes. These types of videos cannot be depicted in a single sentence, but rather give a dull description. The idea of this framework is to utilize time-based dependency among sentences in a paragraph. While generating the next sentence, the semantic context of the previous sentence is put into use. RNNs keep information in memory over a specific time period. However, it is hard to resolve problems that need long temporal dependencies, such as understanding the present frame of a video that requires information from the previous frame or frames. Long short-term memory (LSTM) networks are a specific kind of RNN that works for several tasks and performs better. An LSTM unit comprises a memory cell that can withstand data for a lengthier time period. Rapid development has been made in previous years to learn image features, and numerous pretrained models are offered. However, these deep features cannot be used for videos due to the absence of temporal information. The CNN and LSTM were integrated in [17] to learn spatial (space) and temporal (time) information from videos. Features are mined from a 2D CNN and input into an LSTM network to translate the time-based information of the videos. There are other variations of LSTM, such as BiLSTM (Bidirectional Long short-term memory) [18]. The authors proposed a bidirectional LSTM with a soft attention mechanism. This conserve the global, temporal and visual information of an input video, and the attention process helps in identifying the most important words in the content. The encoder-decoder framework [19] for generating video captions has been proposed. The encoder extract features and encodes the input into vectors. Two models have been used to extract features from videos.