# Natural Language Description of Videos for Smart Surveillance
## 4 Experimental Setup  
The proposed architecture was assessed with a dataset which was manually devised from the 2007/2008 TREC video benchmark for video description. It consisted of 140 video segments used in the experiments. Each video segment duration was between 10 and 30 s with a sole camera shot. Videos were annotated by 40 human beings, and each video was described in two to seven sentences, which were stated as human annotations. They were classified into seven categories:  
* Activity: Humans carrying out some tasks, such as walking or sitting;
* Close up: Human face with the front view that displays expressions or sentiments (e.g., happy or sad);
* News: Reporter, anchor or weatherboards shown in a scene or setting;
* Meeting: Assembly of humans showing an interaction with one another, and the existence of objects normally seen in meeting scenarios (e.g., chairs, tables, mics and projectors);
* Groups: Several humans in any scene or setting;
* Traffic: Automobiles and traffic sign;
* Scene Category: Scene locations more prominent than human actions (e.g., indoor, outdoor, parks and kitchens).  
The second dataset was on UET Video Surveillance (UETVS), which consisted of 1200 surveillance videos and corresponding multi-line textual descriptions. The videos were gathered from four locations (Boys Student Service Center (BSSC), Girls Student Service Center (GSSC), Al-Khawarizmi Institute of Computer Science and the UET Bus Stand). CCTV (Closed-Circuit Television) security cameras were installed in these locations. Each location consisted of 300 videos. The span of each video lied between 7 and 10 s. The frames per second (fps) rate of each video clip was 25. These videos were described by professional English writers, with descriptions ranging between 3 and 6 sentences.  
The third dataset was self-collected (AGRIINTRUSION), which contained around 100 videos. These videos were taken from YouTube. The videos contained the type of intrusions that could happen on agricultural land, such as animals, birds, vehicles or humans. These images were annotated and described with multiple sentences.  
The video description is the explanation of nonverbal elements in a video that include facial expressions, scenery, actors' clothing or body language. It provides a description of visual elements that are important to perceive the plot, which will specifically be of service to visually impaired people. The annotation process was done on three levels: students, teachers and professional experts. Every video was annotated by three different people based on the expertise level defined above. These HLF and video descriptions were cross-checked, and discrepancies were corrected by a data annotation specialist.  
The video description experiments were performed on a GeForce GTX 1080 Ti which had 11 GB of GPU memory. The multitask framework was formed using PyTorch [28]. The CNN and LSTM models were trained and optimized together using a stochastic gradient descent optimizer with a batch size of 32. We used cross-entropy loss, which converged with the annealing learning rate scheme. The TRECViD dataset took around 62 h (~2.5 days) of training time, and UETVS took around 120 h (~5 days) of training. The AGRIINTRUSION dataset took around 45 h of training time. All video frames were regenerated to 256 \(\times\) 256 dimensions before being input into the CNN. Each video in the dataset was fed to the network at 4 frames per second. The data distribution consisted of 75% (training) and 25% (testing).