# Counter-Terrorism Video Analysis Using Hash-Based Algorithms
## 4 Architectural Framework  
This section presents the architectural framework for video extraction. The proposed framework is based on ETL (Extract, Transform and Load) techniques for Big Data, designed specifically for the extraction of terrorist propaganda videos from the Internet, using search engines such as Google or Bing. Big Data is the distinctive feature of this project because it allows for a distributed workflow in video retrieval, performing the process at a much higher speed than if the workflow were non-distributed. Hence a distributed system is more reliable, and it can be easily expanded for future work. In terms of video retrieval, the main focus of the case study was to detect sample videos of terrorist attacks by launching crawler instances with terrorism-related keywords as inputs. Since terrorism is a sensitive topic, search engines such as Google and Bing need to be analyzed to check what kind of videos they provide. After some manual searches looking for terrorism videos, it was concluded that keywords: _terrorism_ and _attack_ together gave better results than using single word inputs or just radical group names. By better results we mean a higher number of real terrorist attack videos or their fragments. After watching the retrieved videos via web crawling, it was determined that the estimated length for a terrorist attack video was usually under 5 min, because when the video was longer it tended to be either a documentary film, a reporter or a fiction video (from movies or video games). Hence, a limit of 5 min has been established as the maximum length per video, this helps prevent the overloading of the system with useless videos while providing more concrete results. As mentioned before, the ETL techniques implemented in the _distributed system_ use web crawlers to retrieve Google and Bing search results, as described in Figure 3. All the search results are obtained from Google's and Bing's Video Section, so video retrieval is based on the input keywords and additional parameters offered by both search engines e.g., such as the video length. Figure 3: Proposed system for storage of retrieved videos. Figure 2: Frame examples. Having explained in detail the different elements of the video retrieval architecture, below we outline every principle of the ETL process and its application:  
* **Extract**: the data retrieved from Google and Bing Crawlers, such as links to terrorist attack videos, are used as the entry point of the extraction process. As established, the length of the videos whose information is extracted is always under 5 min. Once all the links are listed in MongoDB, the output of the Web Crawler goes directly to a noSQL database, as a result those links need to be transformed into useful data i.e., videos. * **Transform**: this process, which in this case can be broken down into two steps, entails transforming data into a desired output: 1. The video is downloaded using a stream and it is written in the system as an MP4 file for its subsequent analysis. 2. Parse and decompose the video into frames that are individually converted into a hash. * **Load**: afterwards all this data is inserted in MongoDB as a single document for every video, containing the respective hash for every listed frame. This is the final output of the ETL process, hence videos can now be processed and analyzed as explained before. As mentioned above, to retrieve videos from search engines such as Google or Bing, it is necessary to develop web crawlers to get the links of the indexed videos on the HTML DOM Tree of the resulting webs or every search engine. Once we have all the video links listed, we use a buffer where we dump all the retrieved data via stream so it can be saved later as an MP4 video file. We propose an application based on web crawlers where we specify the input (keywords introduced into Google and Bing search engines) and the web crawlers automatically retrieve all the links indexed in their respective video pages, download them, and store them as MP4 files.