# A Survey of Data Representation for Multi-Modality Event Detection and Evolution
## 5 Multi-Modality Data Representation and Event Detection  
As the various online media applications proliferate, a large amount of multi-modality content is available on the internet. This content is being created and is massive. Thus, detecting events from the multi-modality data is of great significance. A multi-modality event detection system can detect events and group the related data by the representation of event data. It requires the analysis of large scale data from different sources [78], so it is a great challenge to find events from the online multi-modality data. In multi-modality event detection, the data representation is the most important part. The framework of multi-modality data representation and event detection is shown in Figure 6. The basic idea is to project multiple types of modality data in a commonly shared space to make sure that the different modalities can be compared.  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{1}{c}{**Category**} & **Reference** & **Representation** \\ \hline \multirow{6}{*}{Low-level feature} & \multirow{2}{*}{[58]} & Various static features including Scale-Invariant Feature \\  & & & Transform (SIFT) \\ \cline{3-3}  & & [60] & Three image features including SIFT, ColorSIFT and Transformed \\  & & & Color Histogram (TCH) \\ \cline{2-3}  & & [61] & Motion SIFT (MoSIFT) \\ \cline{3-3}  & & [64] & Spatio-temporal interest points (STIP), Motion SIFT and \\  & & Dense Trajectories \\ \cline{3-3}  & & [58] & Various features including 3D Histogram of Oriented \\  & & Gradients (HoG3D) \\ \cline{3-3}  & & [66] & Gradient local ternary pattern, histogram of oriented gradients and \\  & & Tamura features \\ \hline \multirow{6}{*}{High level representation} & \multirow{2}{*}{[67]} & Semantic model vectors which is an intermediate level semantic \\  & & representation extracted using a set of discriminative \\ \cline{3-3}  & & & semantic classifiers. \\ \cline{2-3}  & & [68] & Decomposed video to a sequence of shots and trained visual \\ \cline{3-3}  & & concept detectors are used to represent video content with model \\ \cline{3-3}  & & [69] & Static features (i.e., SIFT) and dynamic features (i.e., STIP and \\  & & Dense Trajectory Based features) \\ \hline \multirow{6}{*}{Deep learning based methods} & [70] & Deep convolutional neural networks (CNN) \\ \cline{3-3}  & & [71] & Deep convolutional neural networks (CNN) \\ \cline{3-3}  & & [72] & Deep convolutional neural networks (CNN) \\ \cline{3-3}  & & [73] & Deep convolutional neural networks (CNN) \\ \cline{3-3}  & & [74] & Deep convolutional neural networks (CNN) \\ \cline{3-3}  & & [75] & Enhanced Ensemble Deep Learning \\ \cline{3-3}  & & [76] & Three-dimensional convolutional layers \\ \cline{3-3}  & & [77] & Two-stage neural network strategy \\ \hline \hline \end{tabular}
\end{table}
Table 2: Representation of video based event detection.  
Multi-modality event detection can provide more comprehensive understanding of events. Multi-modality data have brought new opportunities for people to scene what is happening in the world and provide a great challenge to researchers. Firstly, the multi-modality data are heterogeneous and it is very hard to analyze the data, the data are from different sources in multiple modalities. In addition, the multi-modality data are of multiple characteristics and attributes. The multiple characteristics and attributes are also heterogeneous and important for events detection. Therefore, the representation of the multi-modality data is of great significance for multi-modality event detection. Many researchers have made a deal of efforts to address these challenges. In this section, we classify these methods into feature fusion methods, matrix factorization based methods, topic model based methods, deep learning based methods and other methods.