# A Survey of Data Representation for Multi-Modality Event Detection and Evolution
## 7 Datasets and Evaluation
### Evaluation  
Table 7 shows the comparison of results of SED 2011-2014 under different challenges. The F-measures as well as the normalized mutual information (NMI) are used to evaluate the performance. The results of the four years are shown in Table 7. In which, the bold number represents the best results of different challenges under SED2011-SED2014. In the first year (2011), [120] achieved the best performance in challenge 1, mainly because it classifies the photos to cities at first and then partitioned the photos into buckets that contains the photos of the same day or same city. In challenge 2, [121] achieved the best performance, mainly because the approach matches the photos to event descriptions retrieved from online event directories. In the second year (2012), the performance of challenge 1 is worse than the performance of challenge 2 and challenge 3, it is because the term \"technical events\" is fuzzy. The \begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Dataset Name** & **Source** & \begin{tabular}{c} **Number of** \\ **Event Types** \\ \end{tabular} & **Content** & \
\begin{tabular}{c} **Documents Number** \\ **of Each Event** \\ \end{tabular} &
\begin{tabular}{c} **Total Documents** \\ **Number** \\ \end{tabular} & **Year** \\ \hline MMSE (version 1) & Flickr & 10 & image and text & 2500–5000 & 36,000 & 2014 \\ MMSE (version 2) & Flickr & 12 & image and text & 3000–6000 & 59,500 & 2014 \\ MMSE (version 3) & Flickr & 18 & image and text & 4000–8000 & 107,600 & 2015 \\ HFUT-mmdata & Flickr & 10 & image and text & 7000–9000 & 74,364 & 2019 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Multi-modality social event dataset based on Flickr. \begin{table}
\begin{tabular}{l l c c c c c} \hline \hline  & \multicolumn{4}{c}{**Challenge 1**} & \multicolumn{2}{c}{**Challenge 2**} & \multicolumn{2}{c}{**Challenge 3**} \\ \hline  & & **F-Score** & **NMI** & **F-Score** & **NMI** & **F-Score** & **NMI** \\ \hline \multirow{4}{*}{SED2011} & [119] & 0.69 & 0.41 & 0.33 & 0.54 & & \\  & [120] & **0.77** & **0.63** & 0.64 & 0.38 & & \\  & [121] & 0.59 & 0.27 & **0.69** & **0.62** & & \\  & [122] & 0.65 & 0.24 & 0.50 & 0.45 & & \\ \hline \multirow{4}{*}{SED2012} & [123] & 0.22 & 0.02 & 0.30 & 0.20 & 0.48 & 0.31 \\  & [124] & **0.85** & **0.72** & **0.91** & **0.85** & **0.90** & **0.74** \\  & [125] & 0.19 & 0.18 & 0.75 & 0.67 & 0.67 & 0.47 \\  & [126] & 0.70 & 0.60 & NA & NA & 0.61 & 0.45 \\ \hline \multirow{4}{*}{SED2013} & [127] & 0.57 & 0.87 & & & & \\  & [128] & **0.95** & **0.99** & & & \\ \cline{1-1}  & [129] & 0.88 & 0.97 & & & \\ \cline{1-1}  & [130] & 0.93 & 0.98 & & & \\ \cline{1-1}  & [131] & 0.88 & 0.97 & & & \\ \cline{1-1}  & [132] & 0.78 & 0.94 & & & \\ \hline \multirow{4}{*}{SED2014} & [133] & NA & 0.98 & & & \\ \cline{1-1}  & [134] & 0.97 & 0.99 & & & \\ \cline{1-1}  & [135] & 0.94 & 0.98 & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results of comparison results on SED 2011–2014 datasets. performance of challenge 2 is better than the performance of challenge 3, it is because soccer events are much clearer than the Indignados events. Ref. [124] achieved the best performance on the three challenges. Because it also contains a city classification step and then LDA is used for topic detection for each city. In the third year (2013), the objectives of challenge 1 has some difference with SED 2011 and SED 2012, and no filtering step was required for all images in SED 2013. Since the photos in dataset were related to metadata, this actually involves a multimodal clustering. Ref. [128] achieved the best performance, because that the challenge of clustering is easier than challenges of SED 2012 and the additional process of filtering or classification is not required. In the fourth year (2014), the three methods achieved similar performance. SED 2013 and SED 2014 performed better than SED 2011 and SED 2012, mainly because in SED 2013 and SED 2014 datasets, the non-events data are filtered in advance.