## 6 Conclusions and Future Work  
We have proposed and presented a flexible pipeline for the annotation, structure mining, and re-ID of objects in broadcast videos by exploring the semantic composition of this pipeline. The high-level features extracted from low- and mid-level visual features provided useful information about various aspects of the analysed videos. A video-mining approach was used to infer high-level semantic concepts from the low-level features extracted from the videos. The results of this video data mining were further improved by exploiting temporal correlations within the video and constructing new features from  
Figure 12: An example of ablation experiment to study the effect of depth in spatial distance estimation. Depth-based distance is found to be more comparable and less erroneous.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{3}{c}{**New Girl (10 episodes)**} & \multicolumn{3}{c}{**Friends (10 episodes)**} \\ \hline
**Class** & **True** & **False** & **True** & **False** \\ \hline bed & 29 & 0 & 152 & 0 \\ bottle & 604 & 153 & 51 & 14 \\ refrigerator & 23 & 0 & 56 & 0 \\ sofa & 76 & 0 & 306 & 13 \\ dining table & 202 & 11 & 87 & 12 \\ vase & 43 & 8 & 143 & 45 \\ bowl & 59 & 0 & 78 & 39 \\ tv & - & - & 51 & 0 \\ cup & - & - & 69 & 20 \\ car & 74 & 13 & - & - \\ handbag & 61 & 0 & - & - \\ potted plant & 20 & 0 & - & - \\ \hline
**Count** & 1212 & 187 & 993 & 143 \\ \hline
**Accuracy** & & 0.866 & & 0.874 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance evaluation of object re-ID.  
them. Boundary prediction algorithms were proposed, which clustered and segmented each video based on its structure. Furthermore, object re-ID was explored and adapted to re-ID static objects in the videos. This helped us to create object timelines, which could be interesting for a variety of applications. Our experiments show that our approach is general enough for all broadcast videos, including different genres and languages. Upon inspecting the failure cases, it was found that the selection of similarity threshold played a vital role in the overall accuracy of the pipeline. Therefore, for future work, we would look into adapting the similarity threshold automatically, which would further improve the efficiency of the pipeline. Moreover, multi-modal features and effective methods to fuse multi-modal information will be investigated. In addition, we would also further optimise the spatial location graph to include dynamic/moving objects. Finally, the framework must be evaluated on a large scale and the models should be improved accordingly.  
Conceptualization, methodology, software, validation, formal analysis, investigation, resources, data curation and writing--original draft preparation: K.K.T.C.; writing--review and editing, visualization, supervision, project administration, and funding acquisition: S.V. Both authors have read and agreed to the published version of the manuscript.  
The research activities as described in this paper were funded by Ghent University, IMEC, and the Flanders Innovation & Entrepreneurship (VLAIO) agency.  
Rai Dataset: [[https://aimagelab.ing.unimore.it/imagelab/researchActivity.asp?idActivity=019](https://aimagelab.ing.unimore.it/imagelab/researchActivity.asp?idActivity=019)]([https://aimagelab.ing.unimore.it/imagelab/researchActivity.asp?idActivity=019](https://aimagelab.ing.unimore.it/imagelab/researchActivity.asp?idActivity=019)).  
The authors declare no conflict of interest.  
References