# Adaptive Multi-Modal Ensemble Network for Video Memorability Prediction
## 1 Introduction  
Human memorability can be regarded as the criterion for judging whether the content of multi-media can be remembered or not [1]. In current social media, humans are exposed to continual and vast multi-media information, such as images, videos, audio and text. Human memorability has different responses to different multi-media contents, some of which are stuck in our memory while others are forgotten easily. For example, if the content appeals to the observers, such as their favorite celebrities or shocking natural scenes, the media information can be remembered easily. Therefore, research on media information memorability can contribute to advertising, intelligent recommendation, and so many other applications. Media memorability aims to score the probability of being remembered according to the media content. Intuitively, media memorability is affected by subjective factors, but some studies have proved that memorability is the inherent attribute of media information [2]. For example, [PERSON] et al. [2] demonstrate that image memorability is a stable intrinsic property. The image memorability (IM) score is defined as the percentage of correct detection by different observers. Based on observation, many works focus on exploiting the factors affecting image memorability and designing models for predicting image memorability. More recently, the amount of videos on social media, such as Tik-Tok, Youtube, etc., has been growing exponentially. Therefore, video memorability is nascent as a new research field. Compared with images, videos are dynamic and contain temporal information. Therefore, the study of video memory is different from that of images. [PERSON] et al. [3] began to wonder what makes a video memorable and propose that the video frames can be used to extract features for predicting video memorability. Similarly, the video memorability (VM) score is defined as the percentage of correct detection by different observers. However, the drawback of current research on video memorability is that the main attributes affecting memorability prediction have not been determined, making the prediction model's design more challenging. This paper analyzes and experimentally verifies the critical impact factors in video memorability and proposes a framework for efficiently predicting video memorability. Firstly, visual and semantic factors play a decisive role in video memorability prediction. The combination of vision and semantics has produced a powerful effect on human memory. Secondly, multi-source feature fusion usually yields better results than isolated modeling generally. However, merging multiple features of the same information source adds complexity and causes heterogeneous errors [4]. Thirdly, it has been studied that the memory interval affects the video memorability [5]. Thus, we consider the memory interval by taking the memory after memorizing a few minutes as the short-term memory and using the long-term memory to define the memory performance after 24-72 h. Choosing different media information sources can predict the change in video memorability from different angles. Finally, we choose temporal 3D information, spatial information, and semantics derived from the three media information, including videos, images and captions, respectively. Considering the redundancy effect of feature fusion, we use different base individual learners for each media information source to predict memorability scores. Concretely, we train the Resnet3D network [6] model to predict video memorability scores. For the selection of the spatial information, based on the previous experimental results [4], Local Binary Patterns (LBP) [7] features are more favorable for predicting video memorability scores than RGB [8] and High Osmolarity Glycerol (HOG) [9]. As for the text features, the semantics are phrases or sentences sufficient to describe the video scene, and we extract features from semantics as input to the semantic modal model.