# Adaptive Multi-Modal Ensemble Network for Video Memorability Prediction
## 2 Related Works
A system for memorability estimation was proposed by [PERSON] et al. [28], which predicts the memorability scores of a video clip by learning from brain functional magnetic resonance imaging (fMRI). Influenced by these studies, [PERSON] et al. [3] began to wonder what makes a video memorable. [PERSON] et al. [3] proposed that the video frames can be used to extract features for predicting video memorability. Their studies have shown that saliency, color, scene complexity, background simplicity, object occurrence, and object attributes are related to the memorability scores. In the same year, [PERSON] et al. [29] focused on analyzing which fusing features could more accurately predict memorability scores. However, in previous experiments measuring video memorability scores, the data volume of the annotated dataset is insufficient. [PERSON] et al. [5] constructed a large-scale dataset named VideoMem, and the composition of VideoMem is 10,000 videos with corresponding video memorability scores. It is worth noting that VideoMem includes short-term human annotations and long-term annotations since the memory of people changed over time. [PERSON] et al. proved that video semantics information is beneficial for predicting video memorability scores. At the same time, [PERSON] et al. [30] attempted to enrich the memorability annotations of the dataset TRECVID 2019. They provided partial data with memorability annotations to carry out the memorability prediction task in MediaEval 2020 [31]. A great many novel ideas have been conceived at previous MediaEval conferences [32; 33; 34; 35; 36]. Such as MediaEval 2018, many researchers have put forward their ideas and carried out experiments to verify them. [PERSON] et al. [37] used CNNs to extract video, text, and image features. Then, features were fused to obtain a vector as global features to predict video memorability. [PERSON] et al. [38] proposed a video memorability prediction framework based on late fusion of text, visual and motion features. [PERSON] et al. [39] proved that text features are effective in the representation of visual semantics required for the video memorability prediction model. [PERSON] et al. [40] propose a novel framework to fuse the text, visual and motion features to predict video memorability. In recent years, with the development of ensemble learning and cross-modal [41; 42; 43; 44; 45], studies on ensemble learning and multi-modal are applied in various fields. [PERSON] et al. [46] presented a Group Ensemble Network (GENet) and a survey on ensemble learning [47] was expanded. [PERSON] et al. [10] proposed the method of domain adaptive ensemble learning. These research studies also encourage scholars to use ensemble learning methods to predict video memorability. [PERSON] et al. [48] proposed the ensemble methods with text, image, audio, and video features that are extracted. However, the method of the multi-modal approach to predict memorability also has some problems. Simply speaking, the ensemble weight is not updated, and the size of the dataset they used is small. In the paper by [PERSON] et al. [49], in MediaEval 2019, they used ensemble transfer learning methods with semantics and their extract features to predict media memorability scores. Specifically, each feature gets a relatively good result through the base learner, then each base learner is integrated into a strong learner by weight or others. [PERSON] et al. [50] introduce the knowledge about ensemble learning in detail and provide the latest inspiration for the current research. In the latest study, [PERSON] et al. [7] analyze the influence of semantics and time decay for video memorability and construct a multi-modal memorability dataset named Memento10k. Unlike VideoMem, Memento10k includes the memorability annotations that occur delays ranging from several seconds to ten minutes. They propose a SemanticMemNet that can predict video memorability at an arbitrary delay.