# Adaptive Multi-Modal Ensemble Network for Video Memorability Prediction
## 3 Proposed Methods
### Weighted Ensemble  
Ensemble learning is an algorithm that builds and combines multiple primary learners to achieve a more robust learning capability. Most integration methods use the same basic learning algorithm to produce homogeneous basic learners, that is, the same kind of learners produce homogeneous integrations, but in this paper, we tried to use a variety of learning algorithms to train different kinds of learners to produce heterogeneous integrations, and these learners were called individual learners. The ensemble learning method improves the generalization ability by combining a group of individual learners rather than choosing the best one among them, so the combination method used is very important. After consideration, we used the weighted method to combine the individual learners of video, image, and text. The method is the most popular and primary combination method, assuming a set of T individual learner \(h_{1},\dots,h_{T}\), where the output of the learner \(h_{i}\) on example \(x\) is \(h_{i}\in R\). The task is to combine the output result of \(h_{i}\) to obtain the final prediction result on the actual value variable. The weighted method brings the combined results by assigning different importance weights to the output results of each learner. The weighted method obtains mixed results by giving different importance weights for the output results of each learner. Specifically, it obtains the combined output \(H(x)\) in such a way that  
\[H(x)=\sum_{i=1}^{T}\omega_{i}h_{i}(x) \tag{4}\]  
where \(\omega_{i}\) represents the weight of \(h_{i}\), usually with the constraint \(\omega_{i}\geq\ 0\) and \(\sum_{i=1}^{T}\omega_{i}=1\). After we obtained \(\mathbf{Vo}_{i}\), \(\mathbf{To}_{i}\) and \(\mathbf{Io}_{i}\), as mentioned in Sections 3.1-3.3, we utilized the weighted method to obtain the \(\mathbf{Vmo}_{i}\). We assigned different initial weights to the outputs of respective models such as \(\omega_{1i}\), \(\omega_{2i}\), \(\omega_{3i}\) for video \(i\). The initial learned \(\mathbf{Vmo}_{i}\) of video \(i\) can be expressed as follows:  
\[\mathbf{Vmo}_{i}=\omega_{1i}\mathbf{Vo}_{i}+\omega_{2i}\mathbf{Io}_{i}+\omega_{3i}\mathbf{To} _{i} \tag{5}\]  
As described in Section 1, video memorability is the probability of being remembered by people. That is to say, the final predicted video-memorability-output (\(\mathbf{Vmo}_{i}\)) is in the range of \([0,1]\). Therefore, the sigmoid function was used to control the prediction result within range \([0,1]\). The final video-memorability-output (\(\mathbf{Vmo}_{i}\)) is  
\[\mathbf{Vmo}_{i}=\frac{1}{1+e^{-\mathbf{Vmo}_{i}}} \tag{6}\]  
Based on the video-memorability-output (\(\mathbf{Vmo}_{i}\)) for each video \(i\), we used short-term memory and long-term memory as groundtruth. We optimized our model by minimizing the mean square error (MSE) between the predicted value and the true value through the stochastic gradient descent (SGD) algorithm, in which the loss function \(L\) can be calculated as follows:  
\[\underset{\omega_{1i},\omega_{2i},\omega_{3i}}{min}L=\frac{1}{n}\sum_{i=1}^{n} (\mathbf{Y}_{groundtruth\_i}-\mathbf{Vmo}_{i})^{2} \tag{7}\]  
where \(\mathbf{Y}_{groundtruth\_i}\) represents the groundtruth of short-term or long-term memororability scores, the number of the sample is \(n\), and \(\mathbf{Vmo}_{i}\) denotes the predicted video memororability scores, which are video-memorability-outputs. To prevent overfitting, \(L1\) regularization and \(L2\) regularization were added, and the final loss function \(L\) was  
\[\underset{\omega_{1i},\omega_{2i},\omega_{3i}}{min}L=\frac{1}{n}\sum_{i=1}^{n }(\mathbf{Y}_{groundtruth\_i}-\mathbf{Vmo}_{i})^{2}+\lambda_{1}\sum_{j=1}^{3}\left\| \omega_{ji}\right\|+\lambda_{2}\sum_{j=1}^{3}\left\|\omega_{ji}\right\|_{2} \tag{8}\]  
where \(\lambda\) is the regularization coefficient, \(\sum_{j=1}^{3}\left\|\omega_{ji}\right\|\) is \(L1\)-norm and \(\sum_{j=1}^{3}\left\|\omega_{ji}\right\|_{2}\) is \(L2\)-norm.