# Ad Creative Discontinuation Prediction with Multi-Modal Multi-Task Neural Survival Networks
## 5 Offline Experiments
### Evaluation Results of Offline Experiments
#### 5.3.1 Comparison of Survival Prediction-Based Models  
First, we compared the conventional method of the Cox time-varying PH and random survival forest in terms of the statistical and text features. Table 4 shows in terms of the CI. We can regard a model with the same features (i.e., the statistical and text features) under a single-task setting as a baseline based on our DNN. Compared the DNN baseline with the conventional methods, our DNN-based framework showed better performance, with an approximate 10-point gain. By using our framework, not only can the performance be improved, but various features, such as image and time-series features, can be accounted for as well. Furthermore, comparing our DNN baseline with the full model of our framework (i.e., the model with all features under multi-task setting), we can confirm an average improvement of 3 points in prediction performance.  
For the short- and long-term models, the models that incorporated all features, such as image features, text features, and time-series features, had the best scores by an average of approximately 4 points. In the short-term model, the text and image features contributed more compared with the long-term model. However, in the long-term model, the time-series features contributed more than in the short-term model. These differences are in line with the trends of the two types of discontinuation. The effect of time-series data is described in Appendix D.  
Compared with the models where the short- and long-term models were trained individually, we observed even better performance in the model that was trained simultaneously with MTL. The performance improved significantly by approximately 5 points and 13 points in the short- and long-term models, respectively. MTL further boosted the overall performance, with a clear improvement of about 17 points.  
Table 5 shows the MTL model performance comparison to confirm the effect of the CTR- and impression-weighting techniques. By introducing the CTR-weighting technique, the prediction performance was improved in the short-term, long-term, and overall models. The impression-weighting technique performed better than the vanilla model (e.g., without  
\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Feature**} & \multicolumn{6}{c}{**Concordance Index**} \\ \cline{3-10} \multicolumn{1}{c}{\multirow{-2}{*}{**Model**}} & \multicolumn{2}{c}{**Short Term**} & \multicolumn{2}{c}{**Long Term**} & \multicolumn{2}{c}{**Overall**} \\ \cline{3-10}  & **Stat.** & **Text** & **Image** & **Time.** & **All** & \begin{tabular}{c} **Top 25\%** \\ **of Sales** \\ \end{tabular} & **All** & \
\begin{tabular}{c} **Top 25\%** \\ **of Sales** \\ \end{tabular} & **All** &
\begin{tabular}{c} **Top 25\%** \\ **of Sales** \\ \end{tabular} \\ \hline Cox time-varying PH [29] & ✓ & ✓ & & & 0.6098 & 0.7293 & 0.6574 & 0.6932 & 0.5287 & 0.5320 \\ Random Survival Forest [30] & ✓ & ✓ & & 0.6213 & 0.7578 & 0.6832 & 0.7294 & 0.5311 & 0.5487 \\ \hline
**Our framework** & Single-task & ✓ & ✓ & & 0.7958 & 0.8262 & 0.7541 & 0.8001 & 0.5874 & 0.6001 \\  & ✓ & & ✓ & 0.7962 & 0.8232 & 0.7536 & 0.8  
any weighting for the loss) but did not show a better performance compared with the CTR-weighting technique.  
In the short-term model with CTR weighting, the CI showed an improvement of approximately 3 points, and in the long-term model, the improvement was approximately 2 points. The improvement in performance was observed even when impression weighting was introduced; however, the performance improved more when the CTR-weighting technique was introduced. We assume that the large variance in the number of impressions is the reason for the difference between these scores. Thus, weighting with CTR enables training for more accurate prediction.