Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization
Year: 2023

Abstract:
Multimodal abstractive summarization (MAS) aims to produce a concise summary given the multimodal data (text and vision). Existing studies mainly focus on how to effectively use the visual features from the perspective of an article, having achieved impressive success on the high-resource English dataset. However, less attention has been paid to the visual features from the perspective of the summary, which may limit the model performance, especially in the lowand zero-resource scenarios. In this paper, we propose to improve the summary quality through summary-oriented visual features. To this end, we devise two auxiliary tasks including vision to summary task and masked image modeling task. Together with the main summarization task, we optimize the MAS model via the training objectives of all these tasks. By these means, the MAS model can be enhanced by capturing the summaryoriented visual features, thereby yielding more accurate summaries. Experiments on 44 languages, covering mid-high-, low-, and zeroresource scenarios, verify the effectiveness and superiority of the proposed approach, which achieves state-of-the-art performance under all scenarios. Additionally, we will contribute a large-scale multilingual multimodal abstractive summarization (MM-Sum) dataset.1

heading: 1 Introduction
text: Given an article and several images as inputs, as shown in Fig. 1, multimodal abstractive summarization (MAS) (Sanabria et al., 2018; Li et al., 2017, 2018a; Zhu et al., 2018; Jangra et al., 2020) aims to generate a concise textual summary, which can help people quickly grasp the core information. Therefore, MAS has widespread application and
∗Work was done when Liang and Wang was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China.
† Jinan Xu is the corresponding author. 1The code and data are publicly available at: https://
github.com/XL2248/SOV-MAS.
attracts increasing attention with the rapid proliferation of multimedia content (Apostolidis et al., 2021; Feng et al., 2022; Qiu et al., 2022).
Recently, many studies have been carried out to effectively inject the visual features into MAS models (Li et al., 2018b, 2020b; Zhu et al., 2020, 2021; Zhang et al., 2021b,a; Palaskar et al., 2019; Liu et al., 2020; Yu et al., 2021a). For instance, Palaskar et al. (2019) and Zhang et al. (2021a) explore the hierarchy between the textual article and visual features, and integrate them into the MAS model. Liu et al. (2020) design a multistage fusion network to model the fine-grained interactions between the two modalities. And Yu et al. (2021a) study multiple multimodal fusion methods to infuse the visual features into generative pre-trained language models, e.g., BART (Lewis et al., 2020). Despite their success on the high-resource English dataset, they only model visual features from the perspective of an article and neglect the relevance of visual features to the summary, which restricts their potential performance especially on the training dataset with limited scale. For example, though the object “black clothes” in the first image of Fig. 1 is associated with the article content (red part), the object
2934
contributes little to the summary. Thus, the MAS model should focus on summary-oriented visual features. However, the visual features are generally implicitly learned via the MAS objective, which cannot help the model learn to explicitly discard the needless visual information.
To address this issue, in this paper, we propose a Summary-Oriented Vision enhanced MAS (SOVMAS) training framework to generate more accurate summaries through explicitly improving the relevance of visual features to the summary. To this end, we design two summary-oriented vision modeling tasks, namely vision to summary task, and masked image modeling task. Specifically, as shown in Fig. 2, (1) the vision to summary task is to produce the concise summary by only taking the image sequence; (2) the masked image modeling task aims to predict the semantic class distribution of the regions in one fully masked image given the summary and the remaining images. Together with the main multimodal summarization task, the MAS model is optimized through the joint objectives of all these tasks. In this way, the model is enhanced to explicitly exploit the summary-oriented visual features, thus leading to more accurate summaries.
To validate the SOV-MAS framework on various languages and diverse settings, we construct the first large-scale Multilingual Multimodal Summarization dataset (MM-Sum) based on XLSum (Hasan et al., 2021), a multilingual summarization dataset. The MM-Sum covers 44 languages with mid-high-, low- and zero-resource scenarios. Experiments on these settings show that our model significantly outperforms related methods in terms of ROUGE (Lin, 2004) scores, especially under the low- and zero-resource settings, demonstrating its effectiveness. Besides, we extend our approach to two previous best MAS models (i.e., VG-BART and VG-T5 (Yu et al., 2021a)). Human evaluation and the results on How2 (Sanabria et al., 2018) benchmark further suggest the superiority and generalizability of our approach. In summary, our main contributions are:
• To the best of our knowledge, we are the first that contributes a large-scale multilingual multimodal summarization dataset (44 languages, 1.1M article-summary pairs with 3.5M images).
• We propose two general summary-oriented vision modeling tasks, which substantially boost the summary quality and are flexible and easy to be extended to existing MAS models.
• Experiments on MM-Sum show that our model builds new state-of-the-art performance in all scenarios, especially on the low and zero resource where the fewer the data are (midhigh→low→zero), the greater the improvement we gain. Besides, results on the How2 dataset show the generalizability of our approach.
• When jointly training the MAS model on multiple languages, we find that our model learns transferable visual features among languages, where the vision serves as an anchor in the zeroresource languages.

heading: 2 Background

heading: 2.1 Problem Formulation
text: Given an input article X={xk}|X |k=1 and the corresponding object sequence O={oij}i≤n,j≤mi=1,j=1 , where xk denotes the k-th token and oij represents the detected j-th object of the i-th image (n, m is the number of images and detected objects in each image, respectively), the MAS task is defined as:
p(Y|X ,O) = |Y|∏
t=1
p(yt|X ,O, y<t),
where y<t indicates the tokens before the t-th time step in the summary Y={yt}|Y|t=1.

heading: 2.2 The MAS Model
text: Based on the pre-trained language models (e.g., BART), Yu et al. (2021a) design a variant of transformer (Vaswani et al., 2017) with four modules: textual encoder, visual encoder, text-vision fusion, and decoder, as shown in the left part of Fig. 2, which achieves good performance on MAS. Textual Encoder. The input text X is firstly tokenized and mapped to a sequence of token embeddings X. Then, the positional encodings Epe are pointwisely added to X to keep the positional information (Vaswani et al., 2017):
Z0T = X+Epe, {Z0T ,X,Epe} ∈ R|X |×d, where d is the feature dimension. It forms the input features Z0T to the encoder, which consists of L stacked layers and each layer includes two sublayers: 1) Multi-Head Attention (MHA) and 2) a position-wise Feed-Forward Network (FFN):
SℓT = MHA(Z ℓ−1 T ) + Z ℓ−1 T , S ℓ T ∈ R|X |×d, ZℓT = FFN(S ℓ T ) + S ℓ T , Z ℓ T ∈ R|X |×d,
where ZℓT is the state of the ℓ-th encoder layer.
Visual Encoder. Following Yu et al. (2021a); Zhang et al. (2021a,b); Liang et al. (2021, 2022a,b), the object sequence O is extracted from the image by the Faster R-CNNs (Ren et al., 2015) (actually, we have several images instead of only one image, please refer to § 3.1 for details). Then the visual features are fed into the visual encoder with H layers. Finally, we obtain the output visual features ZHV :
ShV = MHA(Z h−1 V ) + Z h−1 V , S h V ∈ R|O|×dv , ZhV = FFN(S h V ) + S h V , Z h V ∈ R|O|×dv ,
where ZhV is the extracted visual features O. Text-Vision Fusion. The fusion method is visionguided multi-head attention. Firstly, the query Q is linearly projected from the textual features ZLT , and the key K and value V are linearly projected from the visual features ZHV . Secondly, a Crossmodal Multi-Head Attention (CMHA) is applied to get the text queried visual features M. Then, a forget gate G is used to filter redundant and noisy information from the visual features. Finally, we obtain the vision-guided output ZT+V by concatenating the textual features ZLT and the result of a point-wise multiplication G⊗M, and then linearly project it to the original dimension d. Formally, the text-vision fusion process is:
Q = ZLTWq, Q ∈ R|X |×dc , K = ZHV Wk, V = Z H V Wv, K,V ∈ R|O|×dc , M = CMHA(Q,K,V), M ∈ R|X |×dc , G = Sigmoid(Concat(ZLT ,M)Wg + bg),
ZT+V = Concat(Z L T ,G⊗M)Wz + bz,
where Concat is the concatenation operation and W∗ and b∗ are trainable weights.
Decoder. Similar to the encoder, but each of L decoder layers includes an additional Multi-Head
Cross-Attention sub-layer (MHCA):
Sℓdec = MHA(Z ℓ−1 dec ) + Z ℓ−1 dec , S ℓ−1 dec ∈ R|Y|×d, Cℓdec = MHCA(S ℓ dec,ZT+V ) + S ℓ dec, Zℓdec = FFN(C ℓ dec) +C ℓ dec, C ℓ dec ∈ R|Y|×d, (1) where Zℓdec ∈ R|Y|×d denotes the state of the ℓ-th decoder layer. Then, at each decoding time step t, the top-layer (L-th) decoder hidden state ZLdec,t is fed into the softmax layer to produce the probability distribution of the next target token as:
p(yt|X ,O, y<t) = Softmax(WoZLdec,t + bo), where Wo and bo are trainable weights.
Finally, the loss function is formalized as:
LMAS = − |Y|∑
t=1
log(p(yt|X ,O, y<t)). (2)

heading: 3 SOV-MAS Framework
text: Based on the vision-guided pre-trained language model described in § 2.2, we introduce the proposed Summary-Oriented Vision enhanced MAS ((SOV-MAS)) framework. Specifically, we firstly describe the process of visual features extraction in § 3.1. Then, to make the best use of visual features, we design two summary-oriented vision modeling tasks in § 3.2, namely vision to summary task and masked image modeling task. Finally, we describe the training and inference in § 3.3.

heading: 3.1 Visual Features Extraction
text: As described in § 2.2, there is an image sequence to be extracted by the Faster R-CNNs (Ren et al., 2015) pre-trained on Visual Genome (Krishna et al., 2017). Specifically, for the i-th input image, we obtain a set of detected objects from Faster R-CNNs, i.e., Ii = {vi,1,vi,2,vi,3, ...,vi,m}, where m is the
number of extracted objects and vi,∗ ∈ Rdv . Each object is captured by a dense feature representation, which can be mapped back to a bounding box / region (i.e., Region-of-Interest (RoI)). Finally, the image sequence is converted to visual features I={vij}i≤n,j≤mi=1,j=1 .
Besides these features from Faster R-CNN, given the fact that Transformer (Vasava et al., 2022) is becoming popular in computer vision, we experiment with the visual features extracted by the pretrained Transformer models (i.e., ViT (Dosovitskiy et al., 2020)).
To keep the order information of the image sequence, each image region is encoded as a sum of four types of features (Cho et al., 2021):
oij = vij +E box ij +E img i +E reg j ; i ≤ n, j ≤ m,
where Eboxij ∈ Rdv denotes RoI bounding box coordinates, which are encoded with a linear layer; Eimgi ∈ Rdv denotes image id embedding, which is used to discriminate regions from different images; and Eregj ∈ Rdv denotes region id embedding. The image ids and region ids are encoded with learned embeddings (Devlin et al., 2019). The final visual embeddings are denoted as O={oij}i≤n,j≤mi=1,j=1 . Then, they are fed into the visual encoder for better modeling the intramodal dynamics and enhancing the vision-specific order information.

heading: 3.2 Summary-Oriented Vision Modeling
text: We elaborately design two summary-oriented vision modeling tasks, namely vision to summary task and masked image modeling task, to focus on the summary-oriented visual features. Vision to Summary Task (Vis2Sum). As illustrated in the right part of Fig. 2 (a), given the object sequence O extracted from the image sequence, the Vis2Sum task forces the MAS model to directly generate the corresponding summary Y without seeing the article X . In this manner, the MAS model could acquire the ability to roughly understand the summary and grasp the overall situation. Particularly, we firstly use the visual encoder to encode O, and then use the MAS decoder to predict Y . The training objective of this task can be formulated as:
LVis2Sum = − |Y|∑
t=1
log(p(yt|O, y<t)),
p(yt|O, y<t) = Softmax(WoZL,Vdec,t + bo), (3)
where ZL,Vdec,t is the top-layer decoder hidden state
at the t-th decoding step, while the input of MHCA is the visual features ZHV instead of ZT+V in Eq. 1. Masked Image Modeling Task (MIM). Our MIM task aims to predict the semantic class distribution of the regions in one fully masked image. As illustrated in the right part of Fig. 2 (b), for the input of the visual encoder, we firstly mask all regions in one random image (i.e., m objects/regions), which are replaced with zero vectors. Then, we concatenate the masked object sequence Omask and the summary Y . After feeding the concatenated input [Omask; Y] to the encoder, an MLP classifier is stacked over the output of each masked region to predict the semantic class distribution. Specifically, we denote the predicted class distribution of the r-th masked region as p(ZH,maskV,r ), and use q(Or) to represent the class distribution detected by the Faster R-CNNs (Ren et al., 2015). The loss function for the MIM is to minimize the KL divergence (Kingma and Welling, 2013) of the two class distributions:
LMIM = m∑
r=1
DKL(q(Or)||p(ZH,maskV,r )). (4)
Besides, as a variant, we randomly mask regions in the image sequence with a probability of 15% following previous work (Xing et al., 2021). We denote it as masked region modeling (MRM) and show its effect in Tab. 4.

heading: 3.3 Training and Inference
text: Monolingual Training. For monolingual summarization, with the main MAS task and the two auxiliary tasks, the training objective on one specific language is finally formulated as:
JMono = LMAS + αLVis2Sum + βLMIM, (5) where α and β are balancing factors for the tradeoff between LMAS and the auxiliary objectives. Multilingual Training. For multilingual summarization, the model can deal with inputs in multiple languages and predict the summary in the corresponding language. Specifically, for each language lk in the set of K languages Lang = {l1, l2, ..., lK}, the training objective is:
JMulti = K∑
k=1
(J lkMono). (6)
During inference, the two auxiliary tasks are not involved and only the MAS model is used to conduct summarization.
Monolingual Training Multilingual Training Languages mT5 VG-mT5 SOV-MAS (ours) mT5 VG-mT5 SOV-MAS (ours) Arabic 33.67/14.06/27.83 33.88/14.20/28.00 33.63/13.83/27.64 34.34/14.30/28.43 33.42/13.58/27.62 34.74/14.48/28.84 Chinese 40.20/25.39/33.49 39.99/25.19/33.19
:::::::::::::::: 40.59/25.32/33.36 40.30/24.97/33.04 40.14/25.29/33.31 41.59/26.52/34.53
English 36.99/15.18/29.64 37.17/14.88/29.41 37.26/15.02/29.61 36.65/13.91/28.53 36.62/14.13/28.76 37.86/15.23/29.89 Hindi 33.66/13.14/27.71 34.82/13.94/28.59 34.83/13.60/28.25 35.50/13.91/28.52 35.36/14.16/28.87 36.42/14.95/29.77 Indonesian 35.10/15.44/28.91 35.47/15.47/29.12 35.17/15.35/28.85 35.84/15.66/29.40 36.50/16.31/30.13 37.50/17.33/31.22 Persian 36.14/15.55/29.25 36.12/15.59/29.15 36.44/15.92/29.50 36.39/15.84/29.45 36.71/16.19/29.80 37.69/16.90/30.71 Portuguese 30.13/10.32/22.06 29.69/ 9.82/22.10 29.83/10.05/21.78 30.84/10.92/22.64 31.22/11.43/23.24 32.32/11.90/23.83 Russian 30.01/12.47/24.28 31.38/13.02/25.22
:::::::::::::::: 31.86/13.38/25.45 31.12/12.33/24.67 30.42/12.29/24.38 31.96/13.30/25.69
Spanish 29.51/10.48/22.51 29.50/10.62/22.47 29.27/10.40/22.43 29.91/10.70/22.66 30.57/10.96/23.21 31.20/11.64/23.73 Tamil 22.31/10.08/20.36 22.30/10.15/20.39
:::::::::::::::: 22.82/10.55/20.67 22.96/10.05/20.75 23.04/10.25/20.94 24.22/10.79/21.92
Turkish 30.37/14.39/26.79 30.51/14.41/26.76 :::::::::::::::: 31.02/14.64/27.20 31.93/14.69/27.76 31.44/14.73/27.71 32.94/15.77/29.01 Ukrainian 21.57/ 8.66/18.64 21.71/ 8.89/18.79 21.84/ 8.62/18.69 22.79/ 9.13/19.46 22.60/ 9.27/19.55 23.91/ 9.97/20.53 Urdu 38.22/17.25/31.37 38.07/17.31/31.54 38.10/16.98/31.18 38.15/17.12/31.36 38.04/17.32/31.67 39.38/18.38/32.76 Vietnamese 32.18/15.84/24.83 32.18/15.98/24.84 32.22/15.99/24.95 33.71/16.72/25.97 33.78/17.06/26.32 34.78/17.85/27.17 Avg. 32.14/14.16/26.26 32.34/14.24/26.39 32.49/14.26/26.40 32.88/14.30/26.61 32.84/14.49/26.82 34.04/15.36/27.83
Table 1: The R-1/R-2/R-L results on the mid-high-resource scenario. “
:::: */*/* ” and “ */*/* ” denote statistically
significant better than the “VG-mT5” model with t-test p < 0.05 and p < 0.01 hereinafter, respectively. The “Avg.” indicates average score for each group and the best average scores are bold.

heading: 4 Experiments

heading: 4.1 MM-Sum Dataset
text: There is no multilingual MAS benchmark dataset until now. We construct one as follows. Data Source and Data Construction. Based on the XL-Sum dataset (Hasan et al., 2021), we construct a Multilingual Multimodal abstractive Summarization (MM-Sum) dataset. The original XL-Sum dataset is crawled from the BBC website2 and its quality has been verified and ensured reliability by Hasan et al. (2021). However, the lack of associated image sequence in XL-Sum, makes it impossible to directly conduct research on MAS. Therefore, we strictly follow the procedure of (Hasan et al., 2021) to further offer the image sequence for the corresponding textual summarization dataset, where we maintain the articlesummary pair if it contains images and keep the image order appearing in the article. Dataset Statistics and Splits. Tab. 7 of Appendix A shows the detailed statistic of our MMSum and please refer to it for details. According to the dataset size of each language, we split them into three settings: Mid-High Resource, Low Resource, and Zero Resource. For mid-high and low-resource languages, following Hasan et al. (2021), we utilize about 80% training:10% validation:10% test splitting with one exception (English splitting is 93%:3.5%:3.5%). For zero resource, we following Bugliarello et al. (2022) investigate two scenarios: few-shot and zero-shot. Therefore, we also randomly sample 100 instances as the few-shot
2https://www.bbc.com/
learning data and then split the rest with about 50% validation and 50% test.

heading: 4.2 Setup and Metrics
text: Implementation Details. Please refer to Appendix B for implementation details including data pre-processing and hyper-parameters settings. Metrics. Following Hasan et al. (2021), we use the standard ROUGE scores (R-1, R-2, and R-L) (Lin, 2004) with the statistical significance test (Koehn, 2004) for a fair comparison.

heading: 4.3 Comparison Models
text: Text-Only MAS Systems.
• mT5: We choose the mT5 (Xue et al., 2021), a multilingual language model pre-trained on a large dataset of 101 languages, as the text-only baseline which is fine-tuned on our dataset.
Vision-Guided MAS Systems.
• VG-mT5: We implement the fusion method described in § 2.2 to inject visual features into the mT5 model, which is a strong baseline. • SOV-MAS: It is the proposed model with two summary-oriented auxiliary tasks to enhance MAS model as described in § 3.
All the above models involve two training manners: monolingual training and multilingual training. Specifically, for monolingual training, we train the model on the training dataset of each language. For multilingual training, we train the model on the whole training dataset of mid-highresource and low-resource languages.

heading: 4.4 Main Results
text: Tab. 1, Tab. 2, and Tab. 3 present the main results on mid-high-, low-, and zero-resource scenarios under monolingual and multilingual training settings. Overall, our model obtains notably better results than the text-only “mT5” model on both settings. 1) In the monolingual training setting, we find that the fewer the data are (mid-high→low→zero), the greater the improvement we gain, showing that our approach plays an increasing role in vision modeling. 2) In the multilingual training setting, the results show that our approach learns transferable visual features among languages, especially on the zero-resource ones where the vision serves as an anchor. These results not only show the effectiveness of our approach but also the value of our MM-Sum dataset.
Results on Mid-High-Resource Scenario. In Tab. 1, 1) on the whole, the results of the multilingual training group (e.g., SOV-MAS) substantially outperform those of the monolingual training group, demonstrating the task knowledge among languages is transferable. 2) Under the monolingual training setting, the text-only baseline “mT5” performs worse than the “VG-mT5” model on most languages, showing that the visual features indeed supplement some crucial information for the summarization. With the summary-oriented vision modeling tasks, our model further promotes the quality of the summary (“SOV-MAS” vs. “VGmT5”), demonstrating the effectiveness of our approach. 3) Under the multilingual training setting, our model consistently and significantly surpasses both the text-only and vision-guided baselines by large margins (e.g., the previous best “VG-mT5”,
up to 1.20/0.87/1.01 ROUGE scores on average). Further, in the monolingual setting, the data scale is large while it may be not enough to learn better summary-oriented image features. That’s, the improved image features may not supplement much more information compared with the large textual data. However, in multilingual training, the data scale is much larger and enough for learning the better summary-oriented image features, which help the model capture more summary-related information. Thus, the SOV-MAS achieves more significant results than in a monolingual setting.
Results on Low-Resource Scenario. Under the low-resource languages, in Tab. 2, we observe similar findings as in the Mid-High-Resource scenario. This demonstrates that our conclusions are solid and convincing on general languages. All these results prove the effectiveness of our approach.
Further, in this setting, the data may be not enough for learning the better summary-oriented image features. However, the learned image features still could offer a sketch of the summary and help the model to focus more on the summaryrelated parts. This may compensate for the impact of insufficient data. Therefore, the SOV-MAS also obtains significant gains.
Results on Zero-Resource Scenario (Zero-Shot). On the zero-shot setting in the left group of Tab. 3, the “VG-mT5” model notably exceeds the textonly “mT5” model by averagely 0.56/0.22/0.49↑ ROUGE scores. It indicates that the image in our MM-Sum plays a key role when transferring knowledge from mid-high and low-resource languages to zero-resource languages via considering vision as the anchor, where the vision is free from different
languages. Furthermore, our model presents significant improvements over the “mT5” model by averagely 1.27/0.41/1.05↑ ROUGE gains, which shows its effectiveness again.
Results on Zero-Resource Scenario (Few-Shot). On the few-shot setting, we merge the 100 samples of each zero-resource language to continue training the multilingual training model for 3,000 steps. The results are shown in the right group of Tab. 3, which shows that with a handful of data the models can greatly increase the ROUGE scores compared with zero-shot results. Our approach still achieves the best results, showing the effectiveness of our approach again. It also suggests that there is much room for further improvement using more data or other more advanced text-vision fusion methods.
Besides, we listed the results with the visual features extracted by the pretrained Transformer vision encoder, i.e., ViT (Dosovitskiy et al., 2020), in Tab. 8 and Tab. 9 of the appendix, demonstrating that our SOV-MAS still achieves better performance in almost all cases, showing its superiority.

heading: 5 Analysis

heading: 5.1 Ablation Study
text: We conduct ablation studies to investigate how well the two auxiliary tasks work. The results are shown in Tab. 4. We have the following findings:
• The Vis2Sum task shows a positive impact on the model performance (row 1 vs. row 0), demonstrating that the image sequence may reflect a sketch of the summary, which is beneficial to the summary generation; • The MIM substantially improves the MAS model
in terms of ROUGE scores (row 2 vs. row 0), suggesting that reconstructing the masked image with the summary is helpful to summarization; • The two summary-oriented vision modeling tasks exhibit notable cumulative benefits (row 3 vs. rows 0∼2), showing that focusing on the summary-oriented visual features is effective; • The variant MRM makes relatively smaller contributions to the MAS model compared with the MIM (row 4 vs. row 2). The reason may be that it is easy for the concise summary to complete the masked globally full image rather than the masked locally disordered regions (actually, the local regions might not be mentioned in the summary as described in § 1, and thus it is hard to reconstruct them given the concise summary).

heading: 5.2 Human Evaluation
text: To further evaluate the performances of mT5, VGmT5 and our SOV-MAS, we conduct human studies on 50 samples randomly selected from English and Chinese test sets. We invited three Chinese postgraduate students who are highly proficient in English comprehension 3 to compare the generated
3One student has passed TEM-8 (with 81 points out of 100 points). The other two students have passed the IELTS exam (their scores of reading comprehension are 8.0 and 7.0 out of
summaries under the multilingual training setting and assess each summary from three independent perspectives: fluency (Flu.), conciseness (Conci.) and informativeness (Info.). We ask them to assess each aspect with a score ranging from 1 (worst) to 5 (best). The average results are presented in Tab. 5.
Tab. 5 shows the human results on English and Chinese. We find that our SOV-MAS outperforms all compared models from all criteria in both languages, which further demonstrates the effectiveness and superiority of our model. The Fleiss’ Kappa scores (Fleiss and Cohen, 1973) of Flu., Conci and Info. are 0.69, 0.65 and 0.56, respectively, which indicates a substantial agreement among three evaluators. We also present a case study in Appendix C.

heading: 5.3 Results on How2 Dataset
text: To investigate the generality of the two summaryoriented vision modeling tasks, we extend them to two existing MAS models (i.e., VG-T5 and VGBART (Yu et al., 2021a)), denoted as “SOV-MAS (T5)” and “SOV-MAS (BART)”, respectively. As shown in Tab. 6, we also compare our models with the following systems, including text-only models: S2S, PG, Trans., T5, and BART, and prior best vision-guided models: HA (RNN/Trans.), MFFG (RNN/Trans.), VG-T5, and VG-BART.
The results on How2 dataset (Sanabria et al., 2018), a widely-used English MAS dataset, show that our approach effectively boosts the model performance and notably outperforms both text-only and vision-guided methods, suggesting the effectiveness and generalizability of our approach.

heading: 6 Related Work
text: Abstractive Text Summarization (ATS). Given the input textual article, the goal of ATS is to generate a concise summary (Hermann et al., 2015; Wang et al., 2022b). Thanks to generative pretrained language models (Lewis et al., 2020), ATS has achieved remarkable performance (Paulus et al., 2018; Liu and Lapata, 2019; Zhang et al., 2020;
9.0 points, respectively)
Goodwin et al., 2020; Rothe et al., 2021; Xiao et al., 2022; Xu et al., 2020; Yu et al., 2021b; Liang et al., 2022c; Wang et al., 2022a).
Multimodal Abstractive Summarization (MAS). With the rapid growth of multimedia, many MAS datasets have been built such as: SportsSum (Tjondronegoro et al., 2011), MovieSum (Evangelopoulos et al., 2013), MSMR (Erol et al., 2003), MMSS (Li et al., 2017), MSS (Li et al., 2018a), How2 (Sanabria et al., 2018), MSMO (Zhu et al., 2018), E-DailyMail (Chen and Zhuge, 2018), ECproduct (Li et al., 2020a), and MM-AVS (Fu et al., 2021). All these datasets, covering video summarization, movie summarization, meeting records summarization, sentence summarization, product summarization, and news summarization, aim to generate a summary based on multimodal inputs (text, vision, or audio). With the data resources extensively used, the MAS task has attracted much attention, where the existing work mainly focuses on how to effectively exploit the additional features which are generally implicitly learned by the MAS objective, having achieved impressive performance on these high-resource English datasets (Li et al., 2018b, 2020b; Zhu et al., 2020, 2021; Zhang et al., 2021b,a; Yu et al., 2021a). For example, Palaskar et al. (2019) and Zhang et al. (2021a) explore the hierarchy between the textual article and visual features, and integrate them into the MAS model. Liu et al. (2020) design a multistage fusion network to model the fine-grained interactions between the two modalities. And Yu et al. (2021a) study multiple multimodal fusion methods to infuse the visual features into generative pre-trained language models, e.g., BART (Lewis et al., 2020).
Multilingual Abstractive Summarization. It aims to train a model that can produce a summary in any language. Existing studies mainly pay attention to constructing the multilingual abstractive summarization dataset and there have been many datasets publicly available: MultiLing2015 (Giannakopoulos et al., 2015), GlobalVoices (Nguyen and Daumé III, 2019), MultiSumm (Cao et al., 2020), MLSUM (Scialom et al., 2020), MultiHumES (Yela-Bello et al., 2021), MassiveSumm (Varab and Schluter, 2021), MLGSum (Wang et al., 2021), and XL-Sum (Hasan et al., 2021). Most of these datasets are automatically constructed from online websites due to high human cost, which involves at least two languages.
There are two essential differences between the above work and ours: i) The MAS datasets and multilingual abstractive summarization datasets are either in multimodal or multilingual, while ours includes both. It is obvious that conducting multilingual MAS is more challenging due to the more complex scene (Jangra et al., 2021). Besides, our MM-Sum includes 44 languages, covering three settings: mid-high, low, and zero resource. What is more, our MMSum has the property that the knowledge can be transferred from mid-high resource languages to low- and zero-resource ones through visual features (as the bridge) while they have not. Tab. 10 of Appendix D provides a detailed comparison of available languages, modalities, and scenes for all datasets. ii) We mainly focus on how to obtain the summary-oriented visual features from the perspective of the summary rather than the article as existing work does. We thus propose two summaryoriented vision modeling tasks which are flexible and easy to be extended to existing MAS models.

heading: 7 Conclusion
text: In this paper, we propose to enhance the MAS model through two summary-oriented vision modeling tasks namely vision to summary task and masked image modeling task. They can explicitly force the MAS model to exploit the summaryoriented visual features and thus improve the summary quality. Extensive experiments on multiple settings demonstrate that our model significantly outperforms related baselines in terms of ROUGE scores and human evaluation. Furthermore, we contribute a large-scale multilingual MAS (MM-Sum)
dataset to the research community.
Limitations
Although we show that our SOV-MAS outperforms the VG-mT5 model under different setups, there are some limitations worth considering to study in future work: (1) In this study, we only provide 44 languages and conduct experiments on them, and future work could extend our method to more languages; (2) The used MAS model is based on the generative pre-trained language model, i.e., mT5 (Xue et al., 2021). The large-scale model size can bring promising performance while it also consumes more training time (all mT5-based models in this work cost about five days under the multilingual training setting) and releases more carbon dioxide, which may be inconsistent with the theme of green AI. Therefore, the work related to model compression (e.g., knowledge distillation) may be possibly future work for the multilingual MAS task.
Ethics Statement
In this section, we consider the potential ethical issues of our model. In this paper, we propose SOVMAS which is trained on the publicly-available BBC datasets. Therefore, SOV-MAS might lead to incorrect summaries in applications and involve the same biases and toxic behaviors exhibited by the datasets. Besides, we crawled the dataset from the BBC website4 and its permissions are granted to copy, distribute and modify the contents under the terms of the Creative Commons AttributionShareAlike 3.0 Unported License and Creative Commons CC0 License, respectively.

heading: Acknowledgements
text: The research work described in this paper has been supported by the National Key R&D Program of China (2020AAA0108001) and the National Nature Science Foundation of China (No. 61976015, 61976016, 61876198 and 61370130). The authors would like to thank the anonymous reviewers for their insightful comments and suggestions to improve this paper.

heading: A Dataset Statistics and Splits.
text: Tab. 7 shows that our MM-Sum covers 44 languages and in total includes 1,078,215 articlesummary pairs with 3,479,348 images, where each article-summary pair contains about 3.23 images on average. The average article and summary length for all languages are about 520 and 84, respectively. According to the dataset size of each language, we split them into three settings: Mid-High Resource, Low Resource, and Zero Resource. For mid-high and low-resource languages, following Hasan et al. (2021), we utilize about 80% training:10% validation:10% test splitting with one exception (English splitting is 93%:3.5%:3.5%). For zero resource, we follow Bugliarello et al. (2022) who investigate two scenarios: few-shot and zero-shot. Therefore, we also randomly sample 100 instances as the fewshot learning data and then split the rest with about 50% validation and 50% test.
B Implementation Details
Data Pre-Processing. Following Hasan et al. (2021), we pre-process the textual data by truncating or padding them into sequences of 512 tokens for X and the outputs Y to 84 tokens after using the 250k wordpiece (Xue et al., 2021) vocabulary provided with the mT5 checkpoint. For the image sequence, after the feature extraction as described in § 3.1, we also truncate or pad the sequence length to 180 (i.e., five images: 5 * 36; n=5, m=36).
Hyper-Parameters. Following Hasan et al. (2021), we use the base5 model of mT5 (Xue et al., 2021), in which L = 12 for both encoder and decoder. For the vision-related hyper-parameters mentioned in § 2.2, we follow Yu et al. (2021a) for a fair comparison. Specifically, we use a 4-layer encoder (i.e., H = 4) with 8 attention heads and a 2048 feed-forward dimension. For all models, the dropout is set to 0.1 and the label smoothing is set to 0.1. The d, dc, and dv are 768, 256, and 2048, respectively. The balancing factor α and β in Eq. 5 are set to 1.0, which are not tuned. The K of Eq. 6 is 29, which is the sum of the number of mid-highand low-resource languages. During the monolingual training, we train all models on each language separately for 6-20 epochs (since the total training samples were limited, we had to be careful to prevent overfitting) on an NVIDIA Tesla V100 GPU
5https://huggingface.co/google/mt5-base/tree/ main
with a batch size of 32. The models are optimized using Adam (Kingma and Ba, 2014) with β1=0.9 and β2=0.998. We train all model weights with a slanted learning rate schedule (learning rate to 5e-4). During the multilingual training, following a similar training strategy (Conneau and Lample, 2019; Hasan et al., 2021), we sample each batch from a single language containing 256 samples and use a smoothing factor (0.5) so that batches of low-resource languages would be sampled at a higher rate, increasing their frequency during training. We set the training step to 35,000 steps on a distributed cluster of 8 NVIDIA Tesla V100 GPUs and trained about 5 days. We use the Adafactor optimizer (Shazeer and Stern, 2018) with a linear warm-up of 5,000 steps and the “inverse square root” learning rate schedule.
For inference, we use beam search with beam size 4 and length penalty of γ = 0.6. When calculating the ROUGE scores, we use the multi-lingual rouge6 toolkit following Hasan et al. (2021). All experimental results reported in this paper are the average of three runs with different random seeds.

heading: C Case Study
text: Fig. 3 shows an example multimodal English document, the generated summary, and the ground truth summary. Though all generated summaries exhibit the core idea of the document and present factual consistency, ours has good lexical and semantics overlaps with the ground truth. And it is not difficult to find that with enhanced visual features our SOV-MAS can capture a sketch of the document, i.e., mourning the king with true devotion, and supplement a lot of details, i.e., dressed in black and weeping. These observations show that through two summary-oriented vision modeling tasks, our model could generate a better summary. We also believe that a more informative summary would meet the demand of the user.

heading: D Comparison to the Related Datasets
text: Tab. 10 provides information on the number of available languages, modalities, and scenes for all datasets. Specifically, multimodal abstractive summarization datasets and multilingual abstractive datasets are either multimodal or multilingual,
6https://github.com/csebuetnlp/xl-sum/tree/ master/multilingual_rouge_scoring
Transformer (ViT) (Dosovitskiy et al., 2020). “
:::: */*/* ” and “ */*/* ” denote statistically significant better than the
“VG-mT5” model with t-test p < 0.05 and p < 0.01 hereinafter, respectively. The “Avg.” indicates the average score for each group and the best average scores are bold.
while ours includes both. It is obvious that conducting multilingual multimodal abstractive summarization is more challenging due to the more complex scene (Jangra et al., 2021). Furthermore, our MM-Sum includes 44 languages, covering three settings: mid-high resource, low resource, and zero resource. What is more, our MM-Sum has the property that the knowledge can be transferred for MAS from mid-high-resource languages to lowand zero-resource languages via additional visual features as a bridge while they have not.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
3 A2. Did you discuss any potential risks of your work? 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 4.1
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 9

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 9
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Previous work (Hasan et al., 20) has checked this and our dataset is based on it.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 7 of Appendix
C 3 Did you run computational experiments? 4.2

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4.2
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 5.3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 5.3

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 9

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? 9
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 9
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Limitations
Although we show that our SOV-MAS outperforms the VG-mT5 model under different setups, there are some limitations worth considering to study in future work: (1) In this study, we only provide 44 languages and conduct experiments on them, and future work could extend our method to more languages; (2) The used MAS model is based on the generative pre-trained language model, i. e. , mT5 (Xue et al. , 2021). The large-scale model size can bring promising performance while it also consumes more training time (all mT5-based models in this work cost about five days under the multilingual training setting) and releases more carbon dioxide, which may be inconsistent with the theme of green AI. Therefore, the work related to model compression (e. g. , knowledge distillation) may be possibly future work for the multilingual MAS task